
% \begin{figure*}[t]
%     \centering
%     \begin{subfigure}[]{1\columnwidth}
%         \centering
%         \includegraphics[width=0.9\columnwidth]{img/dog.pdf}
%         \caption{}
%         \label{fig:dog}
%     \end{subfigure}%
    
%     \centering
%     \begin{subfigure}[]{1\columnwidth}
%         \centering
%         \includegraphics[width=0.9\columnwidth]{img/jail.pdf}
%         \caption{}
%         \label{fig:jail}
%     \end{subfigure}%
    
%     \centering
%     \begin{subfigure}[]{1\columnwidth}
%         \centering
%         \includegraphics[width=0.9\columnwidth]{img/park.pdf}
%         \caption{}
%         \label{fig:park}
%     \end{subfigure}%
    
%     \centering
%     \begin{subfigure}[]{1\columnwidth}
%         \centering
%         \includegraphics[width=0.9\columnwidth]{img/marathon.pdf}
%         \caption{}
%         \label{fig:maratho}
%     \end{subfigure}%
    
%     \centering
%     \begin{subfigure}[]{1\columnwidth}
%         \centering
%         \includegraphics[width=0.9\columnwidth]{img/snow.pdf}
%         \caption{}
%         \label{fig:snow}
%     \end{subfigure}%
    
%     \caption{\label{fig:all-prompts} Prompts used in the story writing stage of our data collection. (a) is dog, (b) is jail, (c) is park, (d) is marathon (e) is snow.\ms{Make this a table of images with the labels on the left side}}
% \end{figure*}

We introduce and release \datasetname, containing \ms{1,600} short stories (\S\ref{sec:stories}) paired with author demographics and personality information. 

We pair these stories with third-party ratings (\S\ref{sec:ratings}) to evaluate the effect of writing setup and author identity on story writing. We additionally analyze how rater personality and demographics are associated with story ratings. We release the stories, paired with author demographic information, ratings, and rater demographic information, as the \datasetname Collection\footnote{\url{anonymous.com}}. 

\subsection{Crowdsourcing stories}. 


We began by collecting a dataset of \ta{CHECK: ~$1600$} stories using a volunteer-based online study platform, LabintheWild~\cite{reinecke2015labinthewild}. LabintheWild recruits participants to take studies using intrinsic motivations, such as the the desire to compare oneself to others or support science~\cite{Jun2017}. Following best practices in recruiting on LabintheWild, we advertised our study as a way for participants to see how a simple NLP system would predict their personality based on their writing. The study was advertised on the front page of LabintheWild and posted on social media to recruit participants.




After agreeing to a consent form, participants filled out a demographics survey with their age, gender, race, education level, and native language and then selected 1 of 4 thumbnails to write a story about. The 4 thumbnails were of a dog in the park, a cartoon of two men having an argument, a picture of a marathon, and a cartoon of two people in a park. \liz{add all prompts to appendix?} Each thumbnail represented a series of 5 images that participants saw once selecting a thumbnail. The images were selected from the Visual Storytelling dataset of Flickr images \citep{huang2016visual} and a cartoon dataset~\cite{iyyer2017amazing}. Once participants selected an image, they were presented with the 5 images and a text box in 1 of 2 formats: (1) all at once or (2) broken up in an accordion format. Each format is described in more detail below: 
 \begin{enumerate}
     \item \textbf{all at once}: Participants were shown all 5 images with a text box underneath and asked to write a 5-10 sentence story about the images. For more details, see Figure~\ref{fig:all_interface}.
     \item \textbf{accordion format}: Participants were shown each image separately in an accordion with a text box underneath, where they were instructed to write 1-2 sentences about that image to continue their story. Once they submitted one text box, they were able to see the next image and write about it. Participants were able to go back and see, but not edit, previous text boxes once they were submitted. At the top of the accordion, all the images were shown. For more details, see Figure~\ref{fig:accordion_interface}.  
 \end{enumerate}
 
In both conditions, participants were instructed to tell a story rather than just describe the images, to make sure their story had a clear beginning, middle, and end, and to use correct punctuation. 
 

Following the story writing, participants could optionally fill out a personality questionnaire, using the Ten Item Personality Measure \cite[TIPI;][]{gosling2003very} of the Big 5 personality traits \cite{costa2008revised}.
% Following the story writing, participants were asked if they would like to finish the study and see their predicted personality or if they would like to take a short personality questionnaire first.
% The personality questionnaire was based on the Big-5 personality inventory used in~\cite{}.\ta{Maarten?}. 
Following this questionnaire, participants were asked if they had cheated or provided false information on the study and any comments they had about the study. After these comments participants were presented with a results page that debriefed them on the goals of the study and provided them with their predicted Big-5 personality traits based on their writing and (if they completed it) the results of their personality questionnaire. 


% \ms{
% Our raters were 
% 56\% women, 42\% men, 2\% other or declined
% 40 on average, ($\pm$12)
% 79\% white, 6\% African-American, 6\% Asian, rest other.
% 15($\pm$ 3) years since primary school.}
