
\section{\datasetname Collection}

We introduce and release \datasetname, containing 1,630 short stories (\S\ref{sec:stories}) paired with author demographics and personality information.\footnote{Each stage of data collection was approved by the authors' institutional review board (IRB).}
We pair these stories with third-party ratings (\S\ref{sec:ratings}) to evaluate the effect of writing setup and author identity on story writing.

\subsection{Crowdsourcing Stories}\label{sec:stories}

To construct \datasetname, we first collected 1,630 written stories using a volunteer-based online study platform, LabintheWild~\cite{reinecke2015labinthewild}.\footnote{LabintheWild recruits study participants using intrinsic motivations (as opposed to monetary compensation, cf.~Amazon Mechanical Turk), such as the the desire to compare oneself to others or to support science~\cite{Jun2017}.}
Following best practices in recruiting on LabintheWild~\cite{august2018framing}, we advertised  our study as a way for participants to learn more about themselves by seeing how a simple pronoun-based classifier can predict their personality based on their story writing (described in Appendix \ref{sup:litw-motivator}).
% moved to appedix: Following best practices in recruiting on LabintheWild, we advertised our study as a way for participants to see how a simple NLP system would predict their personality based on their writing. The study was advertised on the front page of LabintheWild and posted on social media to recruit participants.

% \ms{Add one sentence about which motivation we used to have people do our task, and maybe that we used a simple pronoun based classifier to predict their personality?}

\input{03-01-data-stats}

We first collected  participants' identity and demographics (age, gender, race, and education level).
Then, participants chose the \textit{topic} of their story by selecting one of five preview thumbnails, each representing one of five image strips that participants subsequently used as prompts for their story.
% After agreeing to a consent form, participants filled out a demographics survey with their age, gender, race, education level, and native language and then selected 1 of 4 thumbnails to write a story about. The 4 thumbnails were of a dog in the park, a cartoon of two men having an argument, a picture of a marathon, and a cartoon of two people in a park. \liz{add all prompts to appendix?}
We selected the images from the Visual Storytelling dataset of Flickr images \citep{huang2016visual} and a cartoon dataset~\cite{iyyer2017amazing}. All images are shown in full in Figure \ref{fig:all-prompts} in Appendix \ref{sup:data-collection}.

\paragraph{Writing setup}%\ms{Could be shortened?}
After choosing a topic, all authors are presented with a five image-strip corresponding to the topic they chose to write about.
We then randomly assign authors to one of two writing setups: (1) \textit{all at once} or (2) \textit{sequential}, %sh own in Figures \ref{fig:all_interface} and \ref{fig:accordion_interface}, respectively, 
both shown in Figure \ref{fig:writing-setups} in Appendix \ref{sup:data-collection}.
In (1), participants simply write a full 5--10 sentence story.
In (2), participants are instructed to write five sets of 1--2 sentences in an accordion of text boxes, each box corresponding to an image in the strip.
This second writing setup is inspired by recent work on machine-in-the-loop turn-taking writing for story writing~\cite{clark2018creative}.
Once each text box is submitted, participants can no longer edit that text. 

% For each image, participants could continue the story by relying on the box's image and their previously written context.

% \ms{Moved this all to the appendix}
% Once participants selected an image, they were presented with the 5 images and a text box in 1 of 2 formats: (1) all at once or (2) broken up in an accordion format. Each format is described in more detail below: 
%  \begin{enumerate}
%      \item \textbf{all at once}: Participants were shown all 5 images with a text box underneath and asked to write a 5-10 sentence story about the images. For more details, see Figure~\ref{fig:all_interface}.
%      \item \textbf{accordion format}: Participants were shown each image separately in an accordion with a text box underneath, where they were instructed to write 1-2 sentences about that image to continue their story. Once they submitted one text box, they were able to see the next image and write about it. Participants were able to go back and see, but not edit, previous text boxes once they were submitted. At the top of the accordion, all the images were shown. For more details, see Figure~\ref{fig:accordion_interface}.  
%  \end{enumerate}
 
In both setups, participants are instructed to tell a story rather than just describe the images, to make sure their story has a clear beginning, middle, and end, and to use correct punctuation. The task took around 9 minutes in both conditions. 

%\ms{Compress if needed for space -- wait for Noah / Katharina}
Following the story writing, participants can optionally fill out the Ten Item Personality Measure \cite[TIPI;][]{gosling2003very}, a short personality questionnaire based on the Five Factor Model \cite[FFM;][]{costa2008revised}.%
\footnote{The FFM delineates five dimensions of personality (openness to experience, conscientiousness, extraversion, agreeableness, neuroticism), each represented as a continuous score.
For more details, we refer the reader to \url{http://en.wikipedia.org/wiki/Big_Five_personality_traits}.}
%Conscientiousness, for example, measures someone's tendency to be efficient or organized vs. easy going or careless, and agreeableness relates to how friendly or compassionate vs. detached or confrontational someone is.


% \ms{Unsure if we need this? moved to appendix}
% Following this questionnaire, participants were asked if they had cheated or provided false information on the study and any comments they had about the study. After these comments participants were presented with a results page that debriefed them on the goals of the study and provided them with their predicted Big-5 personality traits based on their writing and (if they completed it) the results of their personality questionnaire. 

\paragraph{Author demographics}
Of the authors in \datasetname, 57\% were women and 40\% men (3\% declined to state their gender), with an average age of 25$\pm$12 years and an average of 14.30$\pm$4.20 years of education including primary school.
Of the authors, 56\% were white, 28\% Asian, and 3\% African-American (13\% selected another ethnicity/race); we did not restrict participation to any specific country.
1,133 (70\%) authors took the personality questionnaire.
% \ms{Also how many people declined to take the quiz?} \ta{Not restricted, 497 did not take it.}
%\ms{Tal: is there any way to link authors if they wrote multiple stories? We have retake info but that's not the same.} \ta{No, big issue with LITW, we only have retake data. We could try for IP, but that is messy.}

\subsection{Rating Stories}
\label{sec:ratings}
We create an Amazon Mechanical Turk task to obtain quality ratings for each of the stories collected in our previous task.
For each story, we ask U.S.-based workers to rate stories on 6 dimensions (listed in Table \ref{tab:data-stats}), using a 7-point Likert scale.\footnote{To ensure the quality of responses, we restrict the task to workers with 99\% or above approval rate and at least 1000 HITs approved. Additionally, we ask that workers write out a short piece of feedback to improve the story, to encourage them to think critically while rating stories.}
Those dimensions include 5 fine-grained quality dimensions (e.g., grammaticality, coherence), as well as an overall impression of the story (``I liked this story'').
Each worker also optionally filled out their demographics information (age, race, gender, education level).
Additionally, as a measure of intellect and creativity, workers filled out the four openness items from the Mini-IPIP Big 5 personality scale \cite{donnellan2006mini}.

\paragraph{Rater demographics}
56\% of our raters were women and 42\% were men. 79\% identified as white, 6\% as African-American, and 6\% as Asian.
On average, their age was 40$\pm$12 years, and they had 15$\pm$3 years of education, including primary school.


% \paragraph{Differences in writing setup and topic choice}
% Using our ratings, we determined which of the writing setups led to stories that were preferred (\textbf{RQ3}). 
% We averaged the ratings for each story, and report differences using Cohen's $d$.

% \ta{Maarten, thoughts on moving this to RQ3 in analysis?}
% \ta{Also cohen's d on all these}

% As shown in Table \ref{tab:data-stats}, stories written in full were preferred and rated as being higher quality across all dimensions, compared to stories written sequentially.

% We also find that certain story topics were liked more than others ($F=26.17$, $p<0.$001). 
% Specifically, dog-stories were liked significantly more than others ($p<0.$001), and jail-stories significantly less ($p<0.$001).
% %and stories written about the jail scenes were liked significantly less than all other stories ($p<0.$001).
% To account for these effects, we control for story images and writing setup in all subsequent analyses. 


% \ms{
% Our raters were 
% 56\% women, 42\% men, 2\% other or declined
% 40 on average, ($\pm$12)
% 79\% white, 6\% African-American, 6\% Asian, rest other.
% 15($\pm$ 3) years since primary school.}

% \subsection{Rating Writing Quality}

% Considering that our claim in this paper that volunteer-based online crowd-sourced writing will be of higher quality than in paid crowdworking contexts, it is important for us to define what we mean by `higher quality.'  

% Crowdsourced writing quality has been evaluated in a number of different ways, including by other crowdworkers [], expert raters [], and automatically []. The type of quality checks employed depends on what the writing will eventually be used for: for example, when writing tasks are orchestrated by a single `author' giving writing tasks (such as,  `rewrite this paragraph') to crowdworkers, evaluation of the writing is either expert (or author) rated [], or the writing is checked by other crowd workers []. In large-scale datasets used for NLP purposes (such as SNLI or image annotation datasets), either a subset or the entire dataset is rated for coherence or correctness (i.e., does this sentence entail this other sentence), by crowdworkers []. Often  the writing of these datasets is also analyzed by NLP systems (e.g., [visual dataset survey]). 

% Quality means different things for different writing tasks also: the quality of a journal paper might rely more heavily on grammaticality and coherence and less on creativity, whereas this might not be the case for a short story or poetry []. Annotation or natural language inference datasets (eg. [], []) tend to assess quality in terms of coherence [] and grammaticality [], while past work in other crowdsourced writing tasks additionally assess writing quality on a number of other dimensions, such as writing style, presence of story structure. 

% Based on an informal review of prior work in this area, the most common metric for writing quality tend collect crowdsourced ratings on on \textbf{coherence} [ROC, Mnovel, Writing in loop], \textbf{grammaticality} [Mnovel, Soylent, Writing Loop, Wearwrite], and for story writing [Mnovel, Writing in Loop], \textbf{creativity} and \textbf{entertaining}. 

% To analyze the effectiveness of using volunteer-based crowdsourcing over paid crowdsourcing, We collected crowdsourced ratings of 100 randomly selected [100] stories from our collected stories across the above metrics. We then compared these ratings with [100] stories collected from Mechanical Turk using the same experiment, but without the inclusion of the feedback. This analysis is similar to [Mnovel], which compared their crowdsourced stories with stories created from a simpler crowdsourcing workflow across a number of the above metrics []. 

