\section{Introduction}
Reading or writing a story is an inherently subjective task that depends on the experiences and identity of the author, those of the reader, and the structure of the writing process itself~\cite{morgan1935method,Conway2000-nl, clark2018creative}.   
Despite this subjectivity, many natural language processing tasks treat human judgments of story quality 
as the gold standard for evaluating systems that generate or revise text.
In creative applications, such as machine-in-the-loop story writing systems \citep{clark2018creative}, it is important to understand sources of variation in judgments if we hope to have reliable, reproducible estimates of quality.  

In this work, we investigate how an author's and reader's identity, as well as overall writing setup, influence how stories are written and rated.
We introduce and release \datasetname,%
\footnote{At
\url{http://tinyurl.com/StoriesInTheWild}
%\url{https://gist.github.com/talaugust/edc87a274ae6dcedf43774a8d9e91bb4}
}
containing 1,630 short stories written on a volunteer-based crowdsourcing platform, paired with author demographics and personality information.
For each story, we obtain three sets of ratings from third-party evaluators, along with their demographics and personality.


Our findings confirm that author identity, reader identity, and writing setup affect story writing and rating in \datasetname.
Notably, people in general preferred stories written in one chunk rather than broken up into multiple stages. Raters age 45 and over generally rated stories as less creative, more confusing, and liked them less compared to raters under age 45.
Additionally, we find that, in our corpus, men were more likely than women to write about female characters and their social interactions, and compared to younger authors, older authors wrote more vivid and emotional stories.
We also find evidence of reader and author personality, and their interaction,  influencing ratings of story creativity. 

Our new dataset and results
are first steps in analyzing how writing setup and author and reader traits can influence ratings of story quality, and suggest that these characteristics should be accounted for in human evaluations of story quality. 


% Our results and provided dataset 

%While it is widely understood that the perception of the ``quality'' of a story is inherently subjective, depending on the experiences of the reader \cite{morgan1935method,Conway2000-nl}, natural language processing methodology necessarily views human judgments 

% \ms{need better phrasing} \ta{How's this?}
% Technological support for story writing has traditionally been restricted to copy editing tasks (e.g., spell check), but recently interest is growing in developing more intelligent writing tools for authors~\cite[e.g., machine-in-the-loop story writing systems;][]{clark2018creative}. These new tools often leverage NLG systems, which improve by evaluating the quality of stories with external readers, as illustrated in Figure \ref{fig:introFig}. However, currently, little attention is paid to the effect of author and reader characteristics or identity in writing or rating stories, despite evidence that personal experience influences how people produce and interpret stories \cite{morgan1935method,Conway2000-nl}.


% There is increased interest in developing tools to help authors write better stories \cite[e.g., machine-in-the-loop story writing systems;][]{clark2018creative}.
% To improve, such systems usually require evaluating the quality of stories with external readers, as illustrated in Figure \ref{fig:introFig}.
% % In many story writing tasks, an author will usually write a story which is then rated by a reader, as shown in Figure \ref{fig:introFig}.
% However, currently, little attention is paid to the effect of author and reader characteristics or identity in writing or rating stories, despite evidence that personal experience influences how people produce and interpret stories \cite{morgan1935method,Conway2000-nl}.

% Stories are integral to human communication. People tell stories to explain, entertain, and remember~\cite{gottschall2012storytelling,somethingElse}. How someone tells a story--what words they choose, or where they elaborate \ta{thinking we can site Pennebaker or the Perceptual style test here?}--carries information about the storyteller, such as \ta{their demographics?}~\cite{morgan1935method} \ms{maybe point to Figure \ref{fig:introFig}}. Additionally, writing setup can substantially impact the way people write, potentially influencing the telling of a story\ta{?}\cite{}. 

% something introing writing in the loop earlier
% Machine-in-the-loop writing systems are gaining popularity as a potentially useful tool for story writing assistance. As these systems become more popular, understanding how writing setup and author traits can influence story writing are important considerations for system design. 

% Stories have long been used to convey thoughts, traditions, and memories \cite{gottschall2012storytelling,somethingElse}, and those stories carry information about the storyteller's inner workings \cite{morgan1935method}.
% For example \ms{maybe point to Figure \ref{fig:introFig}, and highlight words indicative of author demographic?}.
% Additionally, the way in which people tell stories can influence its reception \cite{something}.

%Storytelling is a natural way for people to convey their inner thoughts, real or fictional, to others \cite{gottschall2012storytelling,somethingElse}.
%Psychologists have long used storytelling the Thematic Apperception Task \cite[TAT;][]{morgan1935method}.

% \begin{figure}[t]
%     \centering
%     \includegraphics[width=\columnwidth]{img/IntroFig_v0.pdf}
%     \caption{Example of a rated story in \datasetname, along with author and rater information.\ms{maybe update to add images+ illustrate the findings?}}
%     \label{fig:introFig}
% \end{figure}

% We prompt authors on a volunteer-based crowdsourcing platform with a sequence of 5 images, asking them to write a story about them either \textit{in full} or \textit{in chunks}.
%We then collect ratings of quality for each story from third-party evaluators, paired with raters' demographics and personality.

% Paragraph about story setup and author identity
%\ms{Outline the main personality author-reader findings, as well as the author-LIWC personality findings}
%There is also some evidence that stories by authors over age 45 were more likely to be rated as more entertaining by readers who were also over age 45.
% Our findings indicate that stories written in full were significantly preferred over stories written in chunks.

% Paragraph about raters and their identity
% Furthermore, we find that rater's age and personality influence their ratings of stories. Notably, raters over 45 prefer stories written by authors over 45, finding them more entertaining. Additionally, authors who are higher in openness write stories that are rated more creative, and raters who are higher in openness generally find stories more creative. 

% Our contributions are:

% \begin{enumerate}
%     \item the release and analysis of \datasetname, a dataset of 1,600 rated stories paired with author and rater demographic information
%     \item results suggesting reader and writer demographics should be accounted for in human evaluations of story quality, and
% \end{enumerate}


% Our findings inform the design of machine-in-the-loop writing systems by showing the benefits of structuring story writing into a single document rather than as a sentence-by-sentence endeavor. 

% suggest reader and writer identity should be accounted for in human evaluations of story quality. Our
% \ta{I like these takeaways! Feels legit.} \liz{same}

%inform the design of machine-in-the-loop writing systems


