\documentclass[11pt,a4paper]{article}
\usepackage[dvipsnames]{xcolor}
\usepackage[hyperref]{acl2020}
\usepackage{times}
\usepackage{latexsym}
\usepackage{graphicx}
\usepackage{xspace}
\usepackage{booktabs}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{multirow,multicol}
\usepackage{stfloats}
\usepackage{array}


\newcommand{\datasetname}{\textsc{StoriesInTheWild}\xspace}

\newcommand{\kr}[1]{\textcolor{magenta}{\textit{KR}: #1}}
\newcommand{\ta}[1]{\textcolor{cyan}{\textit{TA}: #1}}
\newcommand{\ms}[1]{\textcolor{purple}{\textit{MS}: #1}}
\newcommand{\nas}[1]{\textcolor{blue}{\textit{NO: #1}}}
\newcommand{\liz}[1]{\textcolor{orange}{\textit{EC}: #1}}

\renewcommand{\UrlFont}{\ttfamily\small}
\aclfinalcopy
% \setlength\titlebox{5cm}
\newcommand{\aspaceTwo}{\hspace{2em}}
\newcommand{\aspaceOne}{\hspace{2em}}

\newcommand\uw{$^\dagger$}
\newcommand\aitwo{$^\diamondsuit$}

\author{
    Tal August\uw \aspaceTwo Maarten Sap\uw \aspaceTwo Elizabeth Clark\uw \\
    \textbf{Katharina Reinecke}\uw \aspaceOne \textbf{Noah A. Smith}\uw\aitwo\\
    \uw Paul G. Allen School of Computer Science \& Engineering, \\
    University of Washington, Seattle, WA, USA \\
    \aitwo Allen Institute for Artificial Intelligence \\
}

\title{
    % Exploring Author and Reader Demographic in Storytelling
    Exploring the Effect of Author and Reader Identity\\
    in Online Story Writing: the \datasetname Corpus %\thanks{~~Archival submission}
    % Do you wanna write a story?\\
}


\begin{document}

\maketitle

\begin{abstract}
Current story writing or story editing systems rely on human judgments of story quality for evaluating performance, often ignoring the subjectivity in ratings.
We analyze the effect of author and reader characteristics and story writing setup on the quality of stories in a short storytelling task. 
To study this effect, we create and release \datasetname, containing 1,630 stories collected on a volunteer-based crowdsourcing platform.
Each story is rated by three different readers, and comes paired with the author's and reader's age, gender, and personality.

Our findings show significant effects of authors' and readers' identities, as well as writing setup, on story writing and ratings.
Notably, compared to younger readers, readers age 45 and older consider stories significantly less creative and less entertaining. Readers also prefer stories written all at once, rather than in chunks, finding them more coherent and creative.
We also observe linguistic differences associated with authors' demographics (e.g., older authors wrote more vivid and emotional stories). 
Our findings suggest that reader and writer demographics, as well as writing setup, should be accounted for in story writing evaluations.%\footnote{Archival submission.}
% adapted from extended abstract we wrote for Noah

% There is also some evidence that author and reader personality are associated with ratings of creativity.
% \ms{Compress the following two sentences}
% To study writing setups, we assigned authors to either write an entire 5-10 sentence story about a sequence of images, or write 1-2 sentences sequentially for each image.
% We find that readers prefer stories written with the full story writing setup, finding them more coherent and creative.
% short stories. written about a sequence of 5 images on a volunteer-based crowdsourcing platform.

% We collect 1600 stories in two writing setups: \textit{in full}, where participants write a 5-10 sentence story all at once, and \textit{in chunks}, where participants are restricted to writing 1-2 sentences at a time, each corresponding to one of the images. We then obtain ratings of story quality, coherence, and how engaging the story is for each story through a human evaluation task.
% Notably, readers over the age of 45 rate stories significantly less creative and entertaining, and like them less compared to readers under the age of 45, and there is some evidence that readers over age 45 find stories written by authors over 45 more entertaining. 
% associated with lower preference for interact with the reader's preferences, with readers over 45 finding stories written by authors over 45 more entertaining.
%, and highly creative authors included more high-level insights into events and characters.
% We also find effects of both author and reader demographics in story writing and rating. 
% Notably, raters over 45 prefer stories written by authors over 45, finding them more entertaining. 
% Additionally, authors who are higher in openness write stories that are rated more creative, and raters who are higher in openness generally find stories more creative. 
% \ms{Compress the following two sentences}
% To study writing setups, we assigned authors to either write an entire 5-10 sentence story about a sequence of images, or write 1-2 sentences sequentially for each image.
% We find that readers prefer stories written with the full story writing setup, finding them more coherent and creative.
% We show that raters in general prefer stories written in full, finding them more coherent and creative.
%\ms{Compress the following two sentences}
%We show an additional effect of writing setup on story quality, finding that stories written all at once are preferred over those written sentence-by-sentence.
% As an additional result, we analyze the effect of writing setup on story quality, 
% readers prefer stories written in full, compared to stories written one or two sentences at a time sequentially.

% show that human evaluations of story quality are influenced by reader and writer demographics, 

%provide insights for designing machine-in-the-loop writing systems and 
% From crowdsourcing reliable image labels to writing a full novel, methods for collecting writing on-demand are often plagued by the low quality of individual submissions. To remedy this, many crowdsourcing tasks either use templates to standardize writing or quality check crowd work continuously with complex workflows. Although sometimes effective, these techniques restrict the naturalness of language or require heavy overhead, reducing scalability. On the other hand, volunteer-based crowdsourcing benefits from participants who are intrinsically motivated, leading to higher quality contributions in contexts where the contribution is subjective or difficult to verify, such as writing.  We apply this insight to develop a methodology for sustainably crowdsourcing high quality writing using volunteers. We test this methodology by creating the [] dataset, a collection of visually grounded, naturalistic stories with rich author and prompt metadata. We discuss applications of our methodology to other writing-intensive crowdsourcing tasks and demonstrate potential uses of  the [] dataset, including for Natural Language Generation (NLG) and Inference (NLI), and computational sociolinguistic analysis.
\end{abstract}

%%%%%%%%% Tal notes
% We found that raters over the age of 45 on average found stories significantly less entertaining, more confusing and overall liking them less than raters under 45. \ta{will put in significance, report in a table?}

% Interestingly, these tendencies flip when the author was also over the age of 45: when both the rater and the author are over the age of 45, raters found stories significantly more entertaining and overall liked them more compared to stories authored by those under 45. 
% We found that both reader and writer openness to change is associated with significantly higher ratings of creativity. Interestingly, the interaction between reader and writer openness is negative, suggesting that although openness to change is associated with higher scores of creativity, it is associated with lower scores when the reader has a high Openness and the writer has low openness, or vice versa \ta{did I interpret that correctly?}. 

\input{01-intro}

\input{02-hypotheses}

\input{03-collection}

\input{04-analyses}

% \input{04-related-work}

\input{05-discussion}

% \section{Conclusion}
% We investigate the effect of author and reader identity and personality on how stories are written and rated.
% To this end, we collect and release \datasetname, containing 1,630 stories with quality ratings, along with both author and rater demographics.

% We find significant effects of author's identity on the types of stories written (e.g., men focused on character interactions), as well as significant associations of author and reader demographics (e.g., older readers rated stories as less entertaining and creative in general).
% Our findings suggest that reader and writer demographics should be accounted for in human evaluations of story quality, and could inform the design and improvement of machine-in-the-loop writing systems.

\section*{Acknowledgements}
This work was partially funded by NSF award 1651487, an NSF graduate research fellowship, and the DARPA CwC program through ARO (W911NF-15-1-0543). We thank the anonymous reviewers and members of the UWNLP community for their helpful feedback. 
We would also like to thank the LabInTheWild volunteers for writing stories and the Mechanical Turk workers for reading them.


\bibliographystyle{acl_natbib}
\bibliography{references}

\clearpage
\appendix
\input{08-00-appendix}

\end{document}
