\section{Background and Research Questions}
To guide our study, we craft several research questions informed by existing literature on story writing and the relationship between author identity and language, outlined below.
% We outline below relevant literature on story writing and the relationship between author identity and language. This work guides our own analyses through hypotheses on how story writing and author identity traits are associated. 

% \paragraph{Author and rater traits}
\paragraph{RQ1}\textit{How are author gender, age, and personality traits associated with language variation in stories?}
A wealth of work has shown association between an author's mental states and their language patterns, showing variation in pronoun usage, topic choices, and narrative complexity correlates strongly with the author's age and gender \cite{nguyen2016computational} and moderately with their personality \cite{yarkoni2010personality}.
We aim to confirm these differences in a prompted storytelling setting, since most work has focused on self-narratives \cite[e.g., diaries and  social media posts;][]{Pennebaker1999-gm,Hirsh2009-ic,schwartz2013personality}, with the exception of the essays studied by \citet{pennebaker2014small}.

%\cite[e.g.,][]{Pennebaker1999-gm,argamon2009automatically}.
%Specifically, prior work has shown differential usage of function

%While there is a wealth work that studies the association between language and author mental state, most has focused on self-narratives \cite[e.g., diaries, social media posts][]{Pennebaker1999-gm,Hirsh2009-ic,schwartz2013personality}.
%Little work has analyzed the author characteristics in prompted writing, with the essay analyses by \citet{pennebaker2014small} being a notable exception.
%In much of this work writer's mental state and personality are associated with different usage of \ta{pronouns`\cite{}, etc. Maarten do you have work to show this?}. Much of this work has shown that gender and age are two major sources of language variation~\cite{nguyen2016computational}, as well as author personality~\cite{}. 

% \textbf{H1:} Author gender, age, and personality traits \ta{specifically?} are associated with language variation in stories. 

\paragraph{RQ2}\textit{How are rater gender, age, and personality traits associated with variation in story quality ratings?}
Ratings of stories are often only used to evaluate a story writing system's output \cite[e.g.,][]{fan2018hierarchical,yao2019plan} or to develop automatic evaluation metrics \cite[e.g.,][]{Hashimoto2019UnifyingHA, Purdy2018PredictingGS}, ignoring the rater's identity.
However, prior work has shown differences in crowdsourcing worker's behavior or annotations based on task framing \cite{levin2002new,august2018framing,sap2019risk} or the annotator's own identity or experiences \cite{breitfeller2019findingmicroagressions,geva2019we}.
We seek to confirm and characterize these differences in our story rating task. As a follow-up to \textbf{RQ2}, we also investigate the interaction between author and rater demographics on story ratings.


% \paragraph{\textit{RQ2b}} As a follow-up to \textbf{RQ2}, we also investigate the interaction between author and rater demographics on story ratings. \liz{The format of this is different from the other RQs, and I found it a little distracting. Could we not just put, "We also investigate..." as a single sentence at the end of RQ2?}

% Despite cooperative story writing and natural language generation research heavily relying on human ratings of text for evaluation \cite[e.g.,][]{liu2016not,zhang2020bertscore}, a majority have ignored the rater's identity for such tasks.
% On the other hand, other work has previously studied the effect of rater's identity on annotation tasks \cite{}, suggesting that rater traits are sometimes associated with changes in perception of tweets.
% In a similar vein, work has shown that message framing for advertising online studies~\cite{august2018framing} or products can elicit different reactions in people with different personality or demographics~\cite{levin2002new}.

% Our work is the first to investigate the effect of a rater's identity on the quality ratings of stories. Based on past research, we hypothesize that rater gender and age, and personality traits are associated different ratings of stories:

% \textbf{H2:} Rater gender, age, and personality traits \ta{specifically?} are associated with variation in story quality ratings. 

\paragraph{RQ3} \textit{Is writing setup associated with different ratings of story quality?}
Past work has investigated story writing as a turn-taking game~\cite{clark2018creative} or as a distributed activity~\cite{teevan2016supporting} rather than a single event.
We investigate whether writing setup (writing a story all at once or sentence-by-sentence) impacts overall story quality.
% It is unclear, though, how writing setup can impact overall story quality, which is what we set out to investigate in this paper. 





% \ms{Author demographics in text is a thing that people do (cite comp socioling, pennebaker, lucie flekova stuff, maybe survey paper?), but (1) not a lot of work focuses on stories, and (2) we're the first to look at reader demographics for evaluating stories (people usually just trust human eval at face value without looking at variation)
% Also mention (other than framing of stories).
% }



% crowdsource writing 
% \ms{My sense is that this isn't relevant enough to include (esp. if we're over page limit)} \ta{Sure, happy taking it out.}
% Most past work on collecting story writing from individuals have either scraped stories from online forums~\cite{writing prompts} or sourced small stories from paid crowdsourcing~\cite{roc}. Our focus on collecting author and rater demographics is unique and affords our analysis of the influence of both on story quality ratings.

% Work in crowdsourcing has also explored collective story writing~\cite{mech novel} or writing support tools powered by crowdworkers~\cite{soylent}. Technological support for writing has also included NLG power systems~\cite{clark2018creative}, and systems for breaking writing tasks down into small microtasks to be completed in short moments, such as while mobile~\cite{}. In this work we instead focus on writing setup and reader and writer traits on story writing quality rather than introduce new writing support systems. 


% \ta{Crowdsourcing writing: Mech Novel, Soylent, Ensemble, ROC. Most of these are paid, interested to see how unpayed would work, also rarely focuses on author demographics, just their writing. }
