\section{Conclusion}

In this study we find that differences in author characteristics are associated with linguistic differences in stories and that rater characteristics are associated with differences in ratings. For authors, men were more likely than women to write about female characters and their social interactions, and compared to younger authors, older authors wrote more vivid and emotional stories. Raters preferred stories written all at once rather than broken up into multiple stages, and raters age 45 and older rate stories significantly lower than raters under age 45. We release our dataset, \datasetname, containing 1,630 stories with quality ratings and author and rater demographics.

Our results suggest that author and reader characteristics (e.g., demographics, personality) could explain variations in story writing evaluations.
While work has shown that some study designs are more robust against this variation, \cite[e.g., by ranking instead of rating]{yannakakis2015ratings}, rater differences
%Either way (in ranking and rating settings), reader differences 
could still lead to variation in annotations.
We recommend that evaluations include some ability to collect characteristics, such as a short demographics and personality questionnaire, in order to assess any influence of these variables.

Furthermore, future work could explore alternative ways of collecting author and reader characteristics during evaluations.
While demographic questionnaires are common and short (e.g., to collect gender and age would require two questions), full personality questionnaires are time consuming, asking multiple questions for each characteristic.
Study designers could instead use reduced questionnaires, such as the Mini-IPIP Big 5 personality scale~\cite{donnellan2006mini}.
Alternatively, focusing on fewer, more highly trained raters--that represent a diverse set of demographics and personality--could reduce the cost of collecting many rater demographics. 
Finally, future work should investigate whether annotator variance might be better captured with psychological factors related to reading (e.g., propensity for liking long sentences or fiction) rather than stable traits such as personality or demographics. 

Our results that author personality and gender were associated with topic selection and story writing also suggest that studies could leverage the behavior of participants to predict personality characteristics.
While these results are not yet strong enough to provide robust measures of personality or demographics, future studies could explore how to leverage these associations to predict author characteristics in story writing or other writing evaluations rather than relying on questionnaires. 



%   studying a rater variable that will best and most succinctly capture these variations. Based on these findings, we recommend (e.g., is there one question we could ask all raters that would account for all of this?) In your case, I think you should describe which demographics and personality dimensions should be accounted for in story writing evaluations and how you would recommend that people do that. I think recommending that people always collect age and gender is one thing. But you'll probably also have to recommend that every single NLP paper on this topic will have to report on the demographics of participants that contribute a dataset (e.g., to enable meta analyses) -- this may require a bit of a culture change in NLP. In addition, it would be great to have a quick paragraph on how you think people could do this for personality since it significantly extends the length of a task if you give people a whole personality questionnaire. Are there perhaps other ways of inferring personality? (Perhaps based on the story topic, since you identified that more agreeable people liked the dog stories better?) I'm aware that you may be out of space for this paper, but I think it would be worth it. I would much rather see the writing setup question (R3) completely disappear from the paper (since I think it's not closely related to the other two RQs and distracts from the focus on people's background  introducing biases) if it means that you could have a better discussion. I think ultimately, the discussion is what will make people refer to this paper. You'll want to make some recommendations…
 


% Limitations







% We additionally identify the impact of writing setup on story writing, with stories written all at once being significantly preferred by raters compared to stories written in chunks. 


% Our findings suggest that reader and writer demographics should be accounted for in human evaluations of story quality, and could inform the design and improvement of machine-in-the-loop writing systems.


% Recommendations
% Our findings are a step towards taking rater and author demographics into account when evaluating story writing and could inform the design and improvement of machine-in-the-loop writing systems. 

% Alternatively, focusing on fewer, more highly trained raters--that represent a diverse set of demographics and personality--could reduce the cost of collecting many rater demographics. 

% \ta{Add in rec for story set up, keep?}
% Our findings that stories written in chunks can have a detrimental effect on ratings can also inform the design and improvement of machine-in-the-loop writing systems. Study or system designers interested in exploring story writing can use these results to avoid artificially restricting writers by forcing them to write in chunks. 


% Additionally, more powerful linguistic analyses with more stories are needed  \cite[e.g., ][ analyze 70k authors' writing]{schwartz2013personality} to evaluate the strength and cause of these associations. 

% Conclusion
% We investigate the effect of author and reader identity and personality on how stories are written and rated.
% To this end, we collect and release \datasetname, containing 1,630 stories with quality ratings, along with both author and rater demographics.

% We find significant effects of author's identity on the types of stories written (e.g., men focused on character interactions), as well as significant associations of author and reader demographics (e.g., older readers rated stories as less entertaining and creative in general).
% Our findings suggest that reader and writer demographics should be accounted for in human evaluations of story quality, and could inform the design and improvement of machine-in-the-loop writing systems.


% In your case, I think you should describe which demographics and personality dimensions should be accounted for in story writing evaluations and how you would recommend that people do that. I think recommending that people always collect age and gender is one thing. But you'll probably also have to recommend that every single NLP paper on this topic will have to report on the demographics of participants that contribute a dataset (e.g., to enable meta analyses) -- this may require a bit of a culture change in NLP. In addition, it would be great to have a quick paragraph on how you think people could do this for personality since it significantly extends the length of a task if you give people a whole personality questionnaire. Are there perhaps other ways of inferring personality? 


% Overall, we find that indeed, differences in author characteristics are associated with linguistic differences in stories, and rater characteristics correlate with differences in ratings.
% Our findings are simply measuring associations between aggregate categories (e.g., number of pronouns used, authors over age 45), and should not be interpreted as applying to individual data points with specific contexts.
% Additionally, more powerful linguistic analyses with more stories are needed  \cite[e.g., ][ analyze 70k authors' writing]{schwartz2013personality} to evaluate the strength and cause of these associations. 


% \ms{Make slightly more positive?}
%Many of our results focus on common words that across stories written by different authors.
%While useful as a high-level analysis of writing, looking at individual word usage does not provide insight into the context of these words, such as how they might be used differently depending on the author.
%In addition, our analysis reports on a number of significance tests, the results of which suggest interesting associations between author and reader demographics.
%However, more work should be done in order 

% \ta{Generalize the personality and demographics, points a bit more.}

% Overview -- conclusion
% We investigate the effect of author and reader identity and personality on how stories are written. 