
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Remove the "review" option to generate the final version.
\usepackage{EACL2023}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{booktabs}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{subcaption}
\captionsetup[subfigure]{labelformat=empty}
% If the title and author information does not fit in the area allocated, uncomment the following
%
\setlength\titlebox{5.5cm}
%
% and set <dim> to something 5cm or larger.
\title{EmotionArcs: Emotion Arcs for 9,000 Literary Texts}


\author{Emily Öhman\textsuperscript{\textdagger} \\ 
    School of International Liberal Studies \\
    Waseda University, Japan \\
    {\tt ohman@waseda.jp}\\\And
    Yuri Bizzoni\textsuperscript{\textdagger} \\ 
    Center for Humanities Computing \\ Aarhus University, Denmark \\
    {\tt yuri.bizzoni@cc.au.dk} \\\AND
    Pascale Feldkamp Moreira\textsuperscript{\textdagger} \\ 
    School of Communication and Culture \\ Aarhus University, Denmark \\
    {\tt pascale.moreira@cc.au.dk} \\\And
    Kristoffer L. Nielbo \\ 
    Center for Humanities Computing \\ Aarhus University, Denmark \\
    {\tt kln@cas.au.dk} \\}

\begin{document}
\maketitle
\def\thefootnote{\textsuperscript{\textdagger}}\footnotetext{These authors contributed equally to this work}
\def\thefootnote{\arabic{footnote}}

\begin{abstract}
We introduce `EmotionArcs', a dataset comprising emotional arcs from over 9,000 English novels, assembled to understand the dynamics of emotions represented in text and how these emotions may influence a novel's reception and perceived quality. 
We evaluate emotion arcs manually by comparing them to human annotation and against other similar emotion modeling systems to show that our system produces coherent emotion arcs that correspond to human interpretation. 
We present and make this resource available for further studies of a large collection of emotion arcs and present one application, exploring these arcs for modeling reader appreciation.  
Using information-theoretic measures to analyze the impact of emotions on literary quality, we find that emotional entropy, as well as the skewness and steepness of emotion arcs, correlate with two proxies of literary reception. Our findings may offer insights into how quality assessments relate to emotional complexity and could help with the study of affect in literary novels.
\end{abstract}

\section{Introduction}
% Importance & problems in modelling emotions in lit texts
Sentiment analysis and emotion detection are subjective in nature, as not even humans can typically agree on which emotions any specific text contains \citep{campbell2004perception,bayerl2011determines}. There are also crucial distinctions between whether we are measuring the evocation or association of emotions and whether we are doing this from the reader's or the writer's perspective \citep{mohammad2016practical}. Approaches to sentiment analysis garner critique both for inherent problems in, for example, word-based annotation \cite{swafford2015problems}, but also for being overly focused on evaluation metrics over applicability to downstream tasks \citep{ohman_validity_2021} and how the task of emotion detection to some degree constructs the phenomena it is trying to measure \citep{laaksonen2022hate}. 
The importance of a literary text's emotional profile for its overall quality (``performance'', reception) is hard to overestimate \cite{bal2009narratology}. While literary narratives are far from being only matters of emotions, the emotions touched upon in texts -- in both explicit \textit{and} evocative ways -- determine essential aspects of the reader's experience %of the overall plot formation, 
at the structural and stylistic level \cite{mar2011emotion}. 
However, while this relation between emotions in literary texts and reader experience can seem relatively intuitive, it needs to be more obvious to test or quantify.
This presents us with a few difficulties. The first difficulty is the modeling of ``emotions in the text" -- defining what we mean by that, deciding which emotions to define, and how to measure the emotional content of any given textual unit - word, phrase, sentence, or paragraph. 
Due to the complexity of human readers' interpretations and experiences of texts, this is a difficult task to model. %For the process to yield statistical results, it would moreover be necessary to automate it enough to bring the annotation on a large number of texts.
The second difficulty is quantifying the relation between emotions in text and their reception or perceived quality of a literary narrative.
%we use emotion intensity lexicons enhanced by word embeddings fine-tuned for the domain of literature to construct emotion arcs that we link to the success of novels based on a 9,000+ book literary corpus. %We use over 9,000+ English-language novels to create a semantic vector space model which we then use to generate precise emotion arcs for all 9,000+ works of literature.
In this paper, we introduce a new resource, `EmotionArcs', to explore the relationship between these emotion arcs and literary quality complete with some early analyses. `EmotionArcs', is a dataset that comprises emotional arcs constructed from over 9,000 English novels through a novel approach that utilizes emotion intensity lexicons enhanced by word embeddings fine-tuned for the domain of literature to construct emotion arcs. We use the dataset to analyze and measure how affective language impacts a novel's literary quality, measured both through library holding numbers and GoodReads ratings. 

\section{Related Work}
Computational literary studies (CLS) is an active field of research affiliated with Digital Humanities and applied Natural Language Processing. Sentiments, emotions, and affect are all common research topics within CLS and include work in emotion classification, genre classification, story-type clustering, sentiment tracking, and character analysis \citep{kim_survey_2018}. %While methods in traditional literary studies are qualitative and observational, methods used in more CLS range from simple word count models to highly sophisticated approaches that utilize neural models that combine rule-based emotion lexicons with machine learning. Many studies have attempted to quantify the difference and accuracy of lexicon-based versus machine-learning models as used in sentiment analysis; although the results of such comparative studies often make claims of machine-learning being the superior approach \citep{van2021validity} other studies make the opposite claim \citep{taboada2011lexicon,teodorescu2022frustratingly}.% The fact of the matter is that it is very difficult to accurately compare the results from two very different approaches. With a machine learning-based model, it is very easy to extract f1 scores and compare predicted labels to actual labels, however, this is usually not possible or at the very least not as straight-forward with
%TODO: I might be getting a bit sidetracked here. -Emily

\subsection{Emotion Analysis}
Previous work has tested the potential of sentiment analysis \cite{alm2008affect, jain-etal-2017-sentiment} at the word \cite{vad-acl2018}, sentence \cite{mantyla_evolution_2018}, or paragraph level \cite{li2019exploiting}, for capturing meaningful aspects of  the reading experience \cite{drobot2013affective, cambria2017affective,kim_survey_2018, brooke2015gutentag, jockers2017syuzhet, reagan_emotional_2016}. 
Sentiment arcs have been used in multiple studies to model and evaluate narratives in terms of literary genre \cite{kim_investigating_2017}, plot archetypes \cite{reagan_emotional_2016}, dynamic properties \cite{hu2021dynamic}, narrative mood \citep{ohman2023affect}, and reader preferences and perceived quality \cite{bizzoni2022fractal}.
Previous work has tested the potential of sentiment analysis \cite{alm2008affect, jain-etal-2017-sentiment} at the word \cite{vad-acl2018}, sentence \cite{mantyla_evolution_2018}, or paragraph level \cite{li2019exploiting}, for capturing meaningful aspects of literary texts and the reading experience \cite{drobot2013affective, cambria2017affective,kim_survey_2018, brooke2015gutentag, jockers2017syuzhet, reagan_emotional_2016}.
%Sentiment arcs have been used in multiple studies to model and evaluate narratives in terms of literary genre \cite{kim_investigating_2017}, plot archetypes \cite{reagan_emotional_2016}, dynamic properties \cite{hu2021dynamic}, and reader preferences and perceived quality \cite{bizzoni2022fractal}.
%To model the sentiment arcs of novels \cite{jockers2017syuzhet}, studies have annotated sentiment in narratives, usually drawing sentiment scores of words or sentences from induced lexica \cite{islam2020domain}, or human annotations combined with machine learning \cite{mohammad2013nrc}, and challenges inhere to each approach \cite{da2019computational,ohman_validity_2021,teodorescu2023generating}. 
%-Emotion theories, Plutchik, NRC lexicons, Jockers/Syuzhet, Dodds \& Reagan/LabMT (for affective theory: Hogan, Bortolussi \& Dixon)
%The nature and origin of emotions is an active field of research with many competing schools of thought. Many approaches are based on the theory of universal emotions by \citet{ekman1971universals}, including Plutchik's wheel of emotions \citep{plutchik1980general}, and SenticNet \citep{cambria2012hourglass}. More recently \citet{cowen2021semantic} have tried to expand on these models using computational methods with promising results. Still, current resources are predominantly based on the theories of \citet{plutchik1980general} including the NRC Emotion intensity lexicon \citep{LREC18-AIL} used in this paper. 
Because literary texts have additional layers of affective meaning (cf. the distinction between tone and mood) at more narrative levels, (narrator, character, style, etc.) than other texts, additional challenges accompany annotating emotions in them. However, some recent papers have shown that lexicon-based methods can produce accuracies comparable to machine learning and transformer-based methods using chunks or bin sizes (a set number of tokens) of only a few hundred tokens with the additional benefit of transparency and human interpretability \citep{teodorescu2023generating, elkins_shapes_2022,ohman_validity_2021}. 

\subsection{Literary Quality}
Studies that aim to forecast the perception of literary quality by relying on textual features\footnote{In contrast to the study of \textit{extra-textual} features %contributing to the perception of quality 
\citep{verdaasdonk1983social, lassen_reviewer_2022}} have mostly depended on stylistic features. This includes factors like sentence length and readability \cite{maharjan_multi-task_2017, bizzoni_good_2023}, the proportion of different classes of words \cite{koolen2020literary, moreira_modeling_2023}, and the frequency of word pairs (n-grams) \cite{van2020results}. Other recent studies have explored the use of alternative textual or narrative elements such as 
sentiment analysis \cite{alm2008affect, jain-etal-2017-sentiment}, to model as a significant aspect of the reading experience \cite{drobot2013affective, cambria2017affective,kim_survey_2018, brooke2015gutentag, jockers2017syuzhet, reagan_emotional_2016}.
This strand of research predominantly focuses on sentiment valence with the aim of roughly modeling the sentiment arcs -- the ups and downs -- of novels \cite{jockers2017syuzhet}, but without taking into account  essential aspects like plot variability or the progression of the narrative. 
Once the arcs are computed, it is possible to cluster them based on similarities \cite{reagan_emotional_2016}. %between novels at various levels of complexity. 
For example, a simple sentiment arc clustering approach by identified six fundamental narrative arcs that they speculated might form the basis of narrative construction. 
More recently, \citet{hu2021dynamic} and \citet{bizzoni2022fractal} applied fractal analysis, a technique to study complex systems' dynamics \cite{hu_multifractal_2009}, to model the persistence, coherence, and predictability of sentiment arcs%. This approach aimed to assess the predictability and self-similarity of arcs and 
related to reader appreciation \cite{bizzoni_hc_andersen, bizzoni_nobels, bizzoni_sentimental_2023}. %In a similar vein, \citet{mohseni_fractality_2021} used fractal analysis on traditional stylometric and topical features to distinguish between canonical and non-canonical literary texts.
Systems to distinguish between different emotions have also been applied to study narratives \cite{somasundaran-etal-2020-emotion} and the aesthetics of literary works \cite{haider2020po}. \citet{Maharjan} modeled the ``flow of emotions" in literary texts using the NRC lexicon, showing that the shape of emotion-specific arcs had an effect on predicting whether books were successful (based on GoodReads ratings). The distribution of emotions seemed particularly telling for the ``success'' of a work, as \citet{Maharjan} found emotion intensity and variation (std. deviation) higher for successful than for unsuccessful works. As it has been shown that emotion distribution and levels may vary across genres \cite{mohammad_once_2011}, it is particularly interesting for us to continue this line of assessing the importance of the shapes of emotion-specific arcs on quality perception, in our case examining novels only. 


%Another popular tool is Mathew Jockers' Syuzhet library \cite{Syuzhet1} which contains the Syuzhet lexicon, which is an English sentiment dictionary developed with literary analysis in mind. Syuzhet was criticized for 1: sentence detection, 2: word-value mapping, and 3: filtering \cite{swaford1}. 1 and 2 are mostly related to state of the art of natural language processing \cite{jockerslwopass1}, and can be partially remedied today by not relying on the Syuzhet library for segmentation and analysis. 3 however is more interesting and relates fundamentally to the question of how to extract a time series that reflects the narrative structure. Dodds (2015), for instance, used a custom dictionary to estimate happiness time series of novels in windows of 10.000 words and analyzed the raw signal \cite{dodds}. Jockers (2015) also used a windowing approach with the Syuzhet dictionary and smoothed the time series using a low pass filter to extract and compare abstract (or latent) story arcs \cite{Syuzhet2}. Whether this approach results in artifacts has been a matter of debate \cite{swaford1, jockerslwopass1, jockerslwopass2}, but more recently, Gao (2017) has shown that nonlinear adaptive filtering is more appropriate to studying the dynamic multi-level sentiment evolution of novels \cite{sentiment_Hurst}
%\textbf{Quality Perception Summaries}\\
%\textbf{Emotion Dynamics Modelling}\\

\section{Dataset Construction}

\subsection{Selecting and Curating Novels}

Our data comes from the ``Chicago Corpus''. This corpus consists of 9,089 novels published in the US between 1880 and 2000, making it an unusual collection for both size and modernity, as it contains both more and more recent novels than the works available on most other platforms. \footnote{On average, studies on literary quality and success tend to rely on collections of tens to hundreds of novels, i.a., \cite{ashok2013success}.}. 
The corpus was compiled based on the number of libraries holding numbers worldwide, with a preference for more circulated works. It features works by Nobel laureates (i.e., Ernest Hemingway, Tony Morrison), widely popular works, and ``genre literature'', from Mystery to Science Fiction (e.g., from Agatha Christie to Philip K. Dick) \cite{textlabscorpus}.\footnote{Other quantitative studies are based on this corpus \citep{underwood2018transformation, cheng2020fleshing}, which can be viewed at \url{https://textual-optics-lab.uchicago.edu/us_novel_corpus}.}
%The use of more commonly available or ``popular" books also means that the novels are more likely to be reviewed on tertiary platforms such as GoodReads, allowing us to better examine correlations between public reception of novels and their affective content.
The use of more commonly available or ``popular" books also means that the novels are more likely reviewed on tertiary platforms such as GoodReads, which facilitates the examination of correlations between public reception and novels' affective content. %Additionally, by dealing with fairly modern works, the semantic shifts inherent in language evolution are not as problematic.
%One important limitation of the corpus is, however, its strong Anglophone and American tilt: there are few non-American and non-Anglophone authors, which inevitably situates the entire analysis within the context of an “Anglocentric” literary field. 
The dataset consists of 1,108,108,457 tokens, ranging from 246 tokens to 723,804 tokens per book with an average of 121,918 tokens per book. For parts of our analysis, we split the books into bins each containing 500 tokens, which means there are on average 244 bins per book. We chose a 500-bin size for both practical and theoretical reasons. Multiple studies have shown that using bin sizes of just 200-300 tokens can beat state-of-the-art machine learning models in accuracy \citep{teodorescu2023generating,ohman-validity-2021,ohman2023affect} 
%which is why we use bin sizes of more than 300 tokens. 
Using too large bin sizes, on the other hand, could misrepresent and muddle the emotion arcs. We determined 500 tokens, roughly corresponding to text subsets that are 1-2 paragraphs in length, to be suitable in order to strike a balance between theory, interpretability, and practice. %as that would. 
Note that the token count will be much higher than the word count of the same text. This is especially true for literary texts which tend to have dialogue with quotation marks, dashes, and more punctuation marks all of which count as individual tokens. % as many other types of texts.

\begin{figure}[ht]
     \centering
    \includegraphics[width=7cm]{figures/oldmanemotions.png}
     \caption{Emotion intensities for Hemingway's \textit{The Old Man and The Sea}. For instance, the prevalence of trust might mirror the Santiago-Manolin relation and be a proxy for the protagonist's endurance.}
     \label{fig:oldman}
 \end{figure}

\subsection{Affective Word Embeddings}

We utilize the NRC Affect Intensity Lexicon \citep{LREC18-AIL} for emotion labels %. We chose this lexicon since 
as it is the most extensive emotion intensity lexicon we are aware of. Moreover, both it and its sister lexicon EmoLex \cite{mohammad2013nrc} have been used in countless emotion detection tasks and have proven their accuracy and usability in a variety of tasks. This lexicon was created with the help of human annotators using best-worst scaling. It contains 9,829 lexemes with at least one emotion association and a value between 0 and 1 for each emotion to represent the intensity of the labeled emotion. The emotions included are \textit{anger}, \textit{anticipation}, \textit{disgust}, \textit{fear}, \textit{joy}, \textit{sadness}, \textit{surprise}, and \textit{trust}. As this lexicon is not specific to the domain of literary texts, we used the novels in our dataset to create a semantic vector space model with Word2Vec \citep{mikolov2018advances} and then with the aid of cosine similarity measures expanded the lexicon to make it more domain-specific.
Cosine similarity is a commonly used measurement to determine the similarity between two objects, in this case, lexemes, represented as vectors. %The cosine is calculated as the normalized dot-product of the two vectors. The closer to 1 the cosine value is, the more similar the vectors are, a value of 0 represents orthogonal vectors. 
For vectors a and b we can represent cosine similarity as follows: 
%COSINE
%Using the math environment over the equation one, gives us roughly 1 paragraph extra room. Change back to equation if not necessary.
\begin{math}
    \cos ({\bf a},{\bf b})= {{\bf a} {\bf b} \over \|{\bf a}\| \|{\bf b}\|}
\end{math}
As there has been some criticism of using cosine similarity for similarity measures of high-frequency words \citep{zhou-etal-2022-problems}, we also conducted manual evaluations of the newly added terms to ensure the appropriateness of the modifications. The lexicon was checked for unsubstantiated emotion associations and the lemmas in the novels for words that have an emotion association but were not in the lexicon. 
Following this procedure, we created \textit{emotion intensities} for the whole novels (e.g., see Fig. \ref{fig:oldman}) as well as for each 500-token bin. For the former, the results were normalized by word count; for the latter, the results were simply sums of the word-emotion association intensities. These intensity calculations are available publicly\footnote{https://github.com/yuri-bizzoni/EmoArc}. 

\section{Agreement and Validation}

% Typically sentiment analysis is considered a supervised classification task, and therefore the most common validation methods tend to use f1 scores for accuracy measurements of predicted labels versus actual labels.%, sometimes including precision and recall measures, which are very useful when there is a subset of the data that can be used for such evaluation measurements. 
% When sentiment analysis is not a classification task, such as when using simple emotion association, i.e. whether an emotion is present or not, it can be possible to use similar evaluation metrics for lexicon-based methods as well, however, even in such cases one is usually limited to comparing lexicon-based emotion scores with manual annotations of the target text. If the base unit of evaluation is a sentence or a paragraph, a human would perhaps be able to simply mark that paragraph as containing a specific emotion, however, the lexicon match might indicate multiple matches skewing the comparison metrics. Humans are notoriously bad at rating scales \citep{kiritchenko-mohammad-2016-capturing} and thus this approach is already problematic and unlikely to offer any true indication of the comparability of evaluations. The incompatibility of the traditional evaluation methods only increases when using emotion intensity over mere presence as this amplifies the difficulty of rating on a scale.


% Nonetheless, evaluation is a crucial part of any quantitative methodology and is required for the demonstration of the validity of the results. We, therefore, examine a selection of novels by comparing the emotion arcs with human judgment to validate the accuracy of our methods \citep{jockers2016more}. %ADD back later: ohman2023affect

\begin{figure*}
    \centering
    \includegraphics[scale = .3]{figures/w_roberta.png}
    \caption{\textit{The Old Man and the Sea}, manual evaluations, EmotionArcs (fear minus joy), and RoBERTa. Arcs were smoothed using adaptive filtering \cite{gao_denoising_2010}.}
    \label{fig:maneval}
\end{figure*}

% \subsection{Human Validation}
As the approach used in this project does not allow for traditional accuracy measures often used in machine learning \citep{ohman_validity_2021}, 
we focus our validation efforts on comparing human interpretations with those generated by our lexicon-based model, which has shown to be accurate in multiple prior studies \citep{teodorescu2023generating,ohman2023affect,koljonen2022strategic}. %\citet{koljonen2022strategic} showed that emotion intensity was much more congruent with human interpretations of affective content compared to a simple binary lexicon-based approach especially when looking at the paragraph level and beyond.
We validated the EmotionArcs resource in three different ways: 


(i) Two literary scholars inspected the emotion arcs of select novels, relating style and narrative events to the shapes of emotion arcs. One example of a manual annotation of the correspondence of our emotion arcs with narrative events is shown in Figure \ref{fig:annotated_arc} (see another in the Appendix). Note that while at first sight, the co-occurrence of peaks in fear and joy (especially from chunk 80 on) may appear puzzling, it illustrates an important aspect of Hemingway's style in describing complex emotions and reflects the themes of the story overall: in moments of crisis and violence, Hemingway's protagonist still reflects on the natural beauty and his love for the sea. This creates a mix of complex feelings in key scenes (love and hatred, fear and admiration) so that intensities in these feelings co-occur (see, e.g., box 7 in Fig. \ref{fig:annotated_arc}), which is also a token of the protagonist's endurance and optimistic outlook on life. The slope and generally high levels of trust in the story also follow the progression of narrative events (see, e.g., box 5 in Fig. \ref{fig:annotated_arc}).


(ii) We randomly selected 11 passages from one novel: \textit{The Old Man and the Sea}, asking 20 non-expert volunteer annotators to indicate, for each passage, which emotions were present from a pre-defined set and at what intensity on a 0-1 scale (for the agreement between model and annotator scores, see the Appendix). All passages received 3 to 6 annotations. After a first independent round, annotators were provided with the EmotionArcs scores for the same passage and asked whether they thought the model scores were present in the text, and whether %If they were, they should indicate whether they found the value assigned by the model correct or whether it 
they should be lower or higher \citep{moreira_not_2023,bizzoni_comparing_2023}
Our annotators vastly agreed with the model's categorical choice (see Table \ref{tab:agreement}), while agreement on their intensity varied. In 225 over 232 cases, annotators assessed that the model chose the correct emotions for the text. In 122 of these cases, the annotators also agreed with the intensity score assigned. Of the remaining 110 cases, 43 were given the assessment “could be higher” and 60 the assessment “could be lower”.

\begin{table}[ht]
    \centering
    \small
    \begin{tabular}{l|c|c|c|c}
        \toprule
        & \multicolumn{3}{c|}{\textbf{Agree}} & \textbf{Disagree} \\
        \cline{2-5}
        & Higher & Lower & Correct & \\
        \midrule
        \textbf{Count} & 43 & 60 & 122 & 7 \\
        \midrule
        \textbf{Total} & \multicolumn{3}{c|}{225} & 7 \\
        \bottomrule
    \end{tabular}
    \caption{Annotators' agreement with EmotionArc's scores}
     \label{tab:agreement}
\end{table}

\textit{Joy} was the emotion that elicited most ``could be lower" responses. We believe this is because in Plutchik's eight core emotions \citep{plutchik1980general}, \textit{joy} is the only genuinely positive one. %Hence, if an annotator %  for the original Emotion Lexicon annotations associates a word with positive valence, \textit{joy} may be the only option, even if the label is inadequate. %it is not specifically \textit{joy} that is the origin for the positive valence. 


(iii) Lastly, two novels were selected for close-reading evaluation. We evaluated the EmotionArcs by comparing their scores to valence scores produced by an independent (RoBERTa, fine-tuned for sentiment analysis on tweets \citep{barbieri-etal-2022-xlm}\footnote{Note that to convert RoBERTa's categorical output we used the confidence score of labels as a proxy for sentiment intensity. If the model classifies a sentence as \textit{positive} with a confidence of, for example, 0.89, we interpret it as a valence score of +0.89, and so on. Scores of the \textit{neutral} category were converted to a score of 0.0. For further details SA with Transformers, see \citet{bizzoni_comparing_2023}.}) and average human annotations for valence of the same books.\footnote{Human annotators (n=2) read from beginning to end and scored sentences on a 1 to 10 valence scale.} As emotion annotation has been shown to correlate with valence scores -- most notably joy and fear with positive and negative valence \cite{moreira_not_2023} -- we combined the emotion scores of joy and fear of our method to model arcs of novels, comparing them against the SA and human annotation of the same novel.
%Another ``proof of concept" for the validity of the emotion annotation system is that we can reproduce an independent sentiment analysis scoring by combining our emotion score. 
An example can be seen in Figure \ref{fig:maneval}, where human evaluation closely follows that of our model's \textit{joy} values minus \textit{fear} value as well as that of additional validation produced with the help of RoBERTa scores.
In other words, by combining the most prevalent positive emotion and the most prevalent negative emotion, with a positive and negative sign respectively, it's possible to reproduce a novel's sentiment trendline
\footnote{We focus our comparison on \textit{joy} and \textit{fear} as they are among the most frequent in text and we see them as the purest representatives of unambiguous valence in the available categories and highly representative of overall valence due to the overall overlap of emotions with \textit{fear} and \textit{joy} \citep{bizzoni_comparing_2023,ohman2020challenges,ohman2020emotion}.}


\subsection{Agreement in Emotions}

Certain emotions are more likely to co-occur than others. This can lead to lower accuracy scores in multilabel machine learning models when the features of correlated emotions are muddled, but increased detail in lexicon-based models when we can differentiate better between closely related associations. Figure \ref{fig:enter-label} shows the correlation of emotions in the entire `EmotionArcs' corpus. The negative emotions \textit{anger}, \textit{disgust}, \textit{fear}, and \textit{sadness} show a high rate of co-occurrence as expected, while \textit{joy} is negatively correlated with both \textit{anger} and \textit{fear} and positively so with \textit{anticipation} and \textit{trust}. \textit{Anticipation} strongly correlates not only with \textit{joy}, but also with \textit{trust} \footnote{\textit{trust} is commonly associated with its negative counterpart \textit{distrust}, which is not a label in Plutchik}, an emotion of more ambiguous valence. \textit{Anger} correlates significantly also with \textit{surprise}. It stands to reason that a passage expressing \textit{anger} can be framed as sudden, surprising and, even cathartic.\footnote{Plutchik considers \textit{anger} a positive emotion, counter to how it is used in most NLP models, and it is not immediately clear whether the valence in a literary setting should be reversed from its psychological roots as is standard practice \citep{plutchik1980general,ohman2021language}.}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=8cm]{figures/corr_all.png}
    \caption{Correlation between emotions in all emotion arcs}
    \label{fig:enter-label}
\end{figure}


\begin{figure*}
    \centering
    \includegraphics[width=\textwidth]{figures/annotated_hemingway.png}
    \caption{Arcs of \textit{The Old Man and the Sea} annotated for narrative events.}
    \label{fig:annotated_arc}
\end{figure*}

\section{Quality Proxies}
\subsection{Rationale}
The idea that the distribution and dynamics of the emotions expressed in a text are related to the reception of that text is widespread, and several studies have used both sentiment analysis and emotion detection to %underscore the importance of emotional words as 
capture meaningful aspects of the reading experience %in the "effect" of the text 
\cite{drobot2013affective, cambria2017affective,kim_survey_2018, brooke2015gutentag, jockers2017syuzhet,ohman2023affect}.
In this work, we tried several different resources that approximate the reception of a novel -- specifically, its perceived overall quality -- by either a large number of lay readers (crowd-based proxies) or a small number of expert readers (expert-based proxies).

\subsection{Expert-based and crowd-based proxies}

%Expert-based judgments on literary works come from a small but highly knowledgeable group of readers. These types of authorities on literary quality can be, for example, editors or publishers \cite{karlyn_chadwyck-healey_nodate, vulture_canon_premature_2018} as used in \cite{porter_popularityprestige_2018, algee-hewitt_canonarchive_2016}, single literary scholars \cite{bloom_western_1995}, as in \cite{mohseni_approximate_2022}, or teams behind anthologies like the Norton Anthology, as well as committees of literary awards like the Nobel \cite{bizzoni_nobels} or the Pulitzer prize \cite{manshel_who_2019}. The argument for using quality proxies based on small committees is that trained readers can better judge a literary text than a lay reader.
%We selected several prestigious literary awards for so-called general and genre fiction, such as the Pulitzer Prize, the Hugo Award, and the Nobel for Literature. Similarly, we looked into whether the books in our collection appeared in anthologies, classic book series, and canon lists. %, contributing to the formation of a literary canon.


Expert-based judgments of literary works originate from a limited group of expert readers, such as editors, publishers \cite{karlyn_chadwyck-healey_nodate, vulture_canon_premature_2018}, individual literary scholars \cite{bloom_western_1995}, and award committees like the Nobel prize. Crowd-based judgments, on the other hand, are formed by a large number of readers without a given literary expertise, and offer more inclusivity and statistical robustness. %However, these judgments often reflect the views of "lay readers" who may not have given due attention to the book. 
GoodReads, a social readership platform with over 90 million users, provides insight into such crowd-based judgments \cite{maharjan_multi-task_2017, bizzoni_hc_andersen, jannatus_saba_study_2021, porter_popularityprestige_2018} and especially into reading culture ``in the wild'' \cite{nakamura_2013}, as it catalogs books from different genres and derives ratings from a heterogeneous pool of readers \cite{kousha_goodreads_2017}. There are various issues with using GoodReads' ratings as a metric, among others, how this heterogeneity is conflated into one single score (0-5) that takes no account of differential rating behavior, for example across genres. Beyond the rating or ``stars'' on GoodReads, another option is to use the rating count itself as a proxy of quality perception, supposing that more frequently rated titles are also more popular and liked. 
There are also less clear-cut, more nuanced measures of literary reception. For example, a conceptually hybrid measure between crowd- and expert-based is the number of libraries holding a given title worldwide, as indicated on WorldCat \cite{bennett2003concept}. Expert choice and user demand may influence what titles are acquired by libraries, and since the libraries are many, the compound nature of all title selections approximates crowd-based judgment.


In this work, we selected the latter two proxies: for each book, we collected the number of ratings of GoodReads (as of December 2022) and libraries holdings of the title.\footnote{Note that in our corpus, library holdings and rating count are correlated with a coefficient of 0.50 (p$<$0.01) using a simple Spearman correlation.}
%In our corpus, library holdings and rating count are correlated with a coefficient of 0.50 (p$<$0.01) using a simple Spearmann correlation.

%Similarly, a book's number of translations indicates a degree of market success (if translation is seen as a token of publishers seeking to expand good sales outside the national market) and canonicity or durability (if translation is seen as a token of a work's durable popularity or cultural longevity). Yet also the sheer survival of a book through many generations of readers (the so-called historical selection) is another form of hybrid assessment that falls somewhere between the expert and the crowd. 
%In-between measures used in this study are the number of libraries holdings and the number of translations of a work into other languages.\footnote{Indicated in the \textit{Index Translationum} worldwide database of translations published 1979-2002.}

%moved to limitations
%\textbf{Demographics}\\
%It is hard to control the demographics of each proxy for literary quality and reception. Generally, sources like GoodReads are more diverse and represent a more comprehensive demographic selection than awards committees or anthologies' editorial boards. Beyond this, it is not easy to draw generalizations, as the demographic compositions of awards committees can change quickly, and so do librarians and translators. 

% \subsection{Quality clusters}
% The question of whether literary quality is an entirely subjective and idiosyncratic concept or not is a question that has been debated for a millennium. Proxies for reception and appreciation of novels however, indicate observable groupings that while not unambiguous, form ``clusters'' around resources. In other words, the quality judgments do not appear wholly orthogonal but rather `converge' on large numbers in groups of agreement. For example, the number of ratings from GoodReads shows a slight correlation with the number of libraries' holdings: 

% % Overall, we identify two significant clusters of partial agreement between our resources, one comprising ... and one comprising ... A complete table of correlations is in the Appendix. 





\section{Data Analysis}

\subsection{Emotion Distribution}
Building on previous work \cite{Maharjan}, we examine the association between the emotional content of novels and their perceived quality, we examined the \textbf{overall intensity} of the eight emotions in each novel. As noted, intensity values were length-normalized to ensure comparability across texts of different sizes. To understand the variation in emotions in each novel, we computed the \textbf{entropy} of their emotion intensity distribution. In our context, the concept of entropy serves as a measure of the uncertainty of emotional intensities in novels: a low entropy value indicates that one emotion may dominate the text, being reliably more intense than other emotions. Conversely, high entropy indicates a more diverse emotional profile, where each emotion is represented with comparable intensity. In Fig. \ref{fig:oldman} the emotional profile of \textit{The Old Man and the Sea} appears to have medium-high entropy.

\subsection{Emotion Trends}

Building on work examining the shape and dynamics of narrative arcs \citep{bizzoni_hc_andersen,ohman2023affect,moreira_not_2023}, we relate the linear shapes of the eight emotion arcs to quality perceptions, computing the \textbf{skewness}  and \textbf{slope} steepness %and slope entropy (?)
of each emotion arc; as a score for each emotion separately and as the average score of all eight emotions per novel. %The scores represent the overall shape of the emotions in novels. 
The slope value for each emotion is retrieved by linear regression and represents the development in intensity of that particular emotion across the narrative: if the joy arc increases or decreases linearly across a novel, the slope of its joy arc will be relatively steep \cite{su2012linear}. Skewness captures the symmetry of an emotion arc: an arc with few large values or intensities but many small values is positively skewed, while an arc with an even distribution of large and small values has a skewness approximating 0 \cite{kokoska2000crc}.


%\subsection{Relation with GoodReads and libraries}

\subsection{Overall novel emotion}
The intensity of most emotions appears to hold a correlation with the number of library holdings, but a weak one.
There is also a weak negative correlation between library holdings and the overall entropy of the emotional values of a text (Table \ref{tab:correlations}). 

\begin{table}[ht]
\centering
\small
\begin{tabular}{lcc}
\hline
\textbf{Emotion} & \textbf{Coefficient} \\
\hline
Fear (sum)& 0.14 \\
Sadness (sum)& 0.14 \\
Anger (sum)& 0.14 \\
Disgust (sum)& 0.13 \\
Anticipation (sum)& 0.13 \\
Surprise (sum)& 0.13 \\
Joy (sum) & 0.12 \\ \hline
%Sadness (slope) & -0.10 \\
Entropy (all emotions) & -0.12 \\
\hline
\end{tabular}
\caption{Emotion intensities correlation with library holdings (Spearmann). For all correlations, p $<$ 0.01.}
\label{tab:correlations}
\end{table}

 \begin{table}[ht]
\centering
\small
\begin{tabular}{lll}
\hline
\textbf{Variable} & \textbf{rating count} & \textbf{libraries} \\
\hline
mean skewness > 500 & 0.60* & 0.50* \\
mean skewness < 100 & -0.55* & -0.41* \\
mean slope inclination > 500 & -0.81** & -0.71* \\
mean slope inclination < 100 & 0.83** & 0.69* \\
mean entropy > 500 & 0.76* & 0.29 \\
mean entropy < 100 & -0.69* & -0.63* \\
\hline
\end{tabular}
\caption{Correlations of emotion arc features with reception proxies (Spearman correlation). *p $<$ 0.05, **p $<$ 0.01}
\label{tab:correlations_proxies}
\end{table}

\begin{table*}[ht]
\centering
\small
\begin{tabular}{l|c|c|c|c|c|c|c|c}
\toprule
 & \textbf{Joy} & \textbf{Anger} & \textbf{Sadness} & \textbf{Fear} & \textbf{Disgust} & \textbf{Surprise} & \textbf{Trust} & \textbf{Ant.} \\
\midrule
\textbf{Rating count >500} & -0.656** & -0.861** & -0.560* & -0.694** & -0.686** & -0.809** & -0.764** & -0.667**\\
\hline
\textbf{Rating count <100} & 0.652** & 0.886** & 0.776** & 0.721** & 0.772** & 0.765** & 0.737**& 0.589 **\\
\hline
\textbf{Holdings >500} & -0.938** & -0.953** & -0.913** & -0.885** & -0.875** & -0.835** & -0.839**& -0.794** \\
\hline
\textbf{Holdings <100} & 0.935** & 0.930** & 0.749** & 0.885** & 0.782** & 0.617* & 0.757**& 0.725**\\
\midrule
\textbf{Rating count >500} & 0.272* & 0.068 & 0.288** & 0.019 & 0.309** & 0.453** & 0.548**& -0.774*\\
\hline
\textbf{Rating count <100} & -0.272* & 0.020 & -0.199* & -0.020 & -0.151 & -0.516** & -0.550**& 0.662* \\
\hline
\textbf{Holdings >500} & 0.035 & 0.136 & 0.347** & 0.247* & 0.308** & 0.427** & 0.332** & 0.92*\\
\hline
\textbf{Holdings <100} & 0.047 & -0.188* & -0.324** & -0.138 & -0.233** & -0.527** & -0.477** & 0.93* \\
\bottomrule
\end{tabular}
\caption{Correlation of the emotion arcs' slopes (rows 1-4) and skewness (rows 5-8) with Rating Count and libraries' holdings for both >500 and <100 values. Asterisks reflect p-value: * p<0.05, ** p<0.01.}
\label{tab:correlations_per_emotion}
\end{table*}

 
Yet it seems that the distribution of the data is unfit for standard correlation, as the relation between emotion entropy and library holdings and GoodReads rating count is not linear. Different populations have different distributions: one group of titles with relatively low rating count and low library holdings is present at almost every level of entropy, while a group of titles with increasingly high rating count and library holdings cluster in a subset of the space. %This hill-like, skewed distribution is not measured well by simple correlation. 

\begin{figure}[ht]
     \centering
    \includegraphics[width=5cm]{figures/visual.png}
    \caption{Distribution of library holdings with respect to emotion entropy.}
    \label{fig:visual}
 \end{figure}
 
To account for this hill-like distribution, we divided our data into two groups: one with low and the other with high rating counts (RC) and library holdings (LH), setting a threshold of ratings and library holdings at below 100 or above 500.\footnote{Number of books in each goup: RC$<$100 = 2978, RC$>$500 = 4340; LH$<$100 = 2206, LH$>$500 = 3464} While these thresholds of 100 and 500 are somewhat arbitrary, they represent relatively robust trends in the data that can be reproduced with different cutoff points (see Fig \ref{fig:slopes_concept} for the effect of different upper thresholds).

\begin{figure*}[ht]
    \centering
     \includegraphics[width=1\textwidth, height=7cm]{figures/thresholds.png}
     \caption{Trends in the probability of being in the high- or low-rating group at different cutting points of emotion slope value. While 100 and 500 rating counts and library holdings are somewhat arbitrary thresholds, trends in our data are reproduced at different cutoff points. }
     \label{fig:slopes_concept}
 \end{figure*}
 
With this separation of marginally more ``successful'' and ``unsuccessful'' groups of titles, the relation between emotion entropy and quality perception is more evident: negative correlations of emotion entropy and the quality proxies continue only up to a certain entropy value, before which there is even a positive correlation between entropy and library holdings; and when looking at rating count, the correlation is almost completely positive. In general, it seems that titles with higher entropy of emotions receive a higher number of ratings and -- up to a point -- are held in more libraries (see Fig.\ref{fig:barplots}).

\subsection{Emotion arcs}

Using the same groupings of high and low rating and library holdings titles, we examined the correlation between our quality proxies and the average slope intensity, as well as the average skewness of arcs, averaging the values across slopes and skewness of each of the eight emotion for each title. Again, by grouping before correlating, we find a correlation between quality proxies and arc shape. It seems that more novels with more sloping arcs are rated less often and are held in fewer libraries; and where novels with more skewed emotions seem to have more ratings and library holdings (Table \ref{tab:correlations_proxies}). 
We similarly correlated slopes of each emotion in a novel, as we might suppose that titles (or even genres) exhibit a steep slope for one emotion (not for others), making the mean unrepresentative. Here, we find that the average patterns represented in Table \ref{tab:correlations_proxies} hold for almost any single emotion: titles above 500 ratings and library holdings correlate negatively with slopes, and the reverse is true for titles below 100 ratings and library holdings, while the opposite appears true for skewness (Table \ref{tab:correlations_per_emotion}). 


% \begin{table*}[h]
% \centering
% \begin{tabular}{lll}
% \hline
% \textbf{Variable} & \textbf{rating count} & \textbf{libraries} \\
% \hline
% mean skewness > 500 & 0.60* & 0.50* \\
% mean skewness < 100 & -0.55* & -0.41* \\
% mean slope inclination > 500 & -0.81** & -0.71* \\
% mean slope inclination < 100 & 0.83** & 0.69* \\
% mean entropy > 500 & 0.76* & 0.29 \\
% mean entropy < 100 & -0.69* & -0.63* \\
% \hline
% \end{tabular}
% \caption{Correlations of emotion arc features with quality proxies (Spearman correlation). *p $<$ 0.05, **p $<$ 0.01}
% \label{tab:correlations_proxies}
% \end{table*}

\begin{figure*}[t]
  \centering
  \subfigure[Titles below 100 or above 500 holdings.]{
    \includegraphics[width=0.48\textwidth]{figures/entropy_lib_bar.png}
    %entropy_mean_emotions.png}
    \label{fig:plot1_entropy}
   }
   \hfill
   \subfigure[Titles below 100 or above 500 ratings.]{
    \includegraphics[width=0.48\textwidth]{figures/entropy_rat_bar.png}   
    %entropy_mean_emotions_GR.png}
    \label{fig:plot2_entropy}
   }
   \caption{Probability of having high/low number of library holdings or Goodreads ratings (below 100/above 500) at different values of emotional entropy. All probabilities were computed on populations of at least 10 different titles. The relation with the number of libraries' holdings might point to a "sweet spot".}
   \label{fig:barplots}
 \end{figure*}



\begin{figure*}[t]
  \centering
  \subfigure[Titles below 100 or above 500 holdings.]{%
    \includegraphics[width=0.48\textwidth]{figures/mean_slope_instensity_lib_poly.png}
    \label{fig:plot1_slope}
  }
  \hfill
  \subfigure[Titles below 100 or above 500 ratings.]{%
    \includegraphics[width=0.48\textwidth]{figures/mean_slope_intensity_rat_poly.png}
    \label{fig:plot2_slope}
  }
  \caption{Probability of having high/low library holdings or high/low number of GoodReads' ratings (below 100/above 500) at different levels of average slope steepness. All probabilities were computed on populations of at least 10 different titles. }
  \label{fig:scatterplots}
\end{figure*}



\section{Concluding Discussion}

With `EmotionArcs' we have presented a new resource for the study of emotions in literary novels that we hope will enable many other researchers to investigate how affect in literary works is intertwined with other aspects of literature. We have shown that our method produces reliable, useful, and easily interpretable emotion arcs that can help more traditional literary scholars compare larger corpora of literary works that are possible using only qualitative methods.
It seems that overall emotional entropy, the slopes of emotion arcs, and their level of skewness hold some relation with the reception of the novels as measured via rating count and library holdings. %In synthesis, we observe that:


(i) \textbf{Entropy}. A novel with higher emotional entropy will have an overall higher probability of being rated more than five hundred times on GoodReads. The same holds for its likelihood of being held in a large number of libraries -- up to a point: ``too much entropy'' is related to lower circulation in libraries.


(ii) \textbf{Slope}. A novel with steeper overall emotion arcs will have an overall lower probability of being rated more than five hundred times on GoodReads or being held by more than five hundred libraries; conversely, it will have an increased probability of being rated less than 100 times and held by less than 100 libraries. 


(iii) \textbf{Skewness}. A novel with a low level of overall emotion skewness will have an overall lower probability of being rated more than five hundred times on GoodReads or being held in more than five hundred libraries; conversely, it will have an increased probability of being rated less than 100 times and being held by less than 100 libraries. 
% \begin{enumerate}
%     \item \textbf{Entropy}. A novel with higher emotional entropy will have overall a higher probability of being rated more than five hundred times on GoodReads. The same holds for its likelihood of being held in a large number of libraries -- up to a point: ``too much entropy'' is related to lower circulation in libraries. 
%     \item \textbf{Slope}. A novel with steeper overall emotion arcs will have overall a lower probability of being  rated more than five hundred times on GoodReads or being held by more than five hundred libraries; conversely it will have a increased probability of being rated less than 100 times and held by less than 100 libraries. 
%      \item \textbf{Skewness}. A novel with a low level of overall emotion skewness will have overall a lower probability of being rated more than five hundred times on GoodReads or being held in more than five hundred libraries; conversely it will have a increased probability of being rated  less than 100 times and being held by less than 100 libraries. 
% \end{enumerate}
Our results on entropy might bear a relation to \citet{jautze_topic_2016} regarding topics: novels with relatively few, dominating topics are perceived as being less good than novels that use a larger topical palette. There may be a similar effect at the level of the emotions represented in a text. It is important to remember that we are talking about fine-grained emotions: a novel with a high level of fear does not necessarily correspond to a narrative where characters are constantly scared. Rather, because of its selection of certain events, a text may be more likely to sample from an emotional vocabulary of fear than from that of another emotion. 
Something similar might be inferred from the slopes' steepness and skewness: excessively predictable and smooth emotion arcs might not create as effective a reader experience. This interpretation is corroborated by studies that have found that readers tend to prefer fractal story arcs but only with a moderate level of coherence \cite{hu2021dynamic, bizzoni_hc_andersen}. Story arcs that monotonically focus on one emotion or have a very steep slope will either be overly predictable, and by extension overly coherent, or, at some point, too unpredictable and locally incoherent.   

Finally, in addition to a novel resource, the methods used in this study offer simple and robust tools that should be a part of any lexicon-based emotion projects. We strongly believe our methodology of fine-tuning existing lexicons to be more domain- and period-specific with the help of affective word embeddings should be the first step in any sentiment analysis or emotion detection task that utilizes lexicons as it not only makes the lexicons more attuned to the specific domain at hand but also increases precision and recall in general and can even negate some of the effects of semantic shifts in language.

%We have also presented some early studies on the relation between the emotion profile of the novels and their reception and perceived quality, using two proxies - library holdings and number of GoodReads' ratings. 
In the future, we aim to continue with similar projects further improving and enhancing the lexicon and extending the use cases of emotion arcs to, e.g., the exploration of narrative structure and differences in affective language used by individual authors and across genres. Emotional expectations are likely to vary greatly across genres and might yield further insight into the relation between affect, mood, and reception. We also aim to experiment with different proxies for perceived literary quality, including more expert-based resources such as canon lists and prestigious awards. %Combining the emotion arcs with other essential aspects of literary texts such as their stylistic and topical profile would also be a fruitful base for future studies. 
Finally, we intend to combine our emotional arcs with more sophisticated modeling techniques for fractal analysis and time series forecasting in order to have a more complex view of the relation between the textual representation of emotions and reader experience. 

\section*{Limitations}
%DOES NOT COUNT TOWARDS PAGE COUNT
As emotion annotation is a notoriously difficult task, this study has attempted to make the process as robust as possible, regardless, emotions are always subjective and difficult to measure. Emotions are also partly constructed by the measuring process itself and therefore always a reflection of the methods used \citep{laaksonen2022hate}. %Furthermore, we acknowledge that we are to some extent creating these emotions as we are trying to model them as any application of sentiment analysis techniques to investigate emotional expression in texts amounts to an effort of constructing sentiment measurements – a process essentially driven by judgments made by researchers in an attempt to reconcile diverging conventions and conceptions of research methods.
Methodologically, the choice of lemmatization, and to a lesser extent other preprocessing steps, affects how the semantic vector space is constructed and how words match the affective space. Although English is a comparatively easy language to lemmatize, there were instances of lexemes in the data that could have been further broken down. % matches and there are many competing lemmatizers out there, how to choose the best one\\

Word embeddings are inherently contextual, however, they are not immune to polysemy, particularly when used with a hybrid lexicon-based approach. We reduced the effect of polysemous words and other similar artifacts with our iterative approach, however, it is unlikely we were completely able to remove the effects of semantic shifts or cultural biases that occur in language and stem from the original annotations of the NRC lexicon as well as the diverse nature of the data. Ultimately, unlimited iterations are possible, and we made a balanced choice between feasibility, time, cost, and practicality.

One important limitation of our corpus of novels is its strong Anglophone and American tilt: there are few non-American and non-Anglophone authors, which inevitably situates the entire analysis within the context of an ``Anglocentric" literary field. 

Regarding the proxies of reader appreciation used in this study, it is hard to control the demographics of each proxy for literary quality and reception. Generally, sources like GoodReads are more diverse and represent a more comprehensive demographic selection than awards committees or anthologies' editorial boards. Yet it should be noted that the majority of GoodReads users from the beginning of GoodReads in 2007 were anglophone. The number of library holdings as a proxy reflects a complex interaction of user demand and expert choice, where demographics are difficult to gauge.  %Beyond this, it is not easy to draw generalizations, as the demographic compositions of awards committees can change quickly, 

It is also likely that there is a correlation between reviews on GoodReads and quality, but as with any proxy measurement, it is difficult to concretely distinguish popularity, success, and quality.




\section*{Ethics Statement}
%DOES NOT COUNT TOWARDS PAGE COUNT

%Scientific work published at EMNLP 2023 must comply with the \href{https://www.aclweb.org/portal/content/acl-code-ethics}{ACL Ethics Policy}. We encourage all authors to include an explicit ethics statement on the broader impact of the work, or other ethical considerations after the conclusion but before the references. The ethics statement will not count toward the page limit (8 pages for long, 4 pages for short papers).
We strongly believe in reproducible and replicable science and are therefore making all data and code freely available where possible. We adhere to best practice guidelines in both the creation and publication of the datasets as suggested by \citet{gebru2021datasheets} and \citet{mohammad2022ethics}. We have assessed the lexicon's suitability for the task at hand and tried to mitigate any inherent biases with our lexicon-enhancement process, however, we may have missed some details and welcome feedback.


%UNCOMMENT FOR FINAL VERSION
\section*{Acknowledgements}
This work was supported by JSPS KAKENHI Grant Number 22K18154.

% Entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom2}
\bibliographystyle{acl_natbib}

\appendix

\section{Appendix}
%\label{sec:appendix}

 
\begin{table*}[ht]
\centering
\begin{tabular}{|c|c|c|}
\hline
\textbf{Pair} & \textbf{Coefficient} & \textbf{Type of Correlation} \\ \hline
Anger, Fear & 0.90 & Strong Positive \\ \hline
Anticipation, Joy & 0.77 & Strong Positive \\ \hline
Disgust, Anger & 0.77 & Strong Positive \\ \hline
Disgust, Sadness & 0.78 & Strong Positive \\ \hline
Fear, Sadness & 0.78 & Strong Positive \\ \hline
Anticipation, Trust & 0.76 & Strong Positive \\ \hline
Joy, Trust & 0.71 & Strong Positive \\ \hline
Anger, Entropy & 0.63 & Moderate Positive \\ \hline
Entropy, Joy & -0.53 & Moderate Negative \\ \hline
Entropy, Trust & -0.51 & Moderate Negative \\ \hline
\end{tabular}
\caption{Pairwise correlation of emotions } %PM: Yuri, please fill in the description here
\label{tab:correlations_pairs}
\end{table*}


\begin{figure*}
    \centering
    \includegraphics[width=1\textwidth]{figures/unsmoothed.png}
    \caption{Unsmoothed emotion arcs for \textit{The Old Man and the Sea}}
    \label{fig:unsmooth}
\end{figure*}
% This is a section in the appendix.


\begin{figure*}
    \centering
    \includegraphics[width=1\textwidth]{figures/trufearnt.png}
    \caption{Smoothed arcs for trust, fear, and anticipation for \textit{The Old Man and the Sea}}
    \label{fig:enter-label1}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=1\textwidth]{figures/anotated_portrait.png}
    \caption{Another annotation of the novel \textit{The Portrait of the Artist as a Young Man} by James Joyce. Note the peak of negative emotions in the center for this book, which \citet{jockers2015response} has called a ``man in a hole" narrative.}
    \label{fig:unsmooth}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=1\textwidth]{figures/similar_words.png}
    \caption{Word similarities for Plutchik's core emotions in the corpus in the affective semantic vector space as measured by cosine similarity. We can see that \textit{trust}, although commonly co-occurring with both \textit{joy} and \textit{anticipation} does not overlap with these emotions. On the other hand, the negative emotions both overlap and co-occur.}
    \label{fig:enter-label2}
\end{figure*}

\begin{figure*}[htbp]
    \centering
    \begin{minipage}[b]{0.5\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/annotators_and_models_plots/joy_annotators.png}
        \caption{\textbf{Joy}, human and model scores.}
    \end{minipage}\hfill
    \begin{minipage}[b]{0.5\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/annotators_and_models_plots/fea_annotators.png}
        \caption{\textbf{Fear}, human and model scores.}
    \end{minipage}\\[10pt]
    \begin{minipage}[b]{0.5\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/annotators_and_models_plots/anger_annotators.png}
        \caption{\textbf{Anger}, human and model scores.}
    \end{minipage}\hfill
    \begin{minipage}[b]{0.5\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/annotators_and_models_plots/ant_annotators.png}
        \caption{\textbf{Anticipation}, human and model scores.}
    \end{minipage}\\[10pt]
    \begin{minipage}[b]{0.5\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/annotators_and_models_plots/sad_annotators.png}
        \caption{\textbf{Sadness}, human and model scores.}
    \end{minipage}\hfill
    \begin{minipage}[b]{0.5\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/annotators_and_models_plots/sur_annotators.png}
        \caption{\textbf{Surprise}, human and model scores.}
    \end{minipage}\\[10pt]
    \begin{minipage}[b]{0.5\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/annotators_and_models_plots/tru_annotators.png}
        \caption{\textbf{Trust}, human and model scores.}
    \end{minipage}\hfill
    \begin{minipage}[b]{0.45\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/annotators_and_models_plots/dis__annotators.png}
        \caption{\textbf{Disgust}, human and model scores.}
    \end{minipage}
    \caption*{Annotator and model scores for 11 randomly selected passages per emotion. On the x-axis, each passage with scores arranged as increasing on the y-axis. For each passage, darker dots represent the EmotionArcs score for the emotion of the passage. Note that the number of annotators varies with respect to emotion and passage.}
\end{figure*}

% \begin{figure*}[]
%     \centering
%     \begin{subfigure}[b]{0.45\textwidth}
%         \includegraphics[width=\textwidth]{figures/annotators_and_models_plots/joy_annotators.png}
%         \caption{Figure 1}
%     \end{subfigure}
%     \hfill
%     \begin{subfigure}[b]{0.45\textwidth}
%         \includegraphics[width=\textwidth]{figures/annotators_and_models_plots/fea_annotators.png}
%         \caption{Figure 2}
%     \end{subfigure}
%     \hfill
%     \begin{subfigure}[b]{0.45\textwidth}
%         \includegraphics[width=\textwidth]{figures/annotators_and_models_plots/anger_annotators.png}
%         \caption{Figure 3}
%     \end{subfigure}
%     \hfill
%     \begin{subfigure}[b]{0.45\textwidth}
%         \includegraphics[width=\textwidth]{figures/annotators_and_models_plots/ant_annotators.png}
%         \caption{Figure 4}
%     \end{subfigure}
%     \hfill
%     \begin{subfigure}[b]{0.45\textwidth}
%         \includegraphics[width=\textwidth]{figures/annotators_and_models_plots/sad_annotators.png}
%         \caption{Figure 5}
%     \end{subfigure}
%     \hfill
%     \begin{subfigure}[b]{0.45\textwidth}
%         \includegraphics[width=\textwidth]{figures/annotators_and_models_plots/sur_annotators.png}
%         \caption{Figure 6}
%     \end{subfigure}
%     \hfill
%     \begin{subfigure}[b]{0.45\textwidth}
%         \includegraphics[width=\textwidth]{figures/annotators_and_models_plots/tru_annotators.png}
%         \caption{Figure 7}
%     \end{subfigure}
%     \hfill
%     \begin{subfigure}[b]{0.45\textwidth}
%         \includegraphics[width=\textwidth]{image8}
%         \caption{Figure 8}
%     \end{subfigure}
%     \caption{Annotator and model scores for 11 randomly selected passages per emotion. On the x-axis, each passage with scores arranged as increasing on the y-axis. For each passage, darker dots represent the EmotionArcs score for the emotion for the passage. Note that the number of annotators varies with respect to emotion and passage.}
% \end{figure*}


\end{document}
