% LREC-COLING 2024 Example; 
% LREC Is now using templates similar to the ACL ones. 
\documentclass[10pt, a4paper]{article}

\usepackage{lrec-coling2024} % this is the new style
\usepackage{listings}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{wrapfig}
\usepackage{array}
\usepackage{booktabs}



% Configure the listings package
\lstset{
    basicstyle=\footnotesize\ttfamily,  % use a smaller font size and monospace font
    breaklines=true,                     % enable line wrapping
    breakatwhitespace=false,             % break lines at whitespace or not
    breakindent=0pt,                     % no indentation for wrapped lines
    columns=fullflexible,               % for better column formatting
    aboveskip=1em,                      % space above the listing
    belowskip=1em,                      % space below the listing
    numbers=left,                       % line numbers on the left
    numbersep=5pt,                      % space between line numbers and code
    xleftmargin=2em,                    % left margin (useful if numbers are used)
}




% \title{StorySense: A Comprehensive Corpus for Narrative Analysis and Language Model Story Comprehension}
\title{Reflections \& Resonance: Two-Agent Partnership for Advanced LLM-based Story Annotation}
%Echoed Endeavors: A Two-Agent Approach to Refining LLM Story Annotation

\name{Yuetian Chen, Mei Si} 

\address{Rensselaer Polytechnic Institute \\
         110 8th Street, Troy, New York 12180 \\
         \{cheny63, sim\}@rpi.edu\\}


\abstract{
% Narrative annotation is a profoundly time-consuming and intricate task, reflecting the deep subtleties of storytelling. To tackle this, we present a cutting-edge multi-agent system to generate tailored prompts for LLM story annotation. Agent A embarks on the process with manually designed prompts steeped in literary nuances. Once annotated, Agent B attempts to reconstruct the story using the annotations Agent A created. Any disparities between the original narrative and this reconstruction then guide Agent B's feedback to Agent A, refining the prompts for enhanced accuracy. This dynamic interplay ensures efficiency in capturing story elements. Utilizing our dual-agent annotation approach, we created the ``StorySense'' corpus, a comprehensive collection of 400 stories spanning 7,234 chapters, each thoroughly annotated.
We introduce a novel multi-agent system for automating story annotation through the generation of tailored prompts for a large language model (LLM). This system utilizes two agents: Agent A is responsible for generating prompts that identify the key information necessary for reconstructing the story, while Agent B reconstructs the story from these annotations and provides feedback to refine the initial prompts. Human evaluations and perplexity scores revealed that optimized prompts significantly enhance the model's narrative reconstruction accuracy and confidence, demonstrating that dynamic interaction between agents substantially boosts the annotation process's precision and efficiency. Utilizing this innovative approach, we created the ``StorySense" corpus, containing 615 stories, meticulously annotated to facilitate comprehensive story analysis. The paper also demonstrates the practical application of our annotated dataset by drawing the story arcs of two distinct stories, showcasing the utility of the annotated information in story structure analysis and understanding.
\\ \newline \Keywords{Narrative Analysis, Prompt Large Language Model, Story Annotation} }
%Understanding the intricacies of narratives demands a sophisticated approach to text analysis, character relationship identification, significant event extraction, and literary technique discernment. To meet this multifaceted challenge head-on, we unveil the \textit{``StorySense''} corpus, a robust collection of 400 stories encompassing 7234 chapters, each meticulously annotated. Designed with precision, StorySense'' aims to extract a comprehensive structural narrative profile, allowing for an in-depth evaluation, judgment, and visualization of story elements. Integrating advanced methodologies, we employ prompt engineering and leverage the capabilities of large language models (LLMs) to automate the generation of analytical reports. These reports holistically cover character states, event timelines, literary techniques, setting nuances, character backstories, interactions, and authorial perspectives. Our framework adeptly navigates the spectrum of narrative styles, from the succinct allegories found in Aesop's Fables to the detailed tapestry of epics like 20,000 Leagues Under the Sea. We assert that the ``StorySense'' corpus will be instrumental in computational linguistics, heralding a new era of synergy between narrative analysis and natural language processing.


 

\begin{document}

\maketitleabstract

\section{Introduction}
% Story annotation delves into the intricate layers of narratives, encompassing not just plot points but the subtleties of character arcs, underlying themes, literary techniques, and the moods that define a story's essence. This deep analysis demands a comprehensive linguistic grasp, contextual awareness, and an understanding of storytelling's multifaceted nature. Given these requirements, the process becomes both significantly labor-intensive and challenging in terms of achieving consistent and reliable results.

% Story annotation requires a deep understanding of narratives, including character arcs, themes, literary techniques, and characters' moods, making it a labor-intensive process that demands extensive linguistic knowledge, contextual awareness, and a grasp of complex storytelling elements for consistent and reliable results. The potential of Large Language Models (LLMs) such as GPT4~\cite{OpenAI2023GPT4TR} in story annotation is transformative. Equipped with a vast lexicon and intricate contextual understanding, LLMs are primed to dissect the complexities within narratives. While manual and script-based annotation methods provide depth, they often struggle with scalability and might be influenced by inherent human biases. In stark contrast, LLMs can process extensive texts swiftly, and avoid human bias and inconsistency. 
% Story annotation involves unraveling the complex layers of narratives, including character arcs, themes, literary techniques, and moods, requiring extensive linguistic knowledge, contextual awareness, and a grasp of storytelling elements. Although this makes annotation a labor-intensive task striving for consistency and reliability, the advent of Large Language Models (LLMs) like GPT-4~\cite{OpenAI2023GPT4TR} introduces transformative potential. With its vast vocabulary and sophisticated understanding of context, GPT-4 can meticulously analyze narrative intricacies, offering a swift and scalable solution. While manual and script-based methods offer in-depth analysis, they face challenges with scalability and potential human bias, areas where LLMs excel by ensuring speed and consistency without bias.

Story annotation, a critical process for dissecting and understanding the multifaceted elements of storytelling such as character arcs, themes, and emotional dynamics, necessitates an intricate understanding of linguistic and narrative structures. Historically, this has been a meticulous and resource-intensive endeavor. Nonetheless, the advent of advanced computational tools like GPT-4~\cite{OpenAI2023GPT4TR}, marks a significant leap forward. GPT-4's capability to rapidly and comprehensively analyze narrative components can potentially streamline this process, and mitigate the common limitations of manual annotation, including scalability challenges and inherent biases.

\begin{wrapfigure}{r}{0.45\columnwidth}
    \centering
    \includegraphics[width=0.5\columnwidth]{figure/agent_structure.jpg}
    \caption{Multi-agent Cooperation Flow}
    \label{fig:agent}
\end{wrapfigure}

% 
The application of LLMs in story annotation introduces its own complexities, necessitating a sophisticated blend of narrative theory and the specific functionalities of the LLMs. While narrative theories illuminate the intricacies of story structure and elements, each LLM, such as GPT-4, possesses distinct capabilities and strengths. To address these challenges, we propose a dual-agent system, inspired by the AutoGen framework~\cite{wu2023autogen}, aimed at enhancing prompt effectiveness through collaboration. Agent A initiates the annotation with prompts based on deep narrative understanding. Agent B then evaluates these prompts through the lens of story reconstruction, offering feedback to refine the prompts in accordance with the accuracy of the reconstructed stories. This process fosters a dynamic, iterative refinement, ensuring the prompts are finely tuned for optimal story annotation, as illustrated in Figure~\ref{fig:agent}. 

%The domain of narrative comprehension and analysis has long been a focal area within computational linguistics. As the quest to decode the nuances of storytelling intensifies, the limitations of traditional methodologies become evident, especially when faced with sparse datasets and static algorithms. Current corpora, such as The Shmoop Corpus~\cite{DBLP:journals/corr/abs-1912-13082}, offer multi-paragraph summaries, giving a linear perspective that aligns with the chronological progression of stories. However, they often fall short in providing the depth and breadth needed for comprehensive narrative analysis. The works of ~\citet{Tambwekar2018Controllable}, ~\citet{Ammanabrolu2019Guided}, and ~\citet{Calvillo2016Connectionist} underscore these challenges, emphasizing the need for datasets that can capture the multifaceted nature of narratives.

% Against this backdrop, we identified a pressing need for a more comprehensive dataset, leading to the development of the "StorySense" corpus. Our primary goal was straightforward: to enhance the depth and breadth of narrative analysis in computational linguistics. While previous datasets provided valuable insights, they often glossed over intricate details like authorial intent, character development, and the interplay of literary devices.

% Our research objective with "StorySense" is two-pronged. First, we aim to establish a corpus that provides a comprehensive overview of stories, offering annotations that shed light on various narrative elements. Our approach leans heavily on the capabilities of Large Language Models (LLMs) and prompt engineering, which facilitates detailed and nuanced narrative extractions. Subsequently, our experiments seek to uncover patterns, motivations, and techniques across different narratives, providing a holistic perspective of the storytelling landscape.

% What sets "StorySense" apart is its commitment to a thorough narrative analysis. Our corpus delves deep into character dynamics, literary techniques, and the essence of settings, offering a multifaceted view of each story. This depth of analysis is not limited to a specific genre or style; our corpus spans the concise narratives of Aesop's Fables to the intricate plots of 20,000 Leagues Under the Sea, encapsulating a diverse range of stories, and ensuring its applicability across various research themes.

% Our methodology, rooted in prompt engineering combined with the power of LLMs, provides a robust framework for automated report generation. Through this approach, we aspire to contribute to the ongoing discussions and advancements in the realm of computational narrative analysis.

\section{Related Work}
\paragraph{Existing Story Annotation Datasets} 
% A plethora of datasets exist catering to story generation and understanding, each uniquely crafted to address various facets of narrative processing. CC-Stories~\cite{trinh2018simple} extracts relevant documents from CommonCrawl for common sense reasoning and language modeling, while TVStoryGen~\cite{chen2021tvstorygen} delves into generating intricate TV show episode summaries, requiring an understanding of character descriptions. STORYWARS~\cite{du2023storywars} stands out with its vast array of collaborative stories from numerous authors, supporting extensive tasks ranging from comprehension to generation in various scenarios. The SPGC~\cite{gerlach2020standardized} and Children Stories Text Corpus~\cite{kaggle_children_stories}, both sourced from Project Gutenberg~\cite{project_gutenberg}, offer extensive book collections, with the latter providing a hand-picked selection of children's books. The Shmoop Corpus~\cite{chaudhury2019shmoop} provides multi-paragraph summaries, offering a chronological view of narratives. Despite the variety, there is a noticeable lack of extensive annotations, especially in character emotions, decisions, narrative arcs, and writing techniques like foreshadowing and flashbacks. This scarcity hinders complete story recreation and detailed narrative analysis, highlighting the urgent need for more in-depth annotations in storytelling datasets.
% Numerous datasets are available for story generation and understanding, each tailored to different aspects of narrative processing. CC-Stories~\cite{trinh2018simple} focuses on common sense reasoning and language modeling using documents from CommonCrawl. TVStoryGen~\cite{chen2021tvstorygen} targets generating detailed TV show episode summaries, necessitating an understanding of character descriptions. STORYWARS~\cite{du2023storywars} offers a vast collection of collaborative stories from a variety of authors, suitable for a wide range of tasks, from comprehension to generation in different scenarios. The SPGC~\cite{gerlach2020standardized} and Children Stories Text Corpus~\cite{kaggle_children_stories}, sourced from Project Gutenberg~\cite{project_gutenberg}, provide extensive book collections, with the latter offering a curated selection of children's books. The Shmoop Corpus~\cite{chaudhury2019shmoop} delivers multi-paragraph summaries for a chronological view of narratives. However, despite the diversity, there's a noticeable lack of detailed annotations on character states, emotions, decisions, narrative arcs, and writing techniques, which hampers full story recreation and thorough narrative analysis. This underscores the need for richer annotations in storytelling datasets.

A variety of datasets exist for story generation and understanding, each serving different narrative processing needs. CC-Stories~\cite{trinh2018simple} emphasizes common sense reasoning with CommonCrawl documents. TVStoryGen~\cite{chen2021tvstorygen} focuses on generating detailed TV show summaries, requiring knowledge of character dynamics. STORYWARS~\cite{du2023storywars} compiles collaborative stories from diverse authors for comprehensive tasks. The SPGC~\cite{gerlach2020standardized} and Children Stories Text Corpus~\cite{kaggle_children_stories}, from Project Gutenberg~\cite{project_gutenberg}, offer large book collections, with the latter specialized in children's literature. The Shmoop Corpus~\cite{chaudhury2019shmoop} provides summaries for a chronological narrative view. However, a gap in detailed annotations concerning character states, emotions, decisions, narrative arcs, and writing techniques still exists. This limitation hampers the datasets' utility for complete story recreation and thorough narrative analysis, thereby calling for the creation of new datasets with richer annotations.

\begin{figure*}[h]
    \centering
    \includegraphics[width=0.93\linewidth]{figure/pipe.jpg}
    \caption{High-level workflow of Reflection \& Resonance framework}
    \label{fig:pipe}
\end{figure*}


% Narrative comprehension, pivotal in disciplines from linguistics to cognitive science, demands assimilation of extensive text, tracking dependencies, and application of narrative techniques. Recent research has diversified the understanding of this domain. \citet{Conlen2019Capture} underscored the significance of reader interactions in narrative visualization, while \citet{Lee2020What} connected narrative memory to neural bases through neuroimaging. \citet{Burris2014When} called for diverse narrative comprehension assessments, and \citet{Mar2021Memory} highlighted narratives' advantages in memory retention over expository texts. \citet{Boyd2020The} distinguished narrative structures between fact-driven and story-based texts.
% Narrative comprehension has been a focal point of research across various disciplines, from linguistics to cognitive science. The ability to understand and generate coherent narratives is a complex cognitive task that requires the assimilation of vast textual content, tracking long-range dependencies, and occasionally, the application of common sense and narrative techniques. Recent studies have delved into various aspects of narrative comprehension.~\citet{Conlen2019Capture} emphasized the importance of capturing and analyzing active reading behaviors for interactive articles on the web. Their findings, tools, and corpus of behavioral data offer insights into narrative visualization, suggesting that understanding reader interactions can provide valuable insights into narrative comprehension.~\citet{Lee2020What} highlighted the role of neuroimaging and naturalistic stimulus analysis in understanding narrative memory. Their work provides valuable insights into the neural bases of human memory, emphasizing the importance of narratives in memory research.~\citet{Burris2014When} advocated for the use of both on- and off-line assessments across various narrative types, such as picture books and televised narratives, especially with traditionally underserved and underrepresented populations. Their work underscores the importance of diverse narrative types in comprehension research.~\citet{Mar2021Memory} conducted a meta-analysis of over 75 unique samples and data from more than 33,000 participants, concluding that narratives are more easily understood and better recalled than expository texts. This finding has significant implications for any domain where acquiring and retaining information is crucial.~\cite{Boyd2020The} explored the narrative arc by revealing core narrative structures through text analysis. Their analysis of fact-driven texts revealed structures that differed from story-based narratives, emphasizing the diverse nature of narrative structures. While these recent studies have significantly contributed to the field of narrative analysis, there remains a gap for a comprehensive corpus that not only offers a vast collection of stories but also provides detailed annotations, metadata, and supplementary data. The StorySense corpus aims to fill this gap, positioning itself as a meticulously curated dataset encompassing 400 stories with a cumulative total of 7234 chapters. Each narrative within the corpus is enriched with detailed annotations, metadata, and supplementary data, making it an indispensable asset for scholars and professionals in narrative analysis, natural language processing, and computational linguistics domains.
% \paragraph{AutoGen}
% AutoGen~\cite{wu2023autogen}, a state-of-the-art multi-agent conversation framework, serves as a pivotal foundation for the orchestration, optimization, and automation of Large Language Model (LLM) workflows. It places a pronounced emphasis on enabling multi-agent conversation and collaboration for content generation. At the heart of AutoGen lies the principle of adaptable and interactive agents, proficient in leveraging the cutting-edge functionalities of leading LLMs like GPT-4. Simultaneously, it addresses their inherent constraints by integrating harmoniously with human expertise and auxiliary tools. This fosters robust multi-agent dialogues through automated chat interfaces. 
% AutoGen~\cite{wu2023autogen}, a state-of-the-art multi-agent conversation framework, serves as a pivotal foundation for the orchestration, optimization, and automation of Large Language Model (LLM) workflows. The framework's architecture is methodically designed for constructing multi-agent conversation systems. It is distilled into two critical phases: defining agents with distinct capabilities and roles, and shaping interaction dynamics among these agents, with a particular focus on message exchange response behaviors. %This dialogue paradigm accentuates the collaborative essence among agents, whether they operate autonomously or are moderated via intermediary agents, to foster conversations targeting task completion.

% AutoGen~\cite{wu2023autogen}, a cutting-edge framework, excels in orchestrating multi-agent conversations to collaboratively attain specific objectives, significantly optimizing Large Language Model (LLM) workflows. The architecture meticulously facilitates the development of multi-agent systems, categorizing into two crucial phases: assigning distinct capabilities and roles to each agent, and meticulously coordinating their interactions, with a particular emphasis on managing message exchange and response behaviors. Inspired by AutoGen's emphasis on collective goal achievement through multiple agents, our approach incorporates a dual-agent system for generating precise prompts in story annotation.



\paragraph{Automated Prompt Engineering} 
% Automated Prompt Engineering (APE) is a pivotal technique that aims to amplify the efficacy of Large Language Models (LLMs) across a spectrum of tasks. It achieves this by formulating optimized prompts that steer the model's behavior towards the intended results. The domain comprises a myriad of methodologies, each characterized by distinct strategies and inherent challenges. Vanilla Prompt Engineering, the bedrock of APE, focuses on the design and optimization of prompts to enhance the LLM's performance, making it indispensable for real-world deployments~\cite{jiang2020can, wei2022chain, lampinen2022can}. Despite its intuitive implementation and minimal computational overhead, Vanilla Prompt Engineering might not always achieve optimal accuracy for complex tasks. A significant challenge is its dependency on the expertise of the prompt engineer, potentially leading to outcome variability~\cite{webson2021prompt}. Building upon basic prompt engineering, Prompt Tuning (P-tuning)\cite{liu2022p} introduces a resource-efficient tuning method. It utilizes a compact trainable model to encode text prompts, generating task-specific virtual tokens that are prepended to the prompt before being processed by the LLM. However, its efficacy may diminish for intricate sequence labeling tasks, and it might require more training data and computational power than basic prompting. In-Context Learning (ICL)\cite{dong2022survey}, a strategy that empowers LLMs to execute tasks using few-shot demonstrations without modifying model parameters, conditions LLMs to make predictions based on the prompt, demonstrating proficiency in managing intricate tasks with minimal examples. Despite their significant contributions, these methodologies have certain constraints related to computational efficiency, performance in intricate scenarios, and adaptability to changing task demands. This highlights the need for a more resilient and flexible prompt design mechanism. AutoGen~\cite{wu2023autogen} offers a groundbreaking approach to address these challenges by orchestrating a multi-agent conversational system, enabling a more fluid and context-aware prompt generation process. AutoGen's cooperative multi-agent framework enhances prompt generation efficiency, substantially diminishing manual intervention and coding effort in intricate LLM workflows, especially in domains like supply-chain optimization. This heralds a transformative direction for advanced automated prompt engineering techniques. Through its comprehensive capabilities, AutoGen aspires to simplify the complexities of harnessing LLMs, potentially surmounting the limitations of prior prompt engineering methods, and setting the stage for future LLM innovations.

% Automated Prompt Engineering (APE) enhances Large Language Models' (LLMs) performance across various tasks by optimizing prompts. Vanilla Prompt Engineering, fundamental to APE, focuses on prompt design and optimization~\cite{jiang2020can, wei2022chain, lampinen2022can}, despite its variability in outcomes due to reliance on human expertise~\cite{webson2021prompt}. Prompt Tuning (P-tuning) introduces a resource-efficient tuning method using a compact trainable model for task-specific virtual tokens~\cite{liu2022p, liu2022ptuning}, though it may struggle with complex sequence labeling and demand more data and computational resources. In-context learning (ICL) enables LLMs to handle complex tasks using few-shot examples without altering model parameters~\cite{dong2022survey}. 

% Existing methods for prompt engineering either necessitate manual design, which can be time-consuming and inconsistent, or rely on substantial training data to achieve optimal results. In contrast, our framework is uniquely crafted to automatically adjust the prompts inspired by the AutoGen framework~\cite{wu2023autogen}, aiming for efficiency and precision without the extensive need for manual input or large datasets.

% AutoGen~\cite{wu2023autogen}, a cutting-edge framework, excels in orchestrating multi-agent conversations to collaboratively attain specific objectives, significantly optimizing Large Language Model (LLM) workflows. The architecture meticulously facilitates the development of multi-agent systems, categorizing into two crucial phases: assigning distinct capabilities and roles to each agent, and meticulously coordinating their interactions, with a particular emphasis on managing message exchange and response behaviors. Inspired by AutoGen's emphasis on collective goal achievement through multiple agents, our approach incorporates a dual-agent system for generating precise prompts in story annotation.


Automated Prompt Engineering (APE) boosts the efficacy of Large Language Models (LLMs) by fine-tuning prompts. Vanilla Prompt Engineering, which is crucial for APE, emphasizes the creation and optimization of prompts~\cite{jiang2020can, wei2022chain, lampinen2022can}, although its effectiveness is tempered by the variability inherent in human-driven processes~\cite{webson2021prompt}. Prompt Tuning (P-tuning) proposes a more resource-efficient strategy by utilizing a compact, trainable model for generating task-specific virtual tokens~\cite{liu2022p, liu2022ptuning}, but it can encounter difficulties with intricate tasks and requires significant data and computational power. Meanwhile, In-context learning (ICL) allows LLMs to adapt to complex tasks with minimal examples, without the need for adjusting the underlying model parameters~\cite{dong2022survey}.

However, traditional methods for prompt engineering often rely on manual effort, leading to time consumption and inconsistency, or require large datasets to achieve optimal outcomes. In response, our framework draws inspiration from the AutoGen framework~\cite{wu2023autogen}. AutoGen is designed for collaborative goal achievement using multi-agent systems, optimizing LLM workflows by assigning specific roles to agents and coordinating their interactions. By applying this concept, our approach uses a dual-agent system to generate precise prompts for story annotation. This strategy streamlines the prompt engineering process and improves outcomes by harnessing the collaborative potential of multi-agent systems.

\section{Methodology \& Implementation}
% As shown in Figure~\ref{fig:pipe}, the overarching architecture of our methodology rests on a cooperative paradigm between two agents (Noted as \texttt{Agent A} and \texttt{Agent B} as follow). In practice, we want the framework to receive the base prompt identifying the task and ground truth story to be annotated as the input in a problem initialization stage, where Agent A would receive take this information and generated the baseline annotation, and an multi-agent optimization stage, during which both \texttt{Agent A} is deigned with a primary capability of understanding narrative content and generating detailed annotations and the interation with users for input. \texttt{Agent B} .This collaboration employs a dynamic feedback loop, optimizing the narrative annotation process through iterative refinement. After predefined $N_{iter}$ iterations, we will use the updated prompt to generate the annotation.

In our method, illustrated in Figure~\ref{fig:pipe}, we utilize a collaborative interaction between two main agents, both powered by GPT-4, named \texttt{Agent A} and \texttt{Agent B}. This process involves a dynamic and self-guided iterative feedback loop where \texttt{Agent B} provides guidance to \texttt{Agent A}, aiming to optimize the narrative annotation task. 

\begin{table*}[ht]
\centering
\begin{tabular}{@{}>{\raggedright\arraybackslash}p{\linewidth}@{}}
\toprule
The generated story does utilize many of the annotations you provided and crafts them into a coherent narrative. However, if you are looking for an exact match or specific elements to be included in a certain way, then the prompt needs to be more explicit.\\\\
Here's a more detailed prompt based on your requirements:\\
\textbf{STORY CREATION PROMPT}\\
Use the following annotated predicates and specific information to craft a narrative:\\
\textbf{Character and Relationship Information:}\\
- Ned Land is a harpooner onboard the ship Abraham Lincoln.\\
... ...\\
\textbf{Starting Scenario and Beliefs:}\\
- The story must begin with the Professor onboard the Abraham Lincoln, which is at sea.\\
... ...\\
\textbf{Major Plot Points:}\\
... ...\\
\textbf{Narrative Tone and Perspective:}\\
- The story should maintain a first-person perspective from the Professor's viewpoint.\\
\bottomrule
\end{tabular}
\caption{Agent B's feedback example}
\label{tab:prompt_feedback}
\end{table*}

\subsection{Agent A -- the Annotating Agent}
% During the Problem Initialization Stage, our framework begins with a basic prompt that outlines the specific requirements of the annotation task. This initial prompt includes both basic directions for annotation analysis and essential formatting details. Specifically, it encapsulates crucial details necessary for comprehensive story comprehension. These details fall into two categories:
% \begin{enumerate}
%     \item Story Content: This encompasses aspects such as the characters' states, intentions, motivations, and emotions. It also charts the evolution of these elements throughout the narrative, identifying triggers or reasons for these developments. Such information facilitates a reader's grasp of the story's events.
%     \item Narrative Presentation: This section elucidates the narrative structure, addressing questions like whether events are recounted chronologically or in reverse. It further probes the usage of literary techniques, such as foreshadowing or flashbacks, which can enhance a reader's retention and understanding of the storyline.
% \end{enumerate}
During the Problem Initialization Stage, Agent A initiates a basic prompt that captures the specific requirements of the annotation task. This prompt merges guidelines for analysis with necessary formatting details and includes vital elements for deep story comprehension:
\begin{enumerate}
\item Story Content: This includes characters' states, intentions, motivations, and emotions, and tracks their evolution, shedding light on the narrative's dynamics.
\item Narrative Presentation: It details the narrative structure, such as whether the storytelling is chronological or non-linear, and looks into literary techniques like foreshadowing or flashbacks to enhance understanding and engagement.
\end{enumerate}
Agent A's approach is designed to lay a comprehensive groundwork for annotating and understanding stories effectively. The full prompt is shown in Figure~\ref{fig:pipe}.
% As an example, the base prompt for constructing the ``StorySense'' corpus is shown in Figure~\ref{??}.
% :
% \begin{lstlisting}
% Answer the following questions: 
% Provide predicates that describe the characters and their relationships in the story; 
% Initial states, beliefs, and characters at the start of the story in predicates; 
% ...
% \end{lstlisting}
% Drawing a chronological timeline of events; 
% For each major event, list how characters or objects' states and beliefs are affected;
% Use of "forecast" and "flashback" techniques by the author, detail the events associated with each using predicates.
% Identification of literary tropes in the story, offering explanations for their use; 
% Analysis of the author's change in perspectives

% Additionally, based on these concatenated inputs, \texttt{Agent A} would generate annotations that serve as the start of the iteration, and upon completion, both the annotated story and the original story—which serves as the ground truth—are forwarded to Agent B for further evaluation.
% And, concomitantly, a ground truth story, which serves as a veritable benchmark for the entire process, is also ingested into the system. With these foundational elements in place, \texttt{Agent A} swings into action. With the help of Large Language model's inference, empowered with an innate capability for comprehending narrative intricacies, this agent leverages the base prompt and ground truth story to generate an initial set of annotations, establishing the groundwork for subsequent stages.

\subsection{Agent B -- the Critic Agent}
% Transitioning to the Multi-agent Optimization Stage, the narrative landscape becomes more collaborative and dynamic. Herein, \texttt{Agent A}'s competencies extend beyond mere narrative comprehension. As it crafts detailed annotations, the agent also exhibits an adaptability, wherein it can assimilate inputs and feedback from external user interactions. This feature is paramount for maintaining the relevance and accuracy of the evolving annotations.

% Complementing the efforts of \texttt{Agent A}, \texttt{Agent B} plays a pivotal role during this stage. Operating as a narrative reconstructor, \texttt{Agent B} harnesses the annotations from \texttt{Agent A} to reconstruct the story. That is, Agent B compares the differences between the generated results and the ground truth to directly analyze the shortcomings of the annotation and thus obtain the improvement direction of the prompt. This iterative reconstruction serves as a mirror, reflecting the efficacy of the annotations in capturing the essence of the ground truth story. Disparities, if any, between the reconstructed and original narrative, spark a feedback mechanism. This dynamic interplay, characterized by the feedback loop, fuels the iterative refinement process, allowing both agents to refine and optimize their strategies and outputs.

% After undergoing a predefined number of iterations, encapsulated by the parameter $N_{iter}$, the culmination of this cooperative endeavor is realized. The refined and updated prompt, enriched by the insights gleaned from the multi-agent optimization process, is then wielded to generate the final, optimized annotation.

In Stage II, \texttt{Agent B} takes charge of reconstructing a story from the given annotations from \texttt{Agent A} using the following prompt:
\begin{lstlisting}
Use the following annotated predicates and specific information to craft a story:
[Generated annotations]
\end{lstlisting}

Agent B then compares the newly assembled story, termed the ``Reconstructed Story," with the Ground Truth, utilizing the following guidelines for evaluation:
\begin{lstlisting}
Here is the actual ground truth of the story and instructions for annotation.
[Ground Truth Story]
[Current Prompt]
Analyze the differences and explain what other information you need to improve the generated story.

\end{lstlisting}

% to identify differences and make annotation feedback given current prompt. 
An example of Agent B's feedback is provided in Table~\ref{tab:prompt_feedback}. The feedback is then used by \texttt{Agent A} to revise the prompt for the next iteration using the following instruction:
\begin{lstlisting}
Here is the feedback I have:
[Annotation Feedback from Agent B]
let's fix it in prompt further

\end{lstlisting}

The iteration ends after a predetermined $N_{iter}$ number of cycles. 
% to make a better annotation prompt, which in turn is used to create a new annotation for the next cycle.
% This refined prompt is then employed to produce an improved annotation in the subsequent cycle. After a predetermined number of iterations, $N_{iter}$, we employ the final, refined prompt generated through this iterative process to produce the narrative annotation.

% In the Multi-agent Optimization Stage, the process becomes more collaborative with Agent A not just understanding the narrative but also adapting to feedback from Agent B, ensuring the annotations remain relevant and accurate. Agent B steps in as a narrative reconstructor, utilizing Agent A's annotations to rebuild the story and compare it to the original. This comparison highlights areas of improvement, guiding the refinement of the prompt through an iterative feedback loop. After a set number of iterations, defined by $N_{iter}$, this collaborative process results in a refined prompt, leading to the generation of the final, optimized annotation.

% To illustrate the nature of the feedback and its impact on the prompts, consider the following example:
% \begin{table*}[ht]
% \centering
% \begin{tabular}{@{}>{\raggedright\arraybackslash}p{\linewidth}@{}}
% \toprule
% The generated story does utilize many of the annotations you provided and crafts them into a coherent narrative. However, if you are looking for an exact match or specific elements to be included in a certain way, then the prompt needs to be more explicit.\\\\
% Here's a more detailed prompt based on your requirements:\\
% \textbf{STORY CREATION PROMPT}\\
% Use the following annotated predicates and specific information to craft a narrative:\\
% \textbf{Character and Relationship Information:}\\
% - Ned Land is a harpooner onboard the ship Abraham Lincoln.\\
% - The narrator, known as the Professor, is not only an employee on the Abraham Lincoln ship but also holds a deep respect from Conseil.\\
% - Conseil is the Professor's loyal servant, and they share a strong bond of trust.\\
% \textbf{Starting Scenario and Beliefs:}\\
% - The story must begin with the Professor onboard the Abraham Lincoln, which is at sea.\\
% - The Professor holds an underlying fear that he might drown if he goes overboard.\\
% - Conseil believes in his duty to the Professor and would do anything to protect him.\\
% - Ned Land holds an unconventional belief that the creature they're pursuing isn't organic but made of boilerplate steel.\\
% \textbf{Major Plot Points:}\\
% - The Professor is thrown overboard during their pursuit.\\
% - Conseil immediately jumps to rescue him, showing his unwavering loyalty.\\
% - As they both try to stay afloat, the true nature of the "monster" they're pursuing is revealed: it's a technologically advanced submarine.\\
% - Both the Professor and Conseil are captured by the crew of this submarine.\\
% \textbf{Literary Techniques:}\\
% - Employ dramatic irony. Even though readers might deduce the true nature of the monster, the characters should only come to this realization later.\\
% - Use forecast by hinting at potential dangers or twists in the narrative.\\
% - The revelation of the submarine should serve as a significant backlash, contrasting the characters' original beliefs.\\
% \textbf{Narrative Tone and Perspective:}\\
% - The story should maintain a first-person perspective from the Professor's viewpoint.\\
% \bottomrule
% \end{tabular}
% \caption{Agent B's feedback sample}
% \label{tab:prompt_feedback}
% \end{table*}

% In this feedback, Agent B acknowledges that the generated story is coherent and utilizes the provided annotations. However, it also points out that if the goal is to achieve an exact match or include specific elements in a certain way, the prompt needs to be more explicit. Agent B then proceeds to provide a more detailed prompt based on the identified requirements.

% The revised prompt includes specific information about the characters and their relationships, the starting scenario and beliefs, major plot points, literary techniques to be employed, and the desired narrative tone and perspective. By incorporating these details into the prompt, Agent A can generate annotations that are more closely aligned with the ground truth story and better capture the desired narrative elements.

% This example demonstrates how the feedback from Agent B helps to refine the prompts iteratively, leading to more accurate and comprehensive annotations. By providing specific guidance on the required information and narrative elements, the feedback enables Agent A to generate annotations that better capture the nuances and intricacies of the ground truth story.

\subsection{Construct the StorySense Corpus}
% Utilizing the previously described methodology, we successfully created the StorySense corpus, encompassing 615 stories. The corpus includes a diverse range of narratives, from the concise, moral-driven stories found in Aesop's Fables to the intricate and expansive epics exemplified by ``20,000 Leagues Under the Sea.'' The source of these stories is a compilation of public domain texts, curated from repositories including Project Gutenberg~\cite{project_gutenberg}, Shmoop Corpus\cite{chaudhury2019shmoop}, and Aesop's Fables~\cite{gibbs2002aesop}. In filtering, We kept the word count of each sample between 200 and 1800 tokens. Figure~\ref{fig:token-length} shows statistics about the lengths of the stories and the number of characters in each story. 
Using our methodology, we developed the StorySense corpus with 615 stories, ranging from Aesop's Fables' brief, moral tales to the complex narratives of ``20,000 Leagues Under the Sea." These stories were sourced from public domain collections including Project Gutenberg~\cite{project_gutenberg}, the Shmoop Corpus~\cite{chaudhury2019shmoop}, and Aesop's Fables~\cite{gibbs2002aesop}, ensuring a diverse narrative spectrum. We limited the story length to 200-1800 tokens to maintain uniformity. Figure~\ref{fig:token-length} presents data on story lengths and character counts.
\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{figure/token_length_histogram.jpg}
    \caption{Story lengths and character counts}
    \label{fig:token-length}
\end{figure}

% The StorySense corpus covers a wide array of topics, ranging from adventure and exploration in works like ``The Adventures of Tom Sawyer" and ``Treasure Island," to moral and ethical lessons in Aesop's Fables such as ``The Tortoise and the Hare" and "The Lion and the Mouse." The corpus also includes stories that delve into social commentary, as seen in ``The Scarlet Letter" and ``The Adventures of Huckleberry Finn." This diverse distribution of topics ensures that the corpus is representative of various narrative themes and styles.

% The lengths of the stories in the StorySense corpus vary significantly, reflecting the breadth of the included works. The shortest stories, primarily from Aesop's Fables, contain an average of 300 tokens, while the longest stories, such as ``Moby Dick" and ``The Count of Monte Cristo," have an average of 1,500 tokens per chapter. The mean token count across all stories in the corpus is approximately 850 tokens, with a standard deviation of 350 tokens, indicating a substantial variation in story lengths.

% The number of characters in the stories also varies greatly, depending on the complexity and scope of the narrative. Aesop's Fables typically feature a small cast of characters, often focusing on two or three main actors, such as in ``The Fox and the Grapes" or ``The Wolf in Sheep's Clothing." On the other hand, more expansive works like ``Les Misérables" and ``War and Peace" boast a vast array of characters, with some chapters featuring up to 20 distinct named entities. On average, the stories in the StorySense corpus include 7 named characters, with a standard deviation of 4 characters.

% Each story within the corpus has been annotated using the optimized prompt generated through our baseline prompt, refined via Agent B five times ($N_{iter} = 5$). Both the annotations and the stories are meticulously stored in a structured format with both machine-understandable and human-understandable information for different aspects of stories, facilitating ease of access and analysis. The annotations include detailed information on character relationships, plot progression, and literary techniques employed, making the StorySense corpus a valuable resource for researchers and practitioners in the fields of natural language processing, computational linguistics, and narrative analysis.

% To provide a clearer understanding of the consistency and convergence of the annotations generated by Agent A, we analyzed 615 sample prompts among the corpus and identified the most frequently used tags. 

Each story in the corpus was annotated with a prompt optimized from our baseline, refined five times by Agent B ($N_{iter} = 5$). The full optimized prompt is presented in Table 1 of the supplementary material. Table~\ref{tab:tag_freq} presents the three most frequent tags in three categories, showing their prevalence in the annotations. For instance, the tag \texttt{is\_protagonist(X)} is found in 95\% of the stories, highlighting that nearly all stories feature a protagonist. Conversely, tags under the Significant Events category show a wide variety, reflecting the diversity of narratives in the corpus.%These tags not only demonstrate the agent's ability to consistently identify and describe key elements of the story but also highlight the effectiveness of the annotation structure in capturing essential aspects of the narrative.

\begin{table*}[h]
\centering
\begin{tabular}{p{0.3\linewidth}p{0.25\linewidth}p{0.1\linewidth}p{0.3\linewidth}}
\hline
\textbf{Category} & \textbf{Tag} & \textbf{Freq.} & \textbf{Example} \\[5pt]
\hline
\textbf{Character and Relationship Descriptions in Predicates} & 
\texttt{is\_protagonist(X)} & 95\% & \texttt{is\_protagonist(Alice)} \\[5pt]
 & \texttt{is\_antagonist(X)} & 80\% & \texttt{is\_antagonist(Queen)} \\[5pt]
 & \texttt{is\_mentor(X, Y)} & 65\% & \texttt{is\_mentor(Cheshire\_Cat, Alice)} \\[10pt]
\hline
\textbf{Initial Conditions} & 
fantasy world & 85\% & The story takes place in a whimsical fantasy world. \\[5pt]
 & curiosity & 70\% & The protagonist's curiosity drives the story forward. \\[5pt]
 & talking animals & 60\% & The story features animals with human-like qualities. \\[10pt]
\hline
\textbf{Significant Events in Predicates} & 
\texttt{falls\_into(X, Y)} & 15\% & \texttt{falls\_into(Frodo, Mount\_Doom)} \\[5pt]
 & \texttt{attends(X, Y)} & 12\% & \texttt{attends(Harry, Hogwarts)} \\[5pt]
 & \texttt{confronts(X, Y)} & 12\% & \texttt{confronts(Luke, Darth\_Vader)} \\[10pt]
\hline
% \textbf{Literary Techniques in Depth} & 
% symbolism & 80\% & The story uses symbolism to convey deeper meanings. \\[5pt]
%  & nonlinear narrative & 60\% & The narrative structure is nonlinear, with jumps in time and space. \\[5pt]
%  & irony & 60\% & It's ironic that when the wolf actually appears, no one believes the boy. \\[10pt]
% \hline
% \textbf{Author's Perspective and Techniques in Detail} & 
% omniscient narrator & 90\% & The narrator has an all-knowing perspective on the events and characters. \\[5pt]
%  & unreliable narrator & 40\% & The narrator's credibility is questionable, leaving room for interpretation. \\[5pt]
%  & stream of consciousness & 35\% & The narrative follows the protagonist's uninterrupted flow of thoughts and perceptions. \\[10pt]
% \hline
% \textbf{Tone, Style, and Length} & 
% whimsical & 85\% & The story has a playful and imaginative tone. \\[5pt]
%  & satirical & 70\% & The story uses satire to critique societal norms and conventions. \\[5pt]
%  & novella & 50\% & The story is a short novel, typically between 1000 and 1200 tokens. \\[5pt]
% \hline
\end{tabular}
\caption{Example tags generated during annotation}
\label{tab:tag_freq}
\end{table*} 
% Upon examining the table, it becomes apparent that certain tags are more prevalent than others within their respective sections. For instance, in the ``Character and Relationship Descriptions in Predicates" section, the tag \texttt{is\_protagonist(X)} appears in a significant majority of the samples (95\%). This high frequency indicates that Agent A is adept at recognizing and describing the protagonist using a standardized predicate format. Additionally, the presence of tags like \texttt{is\_antagonist(X)} and \texttt{is\_mentor(X, Y)} suggests that the agent effectively captures the nuances of character relationships and roles.

% Moving on to the ``Initial Conditions" section, we observe that tags related to the story's setting and themes, such as ``fantasy world," ``curiosity," and ``talking animals," appear with high frequencies (85\%, 70\%, and 60\%, respectively). This consistency in identifying and describing the initial context of the story demonstrates Agent A's ability to grasp the foundational elements that set the stage for the narrative.

% The ``Significant Events in Predicates" section showcases the agent's proficiency in pinpointing and representing crucial plot points using standardized predicates. Tags like \texttt{falls\_into(X, Y)}, \texttt{attends(X, Y)}, and \texttt{confronts(X, Y)} appear with notable frequencies, indicating that Agent A consistently identifies and encodes the key events that drive the story forward across various narratives.

% In the ``Literary Techniques in Depth" section, the high frequency of tags such as ``symbolism," ``nonlinear narrative," and ``allusion" demonstrates the agent's capacity to recognize and analyze the use of literary devices within the narrative. This deeper understanding of the story's structure and themes is crucial for providing comprehensive annotations.

% The ``Author's Perspective and Techniques in Detail" section reveals that Agent A reliably identifies the narrative perspective and techniques employed by the author. The high frequencies of tags like ``omniscient narrator" and ``unreliable narrator" indicate that the agent consistently recognizes the storytelling approach, while the presence of tags like ``stream of consciousness" shows its ability to identify specific techniques used to convey information and character thoughts.

% Lastly, the ``Tone, Style, and Length" section demonstrates Agent A's understanding of the overall narrative style and structure. The high frequency of tags such as ``whimsical" and ``satirical" suggests that the agent accurately identifies the tone and genre, while the presence of tags like "novella" indicates its ability to recognize the length and format of the story.

% The consistency and convergence of tags within each section highlight the effectiveness of Agent A's annotation process. By using standardized predicates and a structured approach, the agent ensures that essential aspects of the narrative are consistently captured across different stories. This consistency facilitates comparative analysis and the identification of common themes or patterns, making the annotations a valuable resource for further study and interpretation.

Moreover, the high frequencies of certain tags within each section demonstrate the agent's robustness in identifying and describing key elements of the story, regardless of variations in the specific narratives. This adaptability is crucial for generating comprehensive annotations across a wide range of stories and genres.

% \begin{table}[h]
% \centering
% \begin{tabular}{clcc}
% \hline
% \textbf{Rank} & \textbf{Tag} & \textbf{Frequency} & \textbf{Section(s)} \\
% \hline
% 1 & is\_character(X) & 87\% & Character and Relationship Descriptions \\
% 2 & event(E) & 82\% & Significant Events, Timeline of Events \\
% 3 & setting(S) & 79\% & Initial Conditions, Setting Details \\
% 4 & believes(X, B) & 74\% & State and Belief Changes \\
% 5 & motivated\_by(X, M) & 68\% & Character Backstories and Motivations \\
% 6 & dialogue(X, Y, D) & 63\% & Interactions and Dialogues \\
% 7 & third-person & 58\% & Author's Perspective and Techniques \\
% 8 & foreshadowing & 52\% & Literary Techniques, Tone, Style, and Length \\
% 9 & flashback & 47\% & Literary Techniques, Tone, Style, and Length \\
% 10 & symbolism & 42\% & Literary Techniques \\
% \hline
% \end{tabular}
% \caption{Top 10 most frequently used tags in the generated annotations.}
% \label{tab:tag_stats}
% \end{table}

% The most common tag, appearing in 87\% of the annotations, is is\_character(X), which is used in the "Character and Relationship Descriptions" section to identify characters in the story. The event(E) tag, present in 82\% of the annotations, is used in the "Significant Events" and "Timeline of Events" sections to describe and chronologically organize the major events in the narrative.

% Other frequently used tags include setting(S) (79\%), used in the "Initial Conditions" and "Setting Details" sections to describe the story's setting; believes(X, B) (74\%), used in the "State and Belief Changes" section to capture characters' beliefs; and motivated\_by(X, M) (68\%), used in the "Character Backstories and Motivations" section to explore characters' motivations.

% The presence of literary technique tags such as foreshadowing (52\%), flashback (47\%), and symbolism (42\%) in the "Literary Techniques" and Tone, Style, and Length" sections demonstrate the model's ability to identify and analyze these narrative devices. The third-person tag (58\%), used in the "Author's Perspective and Techniques" section, indicates that the model can recognize the author's point of view in the story.

% The high frequency of these tags across the generated annotations suggests that Agent A consistently identifies and applies these concepts when analyzing stories. This consistency in terminology allows for more efficient and accurate comparisons between different stories, as the annotations follow a structured and standardized format. Furthermore, the convergence of these tags demonstrates the effectiveness of the multi-agent optimization process in refining the prompts to capture the most relevant and informative aspects of the narratives.

% \subsection{Justification for the Number of Iterations}
% The choice of five iterations for refining the optimized prompt was guided by several critical observations. Initially, the iterative process of prompt optimization by Agent B was expected to continuously enhance the reconstruction's fidelity to the original narrative. However, our analysis indicated two significant phenomena: diminishing returns on iterations and a tendency for overfitting.

% We noticed that beyond a certain number of iterations, additional adjustments made by Agent B contributed minimally to the improvement of the reconstructed story's accuracy. The incremental gains from these iterations often resulted in adjustments that were either too minor to substantially enhance the narrative's quality or were overly specific, reducing the general applicability of the prompts. 

% A pronounced risk of overfitting was also observed with an increasing number of iterations. The prompts began to reflect details overly specific to the story at hand, such as nuanced character motivations or unique plot twists, which, while increasing the accuracy of that particular story reconstruction, potentially limited the prompt's utility across a broader spectrum of narratives.

\begin{table*}
    \centering
    \begin{tabular}{lcccccc}
        \hline
        \textbf{Story ID} & \textbf{Ground Truth} & \multicolumn{2}{c}{\textbf{Optimized Prompt}} & \multicolumn{2}{c}{\textbf{Baseline Prompt}} \\
        & \textbf{Perplexity} & \textbf{Perplexity} & \textbf{Relevance Score} & \textbf{Perplexity} & \textbf{Relevance Score} \\
        \hline
        1 & 20.03 & \textbf{13.43} & \textbf{4.00} & 14.74 & 2.92 \\
        2 & 15.78 & \textbf{11.47} & \textbf{3.75} & 12.36 & 3.17 \\
        3 & 15.69 & \textbf{12.26} & \textbf{3.58} & 13.82 & 3.25 \\
        4 & 20.03 & \textbf{9.77}  & \textbf{3.75} & 11.42 & 3.58 \\
        5 & \textbf{10.75} & 13.11 & \textbf{3.92} & 11.05 & 2.42 \\
        \hline
    \end{tabular}
    \caption{Relevance evaluation and perplexity scores for 5 stories}
    \label{tab:merged_scores}
\end{table*}

% These observations led to the strategic decision to limit the number of iterations to five. This number was determined to balance effectively between achieving a significant improvement in narrative reconstruction and mitigating the risks of diminishing returns and overfitting. This balanced approach ensures that the prompts remain sufficiently general to be applicable across different stories while still capturing the essential elements that characterize the unique narrative structure of Aesop's Fables.

% \subsection{Prompt Structure and Key Tags}
% The optimized prompt used in our narrative generation process incorporates several key tags to guide the analysis and annotation of stories. These tags are designed to capture essential aspects of the narrative, ensuring comprehensive and structured outputs. Table~\ref{tab:prompt_sample} presents the original prompt used in our study.

% Character-related tags focus on describing characters and their relationships, such as the \textbf{Character and Relationship Descriptions} tag, which prompts the model to use predicates like \texttt{is\_kid(X)} and \texttt{is\_afraid\_of(X, Y)}. The \textbf{Character Backstories and Motivations} tag explores characters' past experiences and motivations relevant to the story, while the \textbf{Interactions and Dialogues} tag highlights and analyzes significant dialogues or interactions, contributing to character development.

% Event-related tags capture the progression of the story and the impact of major events on characters and the narrative. The \textbf{Initial Conditions} tag details the starting conditions, including main characters, setting, and initial challenges. The \textbf{Significant Events} tag lists and describes major events using predicates, while the \textbf{Timeline of Events} tag provides a chronological representation of these events. The \textbf{State and Belief Changes} tag captures how characters' states, beliefs, or emotions are affected by major events, crucial for understanding character development and the impact of plot points.

% Style-related tags focus on the literary techniques and narrative style employed by the author. The L\textbf{iterary Techniques} tag encourages the model to analyze the use of literary devices, such as flashbacks, foreshadowing, or metaphors, and their impact on the narrative. The \textbf{Author's Perspective and Techniques} tag examines the author's unique storytelling methods, perspective shifts, and writing style, offering insights into the story's structure, tone, and themes. The \textbf{Setting Details} tag prompts the model to provide vivid descriptions of the setting, including ambiance and sensory details, immersing the reader in the story's world.

% The \textbf{Character Backstories and Motivations} tag explores characters' past experiences and motivations relevant to the story, while the \textbf{Interactions and Dialogues} tag highlights and analyzes significant dialogues or interactions, contributing to character development and plot progression.

% By including these key tags in the prompt, the model generates annotations that are comprehensive, structured, and focused on essential aspects of the story, as we further discussed in the Evaluation. This approach leads to higher-quality outputs that capture the nuances of the narrative, facilitating further analysis and interpretation. The structured nature of the annotations, guided by the key tags, ensures that critical information is consistently captured across different stories, enabling comparative analysis and the identification of common themes or patterns.



% \begin{table}[ht]
% \centering
% \begin{tabular}{@{}>{\raggedright\arraybackslash}p{\linewidth}@{}}
% \toprule
% A Shepherd Boy tended his master's Sheep near a dark forest not far from the village. Soon he found life in the pasture very dull. All he could do to amuse himself was to talk to his dog or play on his shepherd's pipe.

% One day as he sat watching the Sheep and the quiet forest, and thinking what he would do should he see a Wolf, he thought of a plan to amuse himself.

% His Master had told him to call for help should a Wolf attack the flock, and the Villagers would drive it away. So now, though he had not seen anything that even looked like a Wolf, he ran toward the village shouting at the top of his voice, ``Wolf! Wolf!"

% As he expected, the Villagers who heard the cry dropped their work and ran in great excitement to the pasture. But when they got there they found the Boy doubled up with laughter at the trick he had played on them.

% A few days later the Shepherd Boy again shouted, "Wolf! Wolf!" Again the Villagers ran to help him, only to be laughed at again.

% Then one evening as the sun was setting behind the forest and the shadows were creeping out over the pasture, a Wolf really did spring from the underbrush and fall upon the Sheep.

% In terror the Boy ran toward the village shouting ``Wolf! Wolf!" But though the Villagers heard the cry, they did not run to help him as they had before. ``He cannot fool us again," they said.
% The Wolf killed a great many of the Boy's sheep and then slipped away into the forest.\\
% \bottomrule
% \end{tabular}
% \caption{Original narrative sample: ``The Shepherd Boy and the Wolf"} % Caption placed here to appear below the table
% \label{tab:story_sample}
% \end{table}

% \begin{table*}[ht]
% \centering
% \begin{tabular}{@{}>{\raggedright\arraybackslash}p{\linewidth}@{}}
% \toprule
% \textbf{Introduction:}\\
% The story revolves around a dispute between Bees and Wasps over the ownership of a store of honey found in a hollow tree. Unable to resolve the issue amongst themselves, they decide to approach the Hornet, a justice of the peace in the woods, to adjudicate the matter. The story touches on themes of justice, wisdom, and the inherent nature of beings.\\\\

% \textbf{Character and Relationship Descriptions in Predicates:}\\
% \begin{itemize}
%     \item is\_insect(X) - X is an insect.
%     \item claims\_honey(X) - X claims the honey.
%     \item is\_judge(X) - X is a judge.
%     \item testifies\_in\_court(X) - X testifies in court.
%     \item is\_wise(X) - X is wise.
% \end{itemize}\\

% \textbf{Initial Conditions:}\\
% \begin{enumerate}
%     \item \textbf{Characters:}
%     \begin{itemize}
%         \item is\_insect(Bees)
%         \item is\_insect(Wasps)
%         \item is\_judge(Hornet)
%     \end{itemize}
%     \item \textbf{Setting:}
%     The story begins in a forest, specifically around a hollow tree.
%     \item \textbf{State of Affairs:}
%     Bees and Wasps are in conflict over the ownership of honey found in the hollow tree.
% \end{enumerate}\\

% \textbf{Significant Events in Predicates:}\\
% \begin{itemize}
%     \item claims\_honey(Bees)
%     \item claims\_honey(Wasps)
%     \item is\_judge(Hornet)
%     \item testifies\_in\_court(Witnesses)
%     \item testifies\_in\_court(Ant)
%     \item is\_wise(Old\_Bee)
% \end{itemize}\\

% \textbf{Timeline of Events:}\\
% \begin{enumerate}
%     \item Discovery of honey in a hollow tree.
%     \item Argument between Bees and Wasps over ownership.
%     \item Decision to let the Hornet judge the matter.
%     \item First hearing where witnesses describe creatures.
%     \item Adjournment of the court for six weeks.
%     \item Resumption with Ant's testimony.
%     \item Old Bee's suggestion to build a honeycomb.
%     \item Wasps protest the suggestion.
%     \item Hornet's verdict in favor of the Bees.
% \end{enumerate}\\

% \textbf{State and Belief Changes for Each Character:}\\
% \begin{itemize}
%     \item \textbf{Bees:} Starts with a claim $\rightarrow$ Remains hopeful $\rightarrow$ Feels vindicated.
%     \item \textbf{Wasps:} Starts with a claim $\rightarrow$ Remains hopeful $\rightarrow$ Feels defeated.
%     \item \textbf{Hornet:} Starts neutral $\rightarrow$ Feels uncertain $\rightarrow$ Makes a wise decision.
% \end{itemize}\\

% \textbf{Literary Techniques in Depth:}\\
% \begin{itemize}
%     \item \textbf{Allegory:} The story can be seen as an allegory for real-world disputes where evidence and inherent nature play a role in judgments.
%     \item \textbf{Characterization:} The wise old Bee and the just Hornet are characterized to bring forth the themes of wisdom and justice.
% \end{itemize}\\

% \textbf{Setting Details and Atmospheric Details:}\\
% The setting is a vibrant forest with a significant hollow tree. The ambiance is tense due to the ongoing dispute, with various insects gathered around, discussing and witnessing the trial.\\\\

% \textbf{Character Backstories and Motivations:}\\
% While not explicitly mentioned, the Bees' motivation is clear in their inherent nature to produce honey and build combs. The Wasps, on the other hand, attempt to lay claim despite not having the ability to produce honey.\\\\

% \textbf{Interactions and Dialogues:}\\
% The old Bee's dialogue proposing the honeycomb test is pivotal. It showcases the Bee's wisdom and the true nature of the Bees and Wasps.\\\\

% \textbf{Tone, Style, and Length:}\\
% The tone is neutral, leaning towards wisdom and justice. The style is straightforward, using a fable-like structure to convey the message. The story is concise, apt for the message it aims to convey.\\\\

% \textbf{Additional Characters or Plot Twists:}\\
% The Ant's introduction serves as a slight twist, redirecting the court's attention momentarily before the old Bee's suggestion.\\\\

% \textbf{Author's Perspective and Techniques in Detail:}\\
% The author uses a simple narrative to convey a profound message about the inherent nature of beings and the wisdom of true justice. By personifying insects, the story becomes relatable and engaging.\\\\

% \textbf{Conclusion:}\\
% ``The Bees and Wasps, and the Hornet" is a tale that emphasizes the importance of wisdom, justice, and understanding one's true nature. Through a simple dispute over honey, the story imparts a lesson on how genuine attributes and abilities cannot be concealed, and true justice prevails when one understands the core nature of things.\\
% \bottomrule
% \end{tabular}
% \caption{Optimized structured prompt for the story ``The Bees and Wasps, and the Hornet"}
% \label{tab:prompt_sample}
% \end{table*}



% \begin{table*}[ht]
% \centering
% \renewcommand{\arraystretch}{0.8}
% \begin{tabular}{@{}>{\raggedright\arraybackslash}p{\linewidth}@{}}
% \toprule
% \textbf{STORY ANALYSIS}\\\\
% \textbf{Title: }[...]\\\\
% \textbf{Introduction:} Provide a brief summary of the story, capturing its essence and main themes.\\\\
% \textbf{Character and Relationship Descriptions in Predicates:}\\
% Emphasize the use of predicates to describe characters and their relationships. For instance:\\
% - is\_kid(X) - X is a kid.\\
% - is\_afraid\_of(X, Y) - X is afraid of Y.\\\\
% \textbf{Initial Conditions:}\\
% Detail the starting conditions, focusing on:\\
% 1. Characters: Introduce the main characters using predicates.\\
% 2. Setting: Describe where and when the story begins.\\
% 3. State of Affairs: Highlight the situations or challenges the characters face at the outset.\\\\
% \textbf{Significant Events in Predicates:}\\
% List and describe the major events, using predicates to structure the information. Emphasize how these events drive the narrative forward.\\\\
% \textbf{Timeline of Events:}\\
% Draft a chronological representation of the significant events, ensuring a clear flow of the story's progression.\\\\
% \textbf{State and Belief Changes for Each Character:}\\
% This section is crucial. For each major event, list how the states, beliefs, or emotions of characters are affected. For instance:\\
% - Kid: Starts confident $\rightarrow$ Becomes fearful $\rightarrow$ Feels relief.\\\\
% \textbf{Literary Techniques in Depth:}\\
% Delve deeper into the literary techniques. For each identified technique, explain how it's employed in the story and its impact on the narrative.\\\\
% \textbf{Setting Details and Atmospheric Details:}\\
% Provide a more vivid and detailed description of the setting. Capture the ambiance, the time of day, and any sensory details that immerse the reader in the story's world.\\\\
% \textbf{Character Backstories and Motivations:}\\
% Dive deeper into any past experiences or histories of the characters that might be relevant to the story. Understand and describe the deeper motivations or desires of the characters.\\\\
% \textbf{Interactions and Dialogues:}\\
% Highlight and analyze significant dialogues or interactions. Explain their relevance and how they contribute to character development or plot progression.\\\\
% \textbf{Tone, Style, and Length:}\\
% Discuss the tone and style in more detail, highlighting how they align with the story's themes. Comment on the story's length and its appropriateness for the conveyed message.\\\\
% \textbf{Additional Characters or Plot Twists:}\\
% Detail any supplementary characters or unexpected plot developments, emphasizing their relevance and impact on the story.\\\\
% \textbf{Author's Perspective and Techniques in Detail:}\\
% Analyze the author's techniques, perspective shifts, and unique storytelling methods in depth. Discuss how these techniques serve the narrative and engage the reader.\\\\
% \textbf{Conclusion:}\\
% Conclude with a comprehensive summary of the story's themes, messages, and lessons. Reflect on the lasting impacts or takeaways for the reader.\\
% \bottomrule
% \end{tabular}
% \caption{Optimized prompt used in the narrative generation process}
% \label{tab:prompt_sample}
% \end{table*}

\section{Evaluation}
% It aims to extract a comprehensive structural narrative profile, allowing for an in-depth evaluation, judgment, and visualization of story elements. These reports holistically cover character states, event timelines, literary techniques, setting nuances, character backstories, interactions, and authorial perspectives. The report coverage adeptly navigates the spectrum of narrative styles, from the succinct allegories found in Aesop's Fables to the detailed tapestry of epics like 20,000 Leagues Under the Sea. 

The goal behind developing this new dataset is to furnish rich data for comprehensive analysis of storytelling methods and to test machine learning models on their ability to rebuild stories solely from annotations. Our evaluation approach is twofold: firstly, we examine the efficacy of our system by assessing how well Agent B provides constructive feedback to improve the quality and applicability of the information extracted. Secondly, we demonstrate the dataset's practical use by visualizing and analyzing two fables, highlighting its capacity to yield deeper insights into narrative techniques and storytelling research.

% In this section we would like to judge the different annotations obtained through the baseline prompt as well as the optimized prompt. We adopt the task applied in \texttt{Agent B} in the Stage II, which is to reconstruct the original story based on the current annotation. We want the language model to be as close as possible to the plot, writing style, and sequence of events of the story to ensure that the annotation provides correct and sufficient information.

\subsection{Effectiveness of the Framework}
% Based on different prompts, we randomly select 5 stories from the Aesop's Fables part of the story corpus as input. The reason for selecting this series of stories is that the narrative logic of each story is relatively simple and the length of the story is relatively appropriate, which can ensure that the public readers can understand the story correctly without hindrance. 

% Since Agent B use the story reconstruction task to tune Agent A's prompt, though iterations, we should be able to observe better reconstructed stories in the sense that they are closer to the original story. 

% For the purpose of comparing the annotation comprehensiveness and validity given from different prompt, we turned our attention to Aesop's Fables within the story corpus. This selection was driven by several factors. Firstly, the narrative structure inherent in these fables is inherently simple, allowing for clear comprehension. Additionally, the length of these tales is relatively consistent, ensuring that any potential ambiguities or complexities do not hinder a reader's understanding. 

% From this selection, we randomly chose five distinct stories. Each of these stories was then subjected to reconstruction based on two distinct prompts. This yielded two separate reconstructions per story. 

% With these anonymized reconstructions, we proceeded to the evaluation phase. Subjects were presented with both reconstructions for each story and were tasked with determining their respective degrees of approximation to the original narrative or what we refer to as the "ground truth". Their evaluations were to be based on a comprehensive understanding of the reconstructed story in relation to the original. Each comparison was scored on a scale ranging from 1 to 5, where a score of 1 signifies a weak approximation and a score of 5 represents a near-perfect reconstruction.

% The human-evaluated score for each of the five stories, based on their similarity to the original, are detailed in Table~\ref{tab:evaluation_scores}. In this table, two columns of scores are represented - one for the reconstruction using \textit{optimized} prompted with $N_{iter} = 5$ and the other for the reconstruction using \textit{baseline} prompt.

% As seen, the optimized approach consistently yielded higher approximation scores across all stories, thereby suggesting its superior capability in preserving the essential narrative elements and structure of the original tales from Aesop's Fables. The average scores for the optimized method generally hover around the 3.75 to 4.00 range, showcasing its efficacy in reconstructing the stories with a high degree of accuracy. In contrast, the baseline method's scores fluctuated more widely, indicating a potential inconsistency in its performance.

% Agent B's role in refining Agent A's prompt through iterative story reconstruction is anticipated to yield increasingly accurate story reconstructions, approaching the original stories. 


To assess our framework's effectiveness in refining prompts and generating effective annotations, we chose five Aesop's Fables from our corpus, leveraging their simplicity and consistent length for a straightforward comparative analysis. The selected stories are ``The Kid and the Wolf," ``The Lion and the Ass," ``The Bees, the Wasps, and the Hornet," ``The Bat and the Weasels," and ``The Wolf and the Shepherd," labeled as story IDs 1-5 in Table~\ref{tab:merged_scores}. These fables were picked for their varied themes. We compared Agent B's story reconstruction capabilities using both the optimized prompt, refined through five iterations and the original baseline prompt for each fable.



\paragraph{Human Evaluation}
To evaluate the effectiveness of our framework, we recruited 15 college students as participants to compare the reconstructed stories with the originals. Participants rated the similarity on a scale from 1 (poor match) to 5 (almost perfect reconstruction).

The results, presented in Table~\ref{tab:merged_scores}, reveal that the optimized prompts consistently achieved higher scores in capturing the original stories' essential narrative elements and structure compared to the baseline prompts. The optimized prompts received average scores between 3.75 and 4.00, suggesting a high fidelity in story reconstruction. In contrast, the baseline prompts showed more varied outcomes, indicating possible inconsistencies in their effectiveness.
% \begin{table}
%     \centering
%     \begin{tabular}{lcc}
%         \hline
%         \textbf{Story ID} & \textbf{Optimized Prompt} & \textbf{Baseline Prompt} \\
%         \hline
%         1 & 4.00 & 2.92 \\
%         2 & 3.75 & 3.17 \\
%         3 & 3.58 & 3.25 \\
%         4 & 3.75 & 3.58 \\
%         5 & 3.92 & 2.42 \\
%         \hline
%     \end{tabular}
%     \caption{Averaged Similarity Scores: Comparing Story Reconstructions to the Original Text}
%     \label{tab:evaluation_scores}
% \end{table}

% \begin{table}
%     \centering
%     \begin{tabular}{lccc}
%         \hline
%         \textbf{Story ID} & \textbf{Ground Truth} & \textbf{Baseline Prompt} & \textbf{Optimized Prompt} \\
%         \hline
%         1 & 20.03 & 14.74 & \textbf{13.43} \\
%         2 & 15.78 & 12.36 & \textbf{11.47} \\
%         3 & 15.69 & 13.82 & \textbf{12.26} \\
%         4 & 20.03 & 11.42 & \textbf{9.77} \\
%         5 & \textbf{10.75} & 11.05 & 13.11 \\
%         \hline
%     \end{tabular}
%     \caption{Perplexity Scores for 5 Stories}
%     \label{tab:perplexity_scores}
% \end{table}


\paragraph{Automated Evaluation}
% In addition to the approximation scores based on human evaluations, we also quantitatively analyzed the narrative structures of the reconstructions by measuring their perplexity. Perplexity serves as a useful metric to gauge how well a probability distribution predicts a sample; in the context of our study, it gives insights into how closely the reconstructed stories align with the language patterns observed in the original tales.

% The perplexity results are consolidated in Table \ref{tab:perplexity_scores}. For each story, the perplexity is given for the ground truth, the baseline reconstruction, and the reconstruction obtained using the optimized prompt with $N_{iter} = 5$.

% A cursory glance at the results indicates that for the majority of the stories (four out of five), the optimized method, as indicated by the lowest perplexity scores, was the closest in alignment with the linguistic patterns of the ground truth. This reinforces our earlier observation, where the optimized approach was perceived to provide a better approximation to the original narrative in human evaluations. However, it is worth noting the exception in Story 5 where the ground truth exhibited the lowest perplexity, suggesting that while the optimized method is generally effective, there might be certain narrative contexts where it doesn't outperform the original or even the baseline.
% Beyond human evaluation, we further assessed the narrative structures of the reconstructions by calculating their perplexity, a reliable metric to understand how well a probability distribution predicts a given sample, and in our context, to measure the alignment of reconstructed stories with the language patterns of the original tales.

% The results, presented in Table \ref{tab:perplexity_scores}, showcase the perplexity scores for the ground truth, baseline reconstruction, and the optimized reconstruction at $N_{iter} = 5$ for each story. From the results, it’s clear that in the majority of cases (four out of five stories), the optimized method yields the lowest perplexity scores, suggesting a stronger alignment with the linguistic patterns of the original narratives. This aligns with our previous findings from human evaluations, affirming the optimized approach’s effectiveness. %However, it’s important to highlight the anomaly in Story 5, where the ground truth scored the lowest perplexity, indicating that the optimized method might not consistently outperform across all narrative contexts.

% Perplexity, a measure of a probability distribution's predictive accuracy, reflects the consistency of language patterns and helps evaluate language models~\cite{jelinek1977perplexity}. In our narrative reconstruction study, we utilized perplexity to assess the alignment between reconstructed stories and the linguistic patterns of the original tales. The quantitative analysis of perplexity provided insights into the model's prediction quality and language consistency, highlighting the effectiveness of optimized prompts in capturing the essential narrative elements and linguistic style of the original stories, as corroborated by our results in Table \ref{tab:merged_scores}. Although the optimized prompts generally led to lower perplexity and a closer approximation to the original stories, some instances indicated room for further improvement.


% We employed perplexity as a measure to evaluate the predictive accuracy of probability distributions, a technique reflective of the language model's consistency with linguistic patterns~\cite{jelinek1977perplexity}. Perplexity offers valuable insights into the alignment between the reconstructed narratives and the original tales, serving as an indicator of the model's capability to predict and replicate the linguistic style and essential narrative elements. Our quantitative analysis, detailed in Table \ref{tab:merged_scores}, leverages perplexity to underscore the effectiveness of optimized prompts in closely mirroring the linguistic style and narrative essence of the original stories. Notably, the optimized prompts generally resulted in lower perplexity scores, suggesting a higher degree of confidence in the generated text and a closer approximation to the original narratives.
We used perplexity to evaluate the language model's consistency with linguistic patterns in the reconstructed narratives~\cite{jelinek1977perplexity}. As shown in Table \ref{tab:merged_scores}, optimized prompts generally resulted in lower perplexity scores, indicating a closer approximation to the original stories' linguistic style and narrative essence.

The improvement in perplexity with optimized prompts indicates that our method effectively guides the language model to capture key components and stylistic features of the original stories. Higher perplexity scores associated with baseline prompts reflect the model's uncertainty and the challenges of maintaining consistency without precise guidance. The ground truth column in Table~\ref{tab:merged_scores} provides a reference point for the perplexity of continuing the original story, which typically presents higher values due to the open-ended nature of the task. The comparative analysis of perplexity scores validates our approach, demonstrating that optimized prompts enable the language model to generate text with greater confidence and focus, affirming the efficacy of our optimization method in enhancing narrative reconstruction.

\begin{figure}
    \centering
    \includegraphics[width=1\columnwidth]{figure/emotion_2_larger.jpg}
    \caption{Timeline for ``The Cock and the Fox'' story}
    \label{fig:emotion2}
\end{figure}

\subsection{Example Applications }
% In this section, we present a method for visualizing the emotional and situational arcs of characters in short stories, specifically fables, based on the ``StorySence'' corpus. For each story, we extract key events and the associated emotional or situational changes of the main characters. These events were mapped onto the x-axis in chronological order, while the y-axis was used to represent the emotional or situational status of the characters. Positive y-values indicate positive emotions or situations (e.g., confidence, safety), and negative y-values indicate negative emotions or situations (e.g., danger, fear). Different line styles were used to represent different characters, with events marked clearly on the timeline for context.

% \paragraph{Case study: ``The Wolf and the Kid"}
% As shown in Figure~\ref{fig:emotion1}, the visualization commences with the Kid exuding confidence, illustrated by an upward blue trajectory. However, as the evening shadows lengthen and danger looms, this line descends sharply, reflecting the Kid's growing trepidation. In contrast, the Wolf's line, depicted in red, rises steeply, mirroring its escalating anticipation. But, as the narrative nears its climax, the Wolf's line plunges downward, representing its dawning regret. The Shepherd Dogs, visualized in green, maintain a steady line until their pivotal intervention, whereupon their trajectory ascends, underscoring their role as saviors.
% In this section, we showcase the results of visualizing the emotional and situational trajectories of characters in a fable, derived from the ``StorySense'' corpus. Figure~\ref{fig:emotion1} illustrates the Kid starting off confident, then rapidly becoming scared as danger approaches. Concurrently, the Wolf’s anticipation grows, then turns to regret towards the climax. The Shepherd Dogs remain constant, rising sharply during their timely intervention.

% % \paragraph{Case study: ``The Cock and the Fox"}
% % Representing in Figure~\ref{fig:emotion2}, the Cock, represented by a blue line, starts with a neutral emotional state, as it roosts. However, the line ascends as the Cock crows confidently, only to oscillate with caution and strategic thinking upon spotting the Fox. The Fox, shown in red, undergoes a roller-coaster of emotions—from opportunism to exhilaration, and finally to fear, each shift clearly delineated in its fluctuating line. The Dogs, though peripheral to the core events, are depicted in green. Their line rises slightly when mentioned, highlighting their indirect influence on the story's outcome.

% Figure~\ref{fig:emotion2} shows the Cock's emotional journey, starting neutral, becoming confident, and then cautious upon Fox’s appearance. The Fox’s emotional state fluctuates from opportunistic to exhilarated, then fearful. The Dogs, represented in green, have a minor role, with a slight rise in their line indicating their subtle impact on the story.

% % From the visualizations of both stories, we can draw high-level pattern similarities without delving into the stories themselves. In each case, there’s a clear triumph of an underdog, using cleverness and strategy to overcome obstacles. The emotional journeys of the protagonists and antagonists are intertwined, reflecting inversely on one another. External characters, like the Dogs in both stories, play crucial roles in resolving conflicts, even from the periphery. And, true to fable tradition, both stories weave in moral lessons, highlighting virtues such as quick thinking, the unpredictability of life, and showcasing that strength isn’t merely physical.

% From the visualizations of both stories, we can draw high-level pattern similarities without delving into the stories themselves. In each case, there’s a clear triumph of an underdog, using cleverness and strategy to overcome obstacles. The emotional journeys of the protagonists and antagonists are intertwined, reflecting inversely on one another. External characters, like the Dogs in both stories, play crucial roles in resolving conflicts, even from the periphery. And, true to fable tradition, both stories weave in moral lessons, highlighting virtues such as quick thinking, and the unpredictability of life, and showcasing that strength isn’t merely physical.

% In this section, we demonstrate the application of our annotated information by visualizing the emotional and situational arcs of characters in two selected fables: "The Wolf and the Kid," illustrated in Figure~\ref{fig:emotion1}, and "The Cock and the Fox," depicted in Figure~\ref{fig:emotion2}. These visualizations are based on annotations that track the changes in emotions or situations of the main characters in conjunction with the pivotal events of each story.

% To create the visualizations, we synthesized information from "Character and Relationship Descriptions in Predicates" to identify the characters, "Timeline of Events" for x-axis events, and "State and Belief Changes for Each Character" for the emotional or situational states. For numerical translation of these states, we prompted GPT-4 with a specific instruction to convert emotional or situational changes into numbers ranging from -4 (extremely negative) to 4 (extremely positive). This conversion facilitated plotting the emotional and situational arcs of characters against the story timeline, enabling us to visually represent the narrative's evolving dynamics and provide a quantified view of the characters' experiences as the story unfolds.

% On the y-axis, positive values represent positive emotions or situations (e.g., confidence, safety), while negative values indicate negative emotions or situations (e.g., danger, fear). Different line styles denote different characters, and key events are marked on the timeline to contextualize the emotional shifts. This method highlights the nuanced emotional journeys of the characters and the complexity of the narrative structure, showcasing the rich insights gained from our annotated dataset.

In this section, we showcase how we use our annotations to visualize the emotional and situational arcs of characters in "The Wolf and the Kid" (Figure~\ref{fig:emotion1}) and "The Cock and the Fox" (Figure~\ref{fig:emotion2}), aligning with key story events. We combined character descriptions, event timelines, and character states from our annotations, then used GPT-4 to translate these into a -4 to 4 scale, indicating the intensity of emotional or situational changes. This approach allowed us to plot characters' emotional journeys and situational changes on the story timeline, visually capturing the narrative's dynamics and the characters' experiences.

Positive and negative values on the y-axis signal positive or negative emotions and situations, respectively, with different line styles representing various characters and significant events marked for context. 

\begin{figure}
    \centering
    \includegraphics[width=1\columnwidth]{figure/emotion_1_larger.jpg}
    \caption{Timeline for ``The Wolf and the Kid'' story}
    \label{fig:emotion1}
\end{figure}

% In "The Wolf and the Kid," depicted in Figure~\ref{fig:emotion1}, the narrative tracks the Kid's shift from confidence to fear as danger nears, while the Wolf transitions from eager anticipation to regret at the story's peak. The Shepherd Dogs maintain a steady presence, with a sharp increase in action during their decisive intervention.

% "The Cock and the Fox," illustrated in Figure~\ref{fig:emotion2}, outlines the Cock's emotional arc from neutrality to confidence and eventually caution when faced with the Fox. The Fox experiences a change from seeing an opportunity to excitement and finally fear. The Dogs play a less prominent role, with a slight uptick in their graph representing a minor influence on the plot.

These visualizations underscore common themes across both stories: the victory of the seemingly weaker party through wit, the intertwined emotional paths of the main characters, and the significant influence of secondary characters. Additionally, these tales convey moral lessons on the value of quick thinking and life's inherent uncertainties.

% \section{Conclusion and Future Work}
% In our study, we introduced an innovative two-agent annotation system aimed at enhancing story annotation and reconstruction. While the small sample size for human evaluation poses constraints, the promising results and the valuable dataset generated underscore the system's potential. 
% % Moving forward, expanding participant diversity, exploring a wider range of narratives, and delving into various domains of story analysis will not only validate our framework's versatility but also enrich our dataset's depth and applicability.
% Moving forward, our method's flexibility and adaptability offer exciting avenues for expansion. Notably, the potential to customize initial prompts and adjust the number of iterations allows for tailored applications across a variety of story lengths and complexities. Moreover, the fundamental model-agnostic nature of our approach suggests that, with minor adjustments, it can be adapted to various language models, broadening its applicability and impact.

% \section{Future Work}
% Future research should focus on increasing participant diversity and sample size in human evaluation studies, investigating the application of our approach to a wider range of narratives and domains, and adapting our method to various language models. Refining the annotation schema, exploring additional dimensions of story analysis, and developing automated evaluation metrics are also important areas for future work. By addressing these aspects, we aim to strengthen the foundation laid by our work and contribute to the advancement of natural language understanding and generation in the context of storytelling.


\section{Conclusion and Future Work}
% In our study, we introduced a two-agent annotation system aimed at enhancing story annotation and reconstruction. While the small sample size for human evaluation poses constraints, the promising results and the valuable dataset generated underscore the system's potential. Our method's flexibility and adaptability offer avenues for expansion, allowing for tailored applications across a variety of story lengths and complexities.
This study introduces a multi-agent system for story annotation and the creation of the StorySense corpus, marking significant advancements in using Large Language Models (LLMs) for narrative analysis and story reconstruction. Our work illustrates LLMs' capability to deeply understand and accurately reconstruct narratives, showcasing improved annotation precision and efficiency.

Looking ahead, our research agenda includes expanding human evaluation diversity and sample size, broadening the application of our methods across various stories, and adapting to different language models. Refining the annotation schema and exploring new dimensions of story analysis are also key. Moreover, developing specialized automated evaluation metrics will enhance our ability to assess narrative reconstruction beyond technical accuracy to include creativity and emotional depth.

By advancing these areas, we aim to further our contributions to natural language understanding and generation within storytelling, bridging LLMs' potential with the rich domain of storytelling.

\section{Bibliographical References}\label{sec:reference}

\bibliographystyle{lrec-coling2024-natbib}
\bibliography{lrec-coling2024-example}

% \section{Language Resource References}
% \label{lr:ref}
% \bibliographystylelanguageresource{lrec-coling2024-natbib}
% \bibliographylanguageresource{languageresource}
\appendix



\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:
