@inproceedings{bissell-etal-2025-theoretical,
    title = "A Theoretical Framework for Evaluating Narrative Surprise in Large Language Models",
    author = "Bissell, Annaliese  and
      Paulin, Ella  and
      Piper, Andrew",
    editor = "Clark, Elizabeth  and
      Lal, Yash Kumar  and
      Chaturvedi, Snigdha  and
      Iyyer, Mohit  and
      Brei, Anneliese  and
      Modi, Ashutosh  and
      Chandu, Khyathi Raghavi",
    booktitle = "Proceedings of the The 7th Workshop on Narrative Understanding",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.wnu-1.7/",
    pages = "26--35",
    ISBN = "979-8-89176-247-3",
    abstract = "Narrative surprise is a core element of storytelling for engaging audiences, and yet it remains underexplored in the context of large language models (LLMs) and narrative generation. While surprise arises from events that deviate from expectations while maintaining retrospective coherence, current computational approaches lack comprehensive frameworks to evaluate this phenomenon. This paper presents a novel framework for assessing narrative surprise, drawing on psychological theories of narrative comprehension and surprise intensity. We operationalize six criteria{---}initiatoriness, immutability violation, predictability, post-dictability, importance, and valence{---}to measure narrative surprise in story endings. Our study evaluates 120 story endings, generated by both human authors and LLMs, across 30 mystery narratives. Through a ranked-choice voting methodology, we identify significant correlations between reader preferences and four of the six criteria. Results underscore the continuing advantage of human-authored endings in achieving compelling narrative surprise, while also revealing significant progress in LLM-generated narratives."
}