@inproceedings{deering-penn-2025-analysis,
    title = "An Analysis of Scoring Methods for Reranking in Large Language Model Story Generation",
    author = "Deering, Megan  and
      Penn, Gerald",
    editor = "Padmakumar, Vishakh  and
      Gero, Katy  and
      Wambsganss, Thiemo  and
      Sterman, Sarah  and
      Huang, Ting-Hao  and
      Zhou, David  and
      Chung, John",
    booktitle = "Proceedings of the Fourth Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2025)",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico, US",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.in2writing-1.10/",
    pages = "109--120",
    ISBN = "979-8-89176-239-8",
    abstract = "Outline-conditioned story generation using Large Language Models (LLMs) offers a promising approach for automating narrative creation. Some outline-conditioned story generation methods use automatic scoring during the generation process in order to improve the story quality. However, current research has shown that automatic scoring is not ideal for assessing story quality. This paper evaluates three proposed automatic story-scoring methods to improve the reranking of outputs during the generation process. These scoring methods leverage different prompting strategies and fine-tuning techniques to enhance the accuracy and relevance of the assessments. By experimenting with these approaches within a beam search framework, we aim to identify the most effective methods for optimizing story-generation outcomes. While we have found no significant overall difference between these methods in terms of their agreement with human ratings during story generation, the overall story ratings by human evaluators are average. These findings motivate the need for improved automatic scoring techniques and datasets while also indicating that simpler, more easily implementable scoring methods for reranking perform comparably to more complex approaches."
}