@inproceedings{subbiah-etal-2024-storysumm,
    title = "{STORYSUMM}: Evaluating Faithfulness in Story Summarization",
    author = "Subbiah, Melanie  and
      Ladhak, Faisal  and
      Mishra, Akankshya  and
      Adams, Griffin Thomas  and
      Chilton, Lydia  and
      McKeown, Kathleen",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.emnlp-main.557/",
    doi = "10.18653/v1/2024.emnlp-main.557",
    pages = "9988--10005",
    abstract = "Human evaluation has been the gold standard for checking faithfulness in abstractive summarization. However, with a challenging source domain like narrative, multiple annotators can agree a summary is faithful, while missing details that are obvious errors only once pointed out. We therefore introduce a new dataset, StorySumm, comprising LLM summaries of short stories with localized faithfulness labels and error explanations. This benchmark is for evaluation methods, testing whether a given method can detect challenging inconsistencies. Using this dataset, we first show that any one human annotation protocol is likely to miss inconsistencies, and we advocate for pursuing a range of methods when establishing ground truth for a summarization dataset. We finally test recent automatic metrics and find that none of them achieve more than 70{\%} balanced accuracy on this task, demonstrating that it is a challenging benchmark for future work in faithfulness evaluation."
}