DeltaScore: Fine-Grained Story Evaluation with Perturbations

Zhuohan Xie, Miao Li, Trevor Cohn, Jey Lau

Abstract
Numerous evaluation metrics have been developed for natural language generation tasks, but their effectiveness in evaluating stories is limited as they are not specifically tailored to assess intricate aspects of storytelling, such as fluency and interestingness. In this paper, we introduce DeltaScore, a novel methodology that uses perturbation techniques for the evaluation of nuanced story aspects. We posit that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations (e.g., the introduction of typos). Given this, we measure the quality of an aspect by calculating the likelihood difference between pre- and post-perturbation states using pre-trained language models. We compare DeltaScore with existing metrics on storytelling datasets from two domains in five fine-grained story aspects: fluency, coherence, relatedness, logicality, and interestingness. DeltaScore demonstrates strong performance, revealing a surprising finding that one specific perturbation proves highly effective in capturing multiple aspects. Source code is available on our GitHub repository.
Anthology ID:
2023.findings-emnlp.353
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5317–5331
Language:
URL:
https://preview.aclanthology.org/icon-24-ingestion/2023.findings-emnlp.353/
DOI:
10.18653/v1/2023.findings-emnlp.353
Bibkey:
Cite (ACL):
Zhuohan Xie, Miao Li, Trevor Cohn, and Jey Lau. 2023. DeltaScore: Fine-Grained Story Evaluation with Perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5317–5331, Singapore. Association for Computational Linguistics.
Cite (Informal):
DeltaScore: Fine-Grained Story Evaluation with Perturbations (Xie et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/icon-24-ingestion/2023.findings-emnlp.353.pdf
PDF Search Fix data