Abstract
Numerous evaluation metrics have been developed for natural language generation tasks, but their effectiveness in evaluating stories is limited as they are not specifically tailored to assess intricate aspects of storytelling, such as fluency and interestingness. In this paper, we introduce DeltaScore, a novel methodology that uses perturbation techniques for the evaluation of nuanced story aspects. We posit that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations (e.g., the introduction of typos). Given this, we measure the quality of an aspect by calculating the likelihood difference between pre- and post-perturbation states using pre-trained language models. We compare DeltaScore with existing metrics on storytelling datasets from two domains in five fine-grained story aspects: fluency, coherence, relatedness, logicality, and interestingness. DeltaScore demonstrates strong performance, revealing a surprising finding that one specific perturbation proves highly effective in capturing multiple aspects. Source code is available on our GitHub repository.- Anthology ID:
- 2023.findings-emnlp.353
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5317–5331
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.353
- DOI:
- 10.18653/v1/2023.findings-emnlp.353
- Cite (ACL):
- Zhuohan Xie, Miao Li, Trevor Cohn, and Jey Lau. 2023. DeltaScore: Fine-Grained Story Evaluation with Perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5317–5331, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- DeltaScore: Fine-Grained Story Evaluation with Perturbations (Xie et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2023.findings-emnlp.353.pdf