Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation
Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
Abstract
Interpretability and efficiency are two important considerations for the adoption of neural automatic metrics. In this work, we develop strong-performing automatic metrics for reference-based summarization evaluation, based on a two-stage evaluation pipeline that first extracts basic information units from one text sequence and then checks the extracted units in another sequence. The metrics we developed include two-stage metrics that can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. We make the developed tools publicly available at https://github.com/Yale-LILY/AutoACU.- Anthology ID:
- 2023.emnlp-main.1018
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 16360–16368
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.1018
- DOI:
- 10.18653/v1/2023.emnlp-main.1018
- Cite (ACL):
- Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023. Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16360–16368, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (Liu et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2023.emnlp-main.1018.pdf