Abstract
This paper tackles automation of the pyramid method, a reliable manual evaluation framework. To construct a pyramid, we transform human-made reference summaries into extractive reference summaries that consist of Elementary Discourse Units (EDUs) obtained from source documents and then weight every EDU by counting the number of extractive reference summaries that contain the EDU. A summary is scored by the correspondences between EDUs in the summary and those in the pyramid. Experiments on DUC and TAC data sets show that our methods strongly correlate with various manual evaluations.- Anthology ID:
- D18-1450
- Volume:
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Month:
- October-November
- Year:
- 2018
- Address:
- Brussels, Belgium
- Editors:
- Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
- Venue:
- EMNLP
- SIG:
- SIGDAT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4177–4186
- Language:
- URL:
- https://aclanthology.org/D18-1450
- DOI:
- 10.18653/v1/D18-1450
- Cite (ACL):
- Tsutomu Hirao, Hidetaka Kamigaito, and Masaaki Nagata. 2018. Automatic Pyramid Evaluation Exploiting EDU-based Extractive Reference Summaries. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4177–4186, Brussels, Belgium. Association for Computational Linguistics.
- Cite (Informal):
- Automatic Pyramid Evaluation Exploiting EDU-based Extractive Reference Summaries (Hirao et al., EMNLP 2018)
- PDF:
- https://preview.aclanthology.org/naacl24-info/D18-1450.pdf