Abstract
We propose a simple but highly effective automatic evaluation measure of summarization, pruned Basic Elements (pBE). Although the BE concept is widely used for the automated evaluation of summaries, its weakness is that it redundantly matches basic elements. To avoid this redundancy, pBE prunes basic elements by (1) disregarding frequency count of basic elements and (2) reducing semantically overlapped basic elements based on word similarity. Even though it is simple, pBE outperforms ROUGE in DUC datasets in most cases and achieves the highest rank correlation coefficient in TAC 2011 AESOP task.- Anthology ID:
- N18-2104
- Volume:
- Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)
- Month:
- June
- Year:
- 2018
- Address:
- New Orleans, Louisiana
- Editors:
- Marilyn Walker, Heng Ji, Amanda Stent
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 661–666
- Language:
- URL:
- https://aclanthology.org/N18-2104
- DOI:
- 10.18653/v1/N18-2104
- Cite (ACL):
- Ukyo Honda, Tsutomu Hirao, and Masaaki Nagata. 2018. Pruning Basic Elements for Better Automatic Evaluation of Summaries. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 661–666, New Orleans, Louisiana. Association for Computational Linguistics.
- Cite (Informal):
- Pruning Basic Elements for Better Automatic Evaluation of Summaries (Honda et al., NAACL 2018)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/N18-2104.pdf
- Data
- Universal Dependencies