Abstract
We present a new framework for evaluating extractive summarizers, which is based on a principled representation as optimization problem. We prove that every extractive summarizer can be decomposed into an objective function and an optimization technique. We perform a comparative analysis and evaluation of several objective functions embedded in well-known summarizers regarding their correlation with human judgments. Our comparison of these correlations across two datasets yields surprising insights into the role and performance of objective functions in the different summarizers.- Anthology ID:
- P17-2005
- Volume:
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- July
- Year:
- 2017
- Address:
- Vancouver, Canada
- Editors:
- Regina Barzilay, Min-Yen Kan
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 26–31
- Language:
- URL:
- https://aclanthology.org/P17-2005
- DOI:
- 10.18653/v1/P17-2005
- Cite (ACL):
- Maxime Peyrard and Judith Eckle-Kohler. 2017. A Principled Framework for Evaluating Summarizers: Comparing Models of Summary Quality against Human Judgments. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 26–31, Vancouver, Canada. Association for Computational Linguistics.
- Cite (Informal):
- A Principled Framework for Evaluating Summarizers: Comparing Models of Summary Quality against Human Judgments (Peyrard & Eckle-Kohler, ACL 2017)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/P17-2005.pdf
- Code
- UKPLab/acl2017-theta_evaluation_summarization