SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling
Forrest Bao, Ge Luo, Hebi Li, Minghui Qiu, Yinfei Yang, Youbiao He, Cen Chen
Abstract
Canonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain. Recently, there have been a growing number of efforts to alleviate either or both of the two drawbacks. In this paper, we present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries. Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries. In cross-domain tests, our strategy outperforms baselines with promising improvements, and show a great advantage in gauging linguistic qualities over all metrics.- Anthology ID:
- 2022.naacl-main.175
- Volume:
- Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, United States
- Editors:
- Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2450–2458
- Language:
- URL:
- https://aclanthology.org/2022.naacl-main.175
- DOI:
- 10.18653/v1/2022.naacl-main.175
- Cite (ACL):
- Forrest Bao, Ge Luo, Hebi Li, Minghui Qiu, Yinfei Yang, Youbiao He, and Cen Chen. 2022. SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2450–2458, Seattle, United States. Association for Computational Linguistics.
- Cite (Informal):
- SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling (Bao et al., NAACL 2022)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2022.naacl-main.175.pdf
- Code
- forrestbao/suenes
- Data
- BigPatent, BillSum, NEWSROOM