Abstract
Evaluating machine-generated summaries without a human-written reference summary has been a need for a long time. Inspired by preference labeling in existing work of summarization evaluation, we propose to judge summary quality by learning the preference rank of summaries using the Bradley-Terry power ranking model from inferior summaries generated by corrupting base summaries. Extensive experiments on several datasets show that our weakly supervised scheme can produce scores highly correlated with human ratings.- Anthology ID:
- 2022.coling-1.515
- Volume:
- Proceedings of the 29th International Conference on Computational Linguistics
- Month:
- October
- Year:
- 2022
- Address:
- Gyeongju, Republic of Korea
- Editors:
- Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, Seung-Hoon Na
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 5896–5903
- Language:
- URL:
- https://aclanthology.org/2022.coling-1.515
- DOI:
- Cite (ACL):
- Ge Luo, Hebi Li, Youbiao He, and Forrest Sheng Bao. 2022. PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5896–5903, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
- Cite (Informal):
- PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment (Luo et al., COLING 2022)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2022.coling-1.515.pdf
- Code
- nkwbtb/prefscore
- Data
- BigPatent, BillSum, NEWSROOM