Abstract
With the rise of task-specific pre-training objectives, abstractive summarization models like PEGASUS offer appealing zero-shot performance on downstream summarization tasks. However, the performance of such unsupervised models still lags significantly behind their supervised counterparts. Similarly to the supervised setup, we notice a very high variance in quality among summary candidates from these models while only one candidate is kept as the summary output. In this paper, we propose to re-rank summary candidates in an unsupervised manner, aiming to close the performance gap between unsupervised and supervised models. Our approach improves the unsupervised PEGASUS by up to 7.27% and ChatGPT by up to 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks ; and achieves relative gains of 7.51% (up to 23.73% from XSum to WikiHow) averaged over 30 zero-shot transfer setups (finetuning on a dataset, evaluating on another).- Anthology ID:
- 2023.findings-acl.529
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2023
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 8341–8376
- Language:
- URL:
- https://aclanthology.org/2023.findings-acl.529
- DOI:
- 10.18653/v1/2023.findings-acl.529
- Cite (ACL):
- Mathieu Ravaut, Shafiq Joty, and Nancy Chen. 2023. Unsupervised Summarization Re-ranking. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8341–8376, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Unsupervised Summarization Re-ranking (Ravaut et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/remove-xml-comments/2023.findings-acl.529.pdf