Abstract
Modeling content importance is an essential yet challenging task for summarization. Previous work is mostly based on statistical methods that estimate word-level salience, which does not consider semantics and larger context when quantifying importance. It is thus hard for these methods to generalize to semantic units of longer text spans. In this work, we apply information theory on top of pre-trained language models and define the concept of importance from the perspective of information amount. It considers both the semantics and context when evaluating the importance of each semantic unit. With the help of pre-trained language models, it can easily generalize to different kinds of semantic units n-grams or sentences. Experiments on CNN/Daily Mail and New York Times datasets demonstrate that our method can better model the importance of content than prior work based on F1 and ROUGE scores.- Anthology ID:
- 2020.emnlp-main.293
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3606–3611
- Language:
- URL:
- https://aclanthology.org/2020.emnlp-main.293
- DOI:
- 10.18653/v1/2020.emnlp-main.293
- Cite (ACL):
- Liqiang Xiao, Lu Wang, Hao He, and Yaohui Jin. 2020. Modeling Content Importance for Summarization with Pre-trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3606–3611, Online. Association for Computational Linguistics.
- Cite (Informal):
- Modeling Content Importance for Summarization with Pre-trained Language Models (Xiao et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/corrections-2024-05/2020.emnlp-main.293.pdf