Abstract
Neural abstractive document summarization is commonly approached by models that exhibit a mostly extractive behavior. This behavior is facilitated by a copy-attention which allows models to copy words from a source document. While models in the mostly extractive news summarization domain benefit from this inductive bias, they commonly fail to paraphrase or compress information from the source document. Recent advances in transfer-learning from large pretrained language models give rise to alternative approaches that do not rely on copy-attention and instead learn to generate concise and abstractive summaries. In this paper, as part of the TL;DR challenge, we compare the abstractiveness of summaries from different summarization approaches and show that transfer-learning can be efficiently utilized without any changes to the model architecture. We demonstrate that the approach leads to a higher level of abstraction for a similar performance on the TL;DR challenge tasks, enabling true natural language compression.- Anthology ID:
- W19-8665
- Volume:
- Proceedings of the 12th International Conference on Natural Language Generation
- Month:
- October–November
- Year:
- 2019
- Address:
- Tokyo, Japan
- Editors:
- Kees van Deemter, Chenghua Lin, Hiroya Takamura
- Venue:
- INLG
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 516–522
- Language:
- URL:
- https://aclanthology.org/W19-8665
- DOI:
- 10.18653/v1/W19-8665
- Cite (ACL):
- Sebastian Gehrmann, Zachary Ziegler, and Alexander Rush. 2019. Generating Abstractive Summaries with Finetuned Language Models. In Proceedings of the 12th International Conference on Natural Language Generation, pages 516–522, Tokyo, Japan. Association for Computational Linguistics.
- Cite (Informal):
- Generating Abstractive Summaries with Finetuned Language Models (Gehrmann et al., INLG 2019)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/W19-8665.pdf