From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting
Griffin Adams, Alex Fabbri, Faisal Ladhak, Eric Lehman, Noémie Elhadad
Abstract
Selecting the “right” amount of information to include in a summary is a difficult task. A good summary should be detailed and entity-centric without being overly dense and hard to follow. To better understand this tradeoff, we solicit increasingly dense GPT-4 summaries with what we refer to as a “Chain of Density” (CoD) prompt. Specifically, GPT-4 generates an initial entity-sparse summary before iteratively incorporating missing salient entities without increasing the length. Summaries generated by CoD are more abstractive, exhibit more fusion, and have less of a lead bias than GPT-4 summaries generated by a vanilla prompt. We conduct a human preference study on 100 CNN DailyMail articles and find that humans prefer GPT-4 summaries that are more dense than those generated by a vanilla prompt and almost as dense as human written summaries. Qualitative analysis supports the notion that there exists a tradeoff between informativeness and readability. 500 annotated CoD summaries, as well as an extra 5,000 unannotated summaries, are freely available on HuggingFace (https://huggingface.co/datasets/griffin/chain_of_density).- Anthology ID:
- 2023.newsum-1.7
- Volume:
- Proceedings of the 4th New Frontiers in Summarization Workshop
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Yue Dong, Wen Xiao, Lu Wang, Fei Liu, Giuseppe Carenini
- Venue:
- NewSum
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 68–74
- Language:
- URL:
- https://aclanthology.org/2023.newsum-1.7
- DOI:
- 10.18653/v1/2023.newsum-1.7
- Cite (ACL):
- Griffin Adams, Alex Fabbri, Faisal Ladhak, Eric Lehman, and Noémie Elhadad. 2023. From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 68–74, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting (Adams et al., NewSum 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.newsum-1.7.pdf