Atomic Self-Consistency for Better Long Form Generations

Raghuveer Thirukovalluru, Yukun Huang, Bhuwan Dhingra

Abstract
Recent work has aimed to improve LLM generations by filtering out hallucinations, thereby improving the precision of the information in responses. Correctness of a long-form response, however, also depends on the recall of multiple pieces of information relevant to the question. In this paper, we introduce Atomic Self-Consistency (ASC), a technique for improving the recall of relevant information in an LLM response. ASC follows recent work, Universal Self-Consistency (USC) in using multiple stochastic samples from an LLM to improve the long-form response. Unlike USC which only focuses on selecting the best single generation, ASC picks authentic subparts from the samples and merges them into a superior composite answer. Through extensive experiments and ablations, we show that merging relevant subparts of multiple samples performs significantly better than picking a single sample. ASC demonstrates significant gains over USC on multiple factoids and open-ended QA datasets - ASQA, QAMPARI, QUEST, ELI5 with ChatGPT and Llama3. Our analysis also reveals untapped potential for enhancing long-form generations using the approach of merging multiple samples.
Anthology ID:
2024.emnlp-main.706
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12681–12694
Language:
URL:
https://preview.aclanthology.org/add_missing_videos/2024.emnlp-main.706/
DOI:
10.18653/v1/2024.emnlp-main.706
Bibkey:
Cite (ACL):
Raghuveer Thirukovalluru, Yukun Huang, and Bhuwan Dhingra. 2024. Atomic Self-Consistency for Better Long Form Generations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12681–12694, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Atomic Self-Consistency for Better Long Form Generations (Thirukovalluru et al., EMNLP 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/add_missing_videos/2024.emnlp-main.706.pdf
PDF Search Fix data