Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?

Marcio Fonseca, Shay Cohen

Abstract
In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-fine-tuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly fine-tuning remains an open problem for domain-specific applications.
Anthology ID:
2024.findings-acl.508
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8599–8618
Language:
URL:
https://aclanthology.org/2024.findings-acl.508
DOI:
Bibkey:
Cite (ACL):
Marcio Fonseca and Shay Cohen. 2024. Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?. In Findings of the Association for Computational Linguistics ACL 2024, pages 8599–8618, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals? (Fonseca & Cohen, Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.findings-acl.508.pdf
PDF Search