Abstract
Automatically generated summaries can be evaluated along different dimensions, one being how faithfully the uncertainty from the source text is conveyed in the summary. We present a study on uncertainty alignment in automatic summarization, starting from a two-tier lexical and semantic categorization of linguistic expression of uncertainty, which we used to annotate source texts and automatically generate summaries. We collected a diverse dataset including news articles and personal blogs and generated summaries using GPT-4. Source texts and summaries were annotated based on our two-tier taxonomy using a markup language. The automatic annotation was refined and validated by subsequent iterations based on expert input. We propose a method to evaluate the fidelity of uncertainty transfer in text summarization. The method capitalizes on a small amount of expert annotations and on the capabilities of Large language models (LLMs) to evaluate how the uncertainty of the source text aligns with the uncertainty expressions in the summary.- Anthology ID:
- 2024.uncertainlp-1.5
- Volume:
- Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024)
- Month:
- March
- Year:
- 2024
- Address:
- St Julians, Malta
- Editors:
- Raúl Vázquez, Hande Celikkanat, Dennis Ulmer, Jörg Tiedemann, Swabha Swayamdipta, Wilker Aziz, Barbara Plank, Joris Baan, Marie-Catherine de Marneffe
- Venues:
- UncertaiNLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 41–61
- Language:
- URL:
- https://aclanthology.org/2024.uncertainlp-1.5
- DOI:
- Cite (ACL):
- Zahra Kolagar and Alessandra Zarcone. 2024. Aligning Uncertainty: Leveraging LLMs to Analyze Uncertainty Transfer in Text Summarization. In Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024), pages 41–61, St Julians, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Aligning Uncertainty: Leveraging LLMs to Analyze Uncertainty Transfer in Text Summarization (Kolagar & Zarcone, UncertaiNLP-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.uncertainlp-1.5.pdf