Abstract
News headlines often evoke sentiment by intentionally portraying entities in particular ways, making targeted sentiment analysis (TSA) of headlines a worthwhile but difficult task. Due to its subjectivity, creating TSA datasets can involve various annotation paradigms, from descriptive to prescriptive, either encouraging or limiting subjectivity. LLMs are a good fit for TSA due to their broad linguistic and world knowledge and in-context learning abilities, yet their performance depends on prompt design. In this paper, we compare the accuracy of state-of-the-art LLMs and fine-tuned encoder models for TSA of news headlines using descriptive and prescriptive datasets across several languages. Exploring the descriptive–prescriptive continuum, we analyze how performance is affected by prompt prescriptiveness, ranging from plain zero-shot to elaborate few-shot prompts. Finally, we evaluate the ability of LLMs to quantify uncertainty via calibration error and comparison to human label variation. We find that LLMs outperform fine-tuned encoders on descriptive datasets, while calibration and F1-score generally improve with increased prescriptiveness, yet the optimal level varies.- Anthology ID:
- 2024.wassa-1.27
- Volume:
- Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Orphée De Clercq, Valentin Barriere, Jeremy Barnes, Roman Klinger, João Sedoc, Shabnam Tafreshi
- Venues:
- WASSA | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 329–343
- Language:
- URL:
- https://aclanthology.org/2024.wassa-1.27
- DOI:
- 10.18653/v1/2024.wassa-1.27
- Cite (ACL):
- Jana Juroš, Laura Majer, and Jan Snajder. 2024. LLMs for Targeted Sentiment in News Headlines: Exploring the Descriptive-Prescriptive Dilemma. In Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 329–343, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- LLMs for Targeted Sentiment in News Headlines: Exploring the Descriptive-Prescriptive Dilemma (Juroš et al., WASSA-WS 2024)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2024.wassa-1.27.pdf