Do LLMs Think Fast and Slow? A Causal Study on Sentiment Analysis
Zhiheng Lyu, Zhijing Jin, Fernando Gonzalez Adauto, Rada Mihalcea, Bernhard Schölkopf, Mrinmaya Sachan
Abstract
Sentiment analysis (SA) aims to identify the sentiment expressed in a piece of text, often in the form of a review. Assuming a review and the sentiment associated with it, in this paper we formulate SA as a combination of two tasks: (1) a causal discovery task that distinguishes whether a review “primes” the sentiment (Causal Hypothesis C1), or the sentiment “primes” the review (Causal Hypothesis C2); and (2) the traditional prediction task to model the sentiment using the review as input. Using the peak-end rule in psychology, we classify a sample as C1 if its overall sentiment score approximates an average of all the sentence-level sentiments in the review, and as C2 if the overall sentiment score approximates an average of the peak and end sentiments. For the prediction task, we use the discovered causal mechanisms behind the samples to improve the performance of LLMs by proposing causal prompts that give the models an inductive bias of the underlying causal graph, leading to substantial improvements by up to 32.13 F1 points on zero-shot five-class SA.- Anthology ID:
- 2024.findings-emnlp.547
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2024
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 9353–9372
- Language:
- URL:
- https://aclanthology.org/2024.findings-emnlp.547
- DOI:
- 10.18653/v1/2024.findings-emnlp.547
- Cite (ACL):
- Zhiheng Lyu, Zhijing Jin, Fernando Gonzalez Adauto, Rada Mihalcea, Bernhard Schölkopf, and Mrinmaya Sachan. 2024. Do LLMs Think Fast and Slow? A Causal Study on Sentiment Analysis. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9353–9372, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- Do LLMs Think Fast and Slow? A Causal Study on Sentiment Analysis (Lyu et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/landing_page/2024.findings-emnlp.547.pdf