A Fine Line Between Irony and Sincerity: Identifying Bias in Transformer Models for Irony Detection
Aaron Maladry, Els Lefever, Cynthia Van Hee, Veronique Hoste
Abstract
In this paper we investigate potential bias in fine-tuned transformer models for irony detection. Bias is defined in this research as spurious associations between word n-grams and class labels, that can cause the system to rely too much on superficial cues and miss the essence of the irony. For this purpose, we looked for correlations between class labels and words that are prone to trigger irony, such as positive adjectives, intensifiers and topical nouns. Additionally, we investigate our irony model’s predictions before and after manipulating the data set through irony trigger replacements. We further support these insights with state-of-the-art explainability techniques (Layer Integrated Gradients, Discretized Integrated Gradients and Layer-wise Relevance Propagation). Both approaches confirm the hypothesis that transformer models generally encode correlations between positive sentiments and ironic texts, with even higher correlations between vividly expressed sentiment and irony. Based on these insights, we implemented a number of modification strategies to enhance the robustness of our irony classifier.- Anthology ID:
- 2023.wassa-1.28
- Volume:
- Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Jeremy Barnes, Orphée De Clercq, Roman Klinger
- Venue:
- WASSA
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 315–324
- Language:
- URL:
- https://aclanthology.org/2023.wassa-1.28
- DOI:
- 10.18653/v1/2023.wassa-1.28
- Cite (ACL):
- Aaron Maladry, Els Lefever, Cynthia Van Hee, and Veronique Hoste. 2023. A Fine Line Between Irony and Sincerity: Identifying Bias in Transformer Models for Irony Detection. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 315–324, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- A Fine Line Between Irony and Sincerity: Identifying Bias in Transformer Models for Irony Detection (Maladry et al., WASSA 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.wassa-1.28.pdf