Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification
Kushal Tatariya, Heather Lent, Johannes Bjerva, Miryam de Lhoneux
Abstract
Emotion classification is a challenging task in NLP due to the inherent idiosyncratic and subjective nature of linguistic expression,especially with code-mixed data. Pre-trained language models (PLMs) have achieved high performance for many tasks and languages, but it remains to be seen whether these models learn and are robust to the differences in emotional expression across languages. Sociolinguistic studies have shown that Hinglish speakers switch to Hindi when expressing negative emotions and to English when expressing positive emotions. To understand if language models can learn these associations, we study the effect of language on emotion prediction across 3 PLMs on a Hinglish emotion classification dataset. Using LIME and token level language ID, we find that models do learn these associations between language choice and emotional expression. Moreover, having code-mixed data present in the pre-training can augment that learning when task-specific data is scarce. We also conclude from the misclassifications that the models may overgeneralise this heuristic to other infrequent examples where this sociolinguistic phenomenon does not apply.- Anthology ID:
- 2024.sigtyp-1.9
- Volume:
- Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP
- Month:
- March
- Year:
- 2024
- Address:
- St. Julian's, Malta
- Editors:
- Michael Hahn, Alexey Sorokin, Ritesh Kumar, Andreas Shcherbakov, Yulia Otmakhova, Jinrui Yang, Oleg Serikov, Priya Rani, Edoardo M. Ponti, Saliha Muradoğlu, Rena Gao, Ryan Cotterell, Ekaterina Vylomova
- Venues:
- SIGTYP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 66–74
- Language:
- URL:
- https://aclanthology.org/2024.sigtyp-1.9
- DOI:
- Cite (ACL):
- Kushal Tatariya, Heather Lent, Johannes Bjerva, and Miryam de Lhoneux. 2024. Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification. In Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 66–74, St. Julian's, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification (Tatariya et al., SIGTYP-WS 2024)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2024.sigtyp-1.9.pdf