Abstract
Recent years have witnessed a growing interest in investigating what Transformer-based language models (TLMs) actually learn from the training data. This is especially relevant for complex tasks such as the understanding of non-literal meaning. In this work, we probe the performance of three black-box TLMs and two intrinsically transparent white-box models on figurative language classification of sarcasm, similes, idioms, and metaphors. We conduct two studies on the classification results to provide insights into the inner workings of such models. With our first analysis on feature importance, we identify crucial differences in model behavior. With our second analysis using an online experiment with human participants, we inspect different linguistic characteristics of the four figurative language types.- Anthology ID:
- 2023.findings-acl.622
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2023
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 9816–9832
- Language:
- URL:
- https://aclanthology.org/2023.findings-acl.622
- DOI:
- 10.18653/v1/2023.findings-acl.622
- Cite (ACL):
- Hyewon Jang, Qi Yu, and Diego Frassinelli. 2023. Figurative Language Processing: A Linguistically Informed Feature Analysis of the Behavior of Language Models and Humans. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9816–9832, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Figurative Language Processing: A Linguistically Informed Feature Analysis of the Behavior of Language Models and Humans (Jang et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.findings-acl.622.pdf