A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models
Francesca De Luca Fornaciari, Begoña Altuna, Itziar Gonzalez-Dios, Maite Melero
Abstract
In this work, we explore idiomatic language processing with Large Language Models (LLMs). We introduce the Idiomatic language Test Suite IdioTS, a dataset of difficult examples specifically designed by language experts to assess the capabilities of LLMs to process figurative language at sentence level. We propose a comprehensive evaluation methodology based on an idiom detection task, where LLMs are prompted with detecting an idiomatic expression in a given English sentence. We present a thorough automatic and manual evaluation of the results and a comprehensive error analysis.- Anthology ID:
- 2024.figlang-1.5
- Volume:
- Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico (Hybrid)
- Editors:
- Debanjan Ghosh, Smaranda Muresan, Anna Feldman, Tuhin Chakrabarty, Emmy Liu
- Venues:
- Fig-Lang | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 35–44
- Language:
- URL:
- https://aclanthology.org/2024.figlang-1.5
- DOI:
- Cite (ACL):
- Francesca De Luca Fornaciari, Begoña Altuna, Itziar Gonzalez-Dios, and Maite Melero. 2024. A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models. In Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024), pages 35–44, Mexico City, Mexico (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models (De Luca Fornaciari et al., Fig-Lang-WS 2024)
- PDF:
- https://preview.aclanthology.org/jeptaln-2024-ingestion/2024.figlang-1.5.pdf