Bar Cohen
2026
ID10M-JAM: Stress-Testing Idiom Identification Under Challenging Context
Kai Golan Hashiloni | Lior Livyatan | Ofri Hefetz | Alon Mannor | Bar Cohen | Kfir Bar
Findings of the Association for Computational Linguistics: ACL 2026
Kai Golan Hashiloni | Lior Livyatan | Ofri Hefetz | Alon Mannor | Bar Cohen | Kfir Bar
Findings of the Association for Computational Linguistics: ACL 2026
Large language models (LLMs) achieve strong performance on idiom identification benchmarks, yet their robustness to misleading contextual signals remains largely untested. We introduce ID10M-JAM, an adversarial extension of the ID10M dataset designed to jam model understanding by injecting coherent but conflicting context before each target sentence. For every sentence containing a potential idiomatic expression (PIE), we construct variants that deliberately invert contextual expectations: placing literal cues before idiomatic uses and idiomatic cues before literal ones. All variants are validated by human annotators to ensure naturalness and unambiguous interpretation for human readers. ID10M-JAM exposes systematic vulnerabilities in LLMs’ contextual reasoning, pushing idiom identification to its breaking point.