Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach
Daichi Haraguchi, Kiyoaki Shirai, Naoya Inoue, Natthawut Kertkeidkachorn
Abstract
Shortcut reasoning is an irrational process of inference, which degrades the robustness of an NLP model. While a number of previous work has tackled the identification of shortcut reasoning, there are still two major limitations: (i) a method for quantifying the severity of the discovered shortcut reasoning is not provided; (ii) certain types of shortcut reasoning may be missed. To address these issues, we propose a novel method for identifying shortcut reasoning. The proposed method quantifies the severity of the shortcut reasoning by leveraging out-of-distribution data and does not make any assumptions about the type of tokens triggering the shortcut reasoning. Our experiments on Natural Language Inference and Sentiment Analysis demonstrate that our framework successfully discovers known and unknown shortcut reasoning in the previous work.- Anthology ID:
- 2023.findings-emnlp.424
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6401–6407
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.424
- DOI:
- 10.18653/v1/2023.findings-emnlp.424
- Cite (ACL):
- Daichi Haraguchi, Kiyoaki Shirai, Naoya Inoue, and Natthawut Kertkeidkachorn. 2023. Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6401–6407, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach (Haraguchi et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2023.findings-emnlp.424.pdf