Abstract
Most evaluations of attribution methods focus on the English language. In this work, we present a multilingual approach for evaluating attribution methods for the Natural Language Inference (NLI) task in terms of faithfulness and plausibility. First, we introduce a novel cross-lingual strategy to measure faithfulness based on word alignments, which eliminates the drawbacks of erasure-based evaluations. We then perform a comprehensive evaluation of attribution methods, considering different output mechanisms and aggregation methods. Finally, we augment the XNLI dataset with highlight-based explanations, providing a multilingual NLI dataset with highlights, to support future exNLP studies. Our results show that attribution methods performing best for plausibility and faithfulness are different.- Anthology ID:
- 2022.emnlp-main.101
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1556–1576
- Language:
- URL:
- https://aclanthology.org/2022.emnlp-main.101
- DOI:
- 10.18653/v1/2022.emnlp-main.101
- Cite (ACL):
- Kerem Zaman and Yonatan Belinkov. 2022. A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1556–1576, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference (Zaman & Belinkov, EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.emnlp-main.101.pdf