A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference

Kerem Zaman, Yonatan Belinkov


Abstract
Most evaluations of attribution methods focus on the English language. In this work, we present a multilingual approach for evaluating attribution methods for the Natural Language Inference (NLI) task in terms of faithfulness and plausibility.First, we introduce a novel cross-lingual strategy to measure faithfulness based on word alignments, which eliminates the drawbacks of erasure-based evaluations.We then perform a comprehensive evaluation of attribution methods, considering different output mechanisms and aggregation methods.Finally, we augment the XNLI dataset with highlight-based explanations, providing a multilingual NLI dataset with highlights, to support future exNLP studies. Our results show that attribution methods performing best for plausibility and faithfulness are different.
Anthology ID:
2022.emnlp-main.101
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1556–1576
Language:
URL:
https://aclanthology.org/2022.emnlp-main.101
DOI:
Bibkey:
Cite (ACL):
Kerem Zaman and Yonatan Belinkov. 2022. A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1556–1576, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference (Zaman & Belinkov, EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-ingestion/2022.emnlp-main.101.pdf