Abstract
In interpretable NLP, we require faithful rationales that reflect the model’s decision-making process for an explained instance. While prior work focuses on extractive rationales (a subset of the input words), we investigate their less-studied counterpart: free-text natural language rationales. We demonstrate that *pipelines*, models for faithful rationalization on information-extraction style tasks, do not work as well on “reasoning” tasks requiring free-text rationales. We turn to models that *jointly* predict and rationalize, a class of widely used high-performance models for free-text rationalization. We investigate the extent to which the labels and rationales predicted by these models are associated, a necessary property of faithful explanation. Via two tests, *robustness equivalence* and *feature importance agreement*, we find that state-of-the-art T5-based joint models exhibit desirable properties for explaining commonsense question-answering and natural language inference, indicating their potential for producing faithful free-text rationales.- Anthology ID:
- 2021.emnlp-main.804
- Original:
- 2021.emnlp-main.804v1
- Version 2:
- 2021.emnlp-main.804v2
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 10266–10284
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.804
- DOI:
- 10.18653/v1/2021.emnlp-main.804
- Cite (ACL):
- Sarah Wiegreffe, Ana Marasović, and Noah A. Smith. 2021. Measuring Association Between Labels and Free-Text Rationales. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10266–10284, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Measuring Association Between Labels and Free-Text Rationales (Wiegreffe et al., EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/remove-xml-comments/2021.emnlp-main.804.pdf
- Code
- allenai/label_rationale_association
- Data
- CoS-E, CommonsenseQA, SNLI, e-SNLI