Exploiting Attention to Reveal Shortcomings in Memory Models
Kaylee Burns, Aida Nematzadeh, Erin Grant, Alison Gopnik, Tom Griffiths
Abstract
The decision making processes of deep networks are difficult to understand and while their accuracy often improves with increased architectural complexity, so too does their opacity. Practical use of machine learning models, especially for question and answering applications, demands a system that is interpretable. We analyze the attention of a memory network model to reconcile contradictory performance on a challenging question-answering dataset that is inspired by theory-of-mind experiments. We equate success on questions to task classification, which explains not only test-time failures but also how well the model generalizes to new training conditions.- Anthology ID:
- W18-5454
- Volume:
- Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
- Month:
- November
- Year:
- 2018
- Address:
- Brussels, Belgium
- Editors:
- Tal Linzen, Grzegorz Chrupała, Afra Alishahi
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 378–380
- Language:
- URL:
- https://aclanthology.org/W18-5454
- DOI:
- 10.18653/v1/W18-5454
- Cite (ACL):
- Kaylee Burns, Aida Nematzadeh, Erin Grant, Alison Gopnik, and Tom Griffiths. 2018. Exploiting Attention to Reveal Shortcomings in Memory Models. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 378–380, Brussels, Belgium. Association for Computational Linguistics.
- Cite (Informal):
- Exploiting Attention to Reveal Shortcomings in Memory Models (Burns et al., EMNLP 2018)
- PDF:
- https://preview.aclanthology.org/ml4al-ingestion/W18-5454.pdf