Rethinking Self-Attention: Towards Interpretability in Neural Parsing
Khalil Mrini, Franck Dernoncourt, Quan Hung Tran, Trung Bui, Walter Chang, Ndapa Nakashole
Abstract
Attention mechanisms have improved the performance of NLP tasks while allowing models to remain explainable. Self-attention is currently widely used, however interpretability is difficult due to the numerous attention distributions. Recent work has shown that model representations can benefit from label-specific information, while facilitating interpretation of predictions. We introduce the Label Attention Layer: a new form of self-attention where attention heads represent labels. We test our novel layer by running constituency and dependency parsing experiments and show our new model obtains new state-of-the-art results for both tasks on both the Penn Treebank (PTB) and Chinese Treebank. Additionally, our model requires fewer self-attention layers compared to existing work. Finally, we find that the Label Attention heads learn relations between syntactic categories and show pathways to analyze errors.- Anthology ID:
- 2020.findings-emnlp.65
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2020
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Trevor Cohn, Yulan He, Yang Liu
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 731–742
- Language:
- URL:
- https://aclanthology.org/2020.findings-emnlp.65
- DOI:
- 10.18653/v1/2020.findings-emnlp.65
- Cite (ACL):
- Khalil Mrini, Franck Dernoncourt, Quan Hung Tran, Trung Bui, Walter Chang, and Ndapa Nakashole. 2020. Rethinking Self-Attention: Towards Interpretability in Neural Parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 731–742, Online. Association for Computational Linguistics.
- Cite (Informal):
- Rethinking Self-Attention: Towards Interpretability in Neural Parsing (Mrini et al., Findings 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2020.findings-emnlp.65.pdf
- Code
- KhalilMrini/LAL-Parser + additional community code
- Data
- Penn Treebank