Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference
Bang An, Jie Lyu, Zhenyi Wang, Chunyuan Li, Changwei Hu, Fei Tan, Ruiyi Zhang, Yifan Hu, Changyou Chen
Abstract
The neural attention mechanism plays an important role in many natural language processing applications. In particular, multi-head attention extends single-head attention by allowing a model to jointly attend information from different perspectives. However, without explicit constraining, multi-head attention may suffer from attention collapse, an issue that makes different heads extract similar attentive features, thus limiting the model’s representation power. In this paper, for the first time, we provide a novel understanding of multi-head attention from a Bayesian perspective. Based on the recently developed particle-optimization sampling techniques, we propose a non-parametric approach that explicitly improves the repulsiveness in multi-head attention and consequently strengthens model’s expressiveness. Remarkably, our Bayesian interpretation provides theoretical inspirations on the not-well-understood questions: why and how one uses multi-head attention. Extensive experiments on various attention models and applications demonstrate that the proposed repulsive attention can improve the learned feature diversity, leading to more informative representations with consistent performance improvement on multiple tasks.- Anthology ID:
- 2020.emnlp-main.17
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 236–255
- Language:
- URL:
- https://aclanthology.org/2020.emnlp-main.17
- DOI:
- 10.18653/v1/2020.emnlp-main.17
- Cite (ACL):
- Bang An, Jie Lyu, Zhenyi Wang, Chunyuan Li, Changwei Hu, Fei Tan, Ruiyi Zhang, Yifan Hu, and Changyou Chen. 2020. Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 236–255, Online. Association for Computational Linguistics.
- Cite (Informal):
- Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference (An et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2020.emnlp-main.17.pdf
- Data
- AGENDA, CoLA, GLUE, MultiNLI