Multi-Head Attention with Disagreement Regularization

Jian Li, Zhaopeng Tu, Baosong Yang, Michael R. Lyu, Tong Zhang

Abstract
Multi-head attention is appealing for the ability to jointly attend to information from different representation subspaces at different positions. In this work, we introduce a disagreement regularization to explicitly encourage the diversity among multiple attention heads. Specifically, we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed approach.
Anthology ID:
D18-1317
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2897–2903
Language:
URL:
https://aclanthology.org/D18-1317
DOI:
10.18653/v1/D18-1317
Bibkey:
Cite (ACL):
Jian Li, Zhaopeng Tu, Baosong Yang, Michael R. Lyu, and Tong Zhang. 2018. Multi-Head Attention with Disagreement Regularization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2897–2903, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Multi-Head Attention with Disagreement Regularization (Li et al., EMNLP 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/nodalida-main-page/D18-1317.pdf
PDF Search