Abstract
This paper proposes a novel attention mechanism for Transformer Neural Machine Translation, “Synchronous Syntactic Attention,” inspired by synchronous dependency grammars. The mechanism synchronizes source-side and target-side syntactic self-attentions by minimizing the difference between target-side self-attentions and the source-side self-attentions mapped by the encoder-decoder attention matrix. The experiments show that the proposed method improves the translation performance on WMT14 En-De, WMT16 En-Ro, and ASPEC Ja-En (up to +0.38 points in BLEU).- Anthology ID:
- 2021.acl-srw.36
- Volume:
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop
- Month:
- August
- Year:
- 2021
- Address:
- Online
- Venues:
- ACL | IJCNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 348–355
- Language:
- URL:
- https://aclanthology.org/2021.acl-srw.36
- DOI:
- 10.18653/v1/2021.acl-srw.36
- Cite (ACL):
- Hiroyuki Deguchi, Akihiro Tamura, and Takashi Ninomiya. 2021. Synchronous Syntactic Attention for Transformer Neural Machine Translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 348–355, Online. Association for Computational Linguistics.
- Cite (Informal):
- Synchronous Syntactic Attention for Transformer Neural Machine Translation (Deguchi et al., ACL-IJCNLP 2021)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2021.acl-srw.36.pdf