Improving Attention Modeling with Implicit Distortion and Fertility for Machine Translation

Shi Feng, Shujie Liu, Nan Yang, Mu Li, Ming Zhou, Kenny Q. Zhu


Abstract
In neural machine translation, the attention mechanism facilitates the translation process by producing a soft alignment between the source sentence and the target sentence. However, without dedicated distortion and fertility models seen in traditional SMT systems, the learned alignment may not be accurate, which can lead to low translation quality. In this paper, we propose two novel models to improve attention-based neural machine translation. We propose a recurrent attention mechanism as an implicit distortion model, and a fertility conditioned decoder as an implicit fertility model. We conduct experiments on large-scale Chinese–English translation tasks. The results show that our models significantly improve both the alignment and translation quality compared to the original attention mechanism and several other variations.
Anthology ID:
C16-1290
Volume:
Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
Month:
December
Year:
2016
Address:
Osaka, Japan
Venue:
COLING
SIG:
Publisher:
The COLING 2016 Organizing Committee
Note:
Pages:
3082–3092
Language:
URL:
https://aclanthology.org/C16-1290
DOI:
Bibkey:
Cite (ACL):
Shi Feng, Shujie Liu, Nan Yang, Mu Li, Ming Zhou, and Kenny Q. Zhu. 2016. Improving Attention Modeling with Implicit Distortion and Fertility for Machine Translation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3082–3092, Osaka, Japan. The COLING 2016 Organizing Committee.
Cite (Informal):
Improving Attention Modeling with Implicit Distortion and Fertility for Machine Translation (Feng et al., COLING 2016)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/C16-1290.pdf