Non-Autoregressive Machine Translation with Latent Alignments

Chitwan Saharia, William Chan, Saurabh Saxena, Mohammad Norouzi

Abstract
This paper presents two strong methods, CTC and Imputer, for non-autoregressive machine translation that model latent alignments with dynamic programming. We revisit CTC for machine translation and demonstrate that a simple CTC model can achieve state-of-the-art for single-step non-autoregressive machine translation, contrary to what prior work indicates. In addition, we adapt the Imputer model for non-autoregressive machine translation and demonstrate that Imputer with just 4 generation steps can match the performance of an autoregressive Transformer baseline. Our latent alignment models are simpler than many existing non-autoregressive translation baselines; for example, we do not require target length prediction or re-scoring with an autoregressive model. On the competitive WMT’14 En→De task, our CTC model achieves 25.7 BLEU with a single generation step, while Imputer achieves 27.5 BLEU with 2 generation steps, and 28.0 BLEU with 4 generation steps. This compares favourably to the autoregressive Transformer baseline at 27.8 BLEU.
Anthology ID:
2020.emnlp-main.83
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1098–1108
Language:
URL:
https://aclanthology.org/2020.emnlp-main.83
DOI:
10.18653/v1/2020.emnlp-main.83
Bibkey:
Cite (ACL):
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. Non-Autoregressive Machine Translation with Latent Alignments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1098–1108, Online. Association for Computational Linguistics.
Cite (Informal):
Non-Autoregressive Machine Translation with Latent Alignments (Saharia et al., EMNLP 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/add_acl24_videos/2020.emnlp-main.83.pdf
Video:
 https://slideslive.com/38938713
Code
 additional community code
PDF Search Code Video