Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism
Pan Xie, Zhi Cui, Xiuying Chen, XiaoHui Hu, Jianwei Cui, Bin Wang
Abstract
Non-autoregressive models generate target words in a parallel way, which achieve a faster decoding speed but at the sacrifice of translation accuracy. To remedy a flawed translation by non-autoregressive models, a promising approach is to train a conditional masked translation model (CMTM), and refine the generated results within several iterations. Unfortunately, such approach hardly considers the sequential dependency among target words, which inevitably results in a translation degradation. Hence, instead of solely training a Transformer-based CMTM, we propose a Self-Review Mechanism to infuse sequential information into it. Concretely, we insert a left-to-right mask to the same decoder of CMTM, and then induce it to autoregressively review whether each generated word from CMTM is supposed to be replaced or kept. The experimental results (WMT14 En ↔ De and WMT16 En ↔ Ro) demonstrate that our model uses dramatically less training computations than the typical CMTM, as well as outperforms several state-of-the-art non-autoregressive models by over 1 BLEU. Through knowledge distillation, our model even surpasses a typical left-to-right Transformer model, while significantly speeding up decoding.- Anthology ID:
- 2020.coling-main.2
- Volume:
- Proceedings of the 28th International Conference on Computational Linguistics
- Month:
- December
- Year:
- 2020
- Address:
- Barcelona, Spain (Online)
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 15–25
- Language:
- URL:
- https://aclanthology.org/2020.coling-main.2
- DOI:
- 10.18653/v1/2020.coling-main.2
- Cite (ACL):
- Pan Xie, Zhi Cui, Xiuying Chen, XiaoHui Hu, Jianwei Cui, and Bin Wang. 2020. Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism. In Proceedings of the 28th International Conference on Computational Linguistics, pages 15–25, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- Cite (Informal):
- Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism (Xie et al., COLING 2020)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2020.coling-main.2.pdf
- Code
- PanXiebit/Self-Review-NAT