Abstract
In this work, we present an empirical study of generation order for machine translation. Building on recent advances in insertion-based modeling, we first introduce a soft order-reward framework that enables us to train models to follow arbitrary oracle generation policies. We then make use of this framework to explore a large variety of generation orders, including uninformed orders, location-based orders, frequency-based orders, content-based orders, and model-based orders. Curiously, we find that for the WMT’14 English → German and WMT’18 English → Chinese translation tasks, order does not have a substantial impact on output quality. Moreover, for English → German, we even discover that unintuitive orderings such as alphabetical and shortest-first can match the performance of a standard Transformer, suggesting that traditional left-to-right generation may not be necessary to achieve high performance.- Anthology ID:
- 2020.emnlp-main.464
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5764–5773
- Language:
- URL:
- https://aclanthology.org/2020.emnlp-main.464
- DOI:
- 10.18653/v1/2020.emnlp-main.464
- Cite (ACL):
- William Chan, Mitchell Stern, Jamie Kiros, and Jakob Uszkoreit. 2020. An Empirical Study of Generation Order for Machine Translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5764–5773, Online. Association for Computational Linguistics.
- Cite (Informal):
- An Empirical Study of Generation Order for Machine Translation (Chan et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/remove-xml-comments/2020.emnlp-main.464.pdf