Abstract
Modern neural machine translation (NMT) models have achieved competitive performance in standard benchmarks such as WMT. However, there still exist significant issues such as robustness, domain generalization, etc. In this paper, we study NMT models from the perspective of compositional generalization by building a benchmark dataset, CoGnition, consisting of 216k clean and consistent sentence pairs. We quantitatively analyze effects of various factors using compound translation error rate, then demonstrate that the NMT model fails badly on compositional generalization, although it performs remarkably well under traditional metrics.- Anthology ID:
- 2021.acl-long.368
- Volume:
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Month:
- August
- Year:
- 2021
- Address:
- Online
- Editors:
- Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
- Venues:
- ACL | IJCNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4767–4780
- Language:
- URL:
- https://aclanthology.org/2021.acl-long.368
- DOI:
- 10.18653/v1/2021.acl-long.368
- Cite (ACL):
- Yafu Li, Yongjing Yin, Yulong Chen, and Yue Zhang. 2021. On Compositional Generalization of Neural Machine Translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4767–4780, Online. Association for Computational Linguistics.
- Cite (Informal):
- On Compositional Generalization of Neural Machine Translation (Li et al., ACL-IJCNLP 2021)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2021.acl-long.368.pdf
- Code
- yafuly/CoGnition
- Data
- CFQ, ROCStories, SCAN