Abstract
Multilingual Neural Machine Translation (MNMT) facilitates knowledge sharing but often suffers from poor zero-shot (ZS) translation qualities. While prior work has explored the causes of overall low zero-shot translation qualities, our work introduces a fresh perspective: the presence of significant variations in zero-shot performance. This suggests that MNMT does not uniformly exhibit poor zero-shot capability; instead, certain translation directions yield reasonable results. Through systematic experimentation, spanning 1,560 language directions across 40 languages, we identify three key factors contributing to high variations in ZS NMT performance: 1) target-side translation quality, 2) vocabulary overlap, and 3) linguistic properties. Our findings highlight that the target side translation quality is the most influential factor, with vocabulary overlap consistently impacting zero-shot capabilities. Additionally, linguistic properties, such as language family and writing system, play a role, particularly with smaller models. Furthermore, we suggest that the off-target issue is a symptom of inadequate performance, emphasizing that zero-shot translation challenges extend beyond addressing the off-target problem. To support future research, we release the data and models as a benchmark for the study of ZS NMT.- Anthology ID:
- 2023.emnlp-main.836
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 13553–13568
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2023.emnlp-main.836/
- DOI:
- 10.18653/v1/2023.emnlp-main.836
- Cite (ACL):
- Shaomu Tan and Christof Monz. 2023. Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13553–13568, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance (Tan & Monz, EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2023.emnlp-main.836.pdf