Abstract
Using a language model (LM) pretrained on two languages with large monolingual data in order to initialize an unsupervised neural machine translation (UNMT) system yields state-of-the-art results. When limited data is available for one language, however, this method leads to poor translations. We present an effective approach that reuses an LM that is pretrained only on the high-resource language. The monolingual LM is fine-tuned on both languages and is then used to initialize a UNMT model. To reuse the pretrained LM, we have to modify its predefined vocabulary, to account for the new language. We therefore propose a novel vocabulary extension method. Our approach, RE-LM, outperforms a competitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk) and English-Albanian (En-Sq), yielding more than +8.3 BLEU points for all four translation directions.- Anthology ID:
- 2020.emnlp-main.214
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2703–2711
- Language:
- URL:
- https://aclanthology.org/2020.emnlp-main.214
- DOI:
- 10.18653/v1/2020.emnlp-main.214
- Cite (ACL):
- Alexandra Chronopoulou, Dario Stojanovski, and Alexander Fraser. 2020. Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2703–2711, Online. Association for Computational Linguistics.
- Cite (Informal):
- Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT (Chronopoulou et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2020.emnlp-main.214.pdf
- Code
- alexandra-chron/relm_unmt