Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT

Alexandra Chronopoulou, Dario Stojanovski, Alexander Fraser


Abstract
Using a language model (LM) pretrained on two languages with large monolingual data in order to initialize an unsupervised neural machine translation (UNMT) system yields state-of-the-art results. When limited data is available for one language, however, this method leads to poor translations. We present an effective approach that reuses an LM that is pretrained only on the high-resource language. The monolingual LM is fine-tuned on both languages and is then used to initialize a UNMT model. To reuse the pretrained LM, we have to modify its predefined vocabulary, to account for the new language. We therefore propose a novel vocabulary extension method. Our approach, RE-LM, outperforms a competitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk) and English-Albanian (En-Sq), yielding more than +8.3 BLEU points for all four translation directions.
Anthology ID:
2020.emnlp-main.214
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2703–2711
Language:
URL:
https://aclanthology.org/2020.emnlp-main.214
DOI:
10.18653/v1/2020.emnlp-main.214
Bibkey:
Cite (ACL):
Alexandra Chronopoulou, Dario Stojanovski, and Alexander Fraser. 2020. Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2703–2711, Online. Association for Computational Linguistics.
Cite (Informal):
Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT (Chronopoulou et al., EMNLP 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/2020.emnlp-main.214.pdf
Optional supplementary material:
 2020.emnlp-main.214.OptionalSupplementaryMaterial.zip
Video:
 https://slideslive.com/38938785
Code
 alexandra-chron/relm_unmt