Abstract
This work presents our ongoing research of unsupervised pretraining in neural machine translation (NMT). In our method, we initialize the weights of the encoder and decoder with two language models that are trained with monolingual data and then fine-tune the model on parallel data using Elastic Weight Consolidation (EWC) to avoid forgetting of the original language modeling task. We compare the regularization by EWC with the previous work that focuses on regularization by language modeling objectives. The positive result is that using EWC with the decoder achieves BLEU scores similar to the previous work. However, the model converges 2-3 times faster and does not require the original unlabeled training data during the fine-tuning stage. In contrast, the regularization using EWC is less effective if the original and new tasks are not closely related. We show that initializing the bidirectional NMT encoder with a left-to-right language model and forcing the model to remember the original left-to-right language modeling task limits the learning capacity of the encoder for the whole bidirectional context.- Anthology ID:
- P19-2017
- Volume:
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop
- Month:
- July
- Year:
- 2019
- Address:
- Florence, Italy
- Editors:
- Fernando Alva-Manchego, Eunsol Choi, Daniel Khashabi
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 130–135
- Language:
- URL:
- https://aclanthology.org/P19-2017
- DOI:
- 10.18653/v1/P19-2017
- Cite (ACL):
- Dušan Variš and Ondřej Bojar. 2019. Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 130–135, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation (Variš & Bojar, ACL 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/P19-2017.pdf