Being Generous with Sub-Words towards Small NMT Children
Arne Defauw, Tom Vanallemeersch, Koen Van Winckel, Sara Szoc, Joachim Van den Bogaert
Abstract
In the context of under-resourced neural machine translation (NMT), transfer learning from an NMT model trained on a high resource language pair, or from a multilingual NMT (M-NMT) model, has been shown to boost performance to a large extent. In this paper, we focus on so-called cold start transfer learning from an M-NMT model, which means that the parent model is not trained on any of the child data. Such a set-up enables quick adaptation of M-NMT models to new languages. We investigate the effectiveness of cold start transfer learning from a many-to-many M-NMT model to an under-resourced child. We show that sufficiently large sub-word vocabularies should be used for transfer learning to be effective in such a scenario. When adopting relatively large sub-word vocabularies we observe increases in performance thanks to transfer learning from a parent M-NMT model, both when translating to and from the under-resourced language. Our proposed approach involving dynamic vocabularies is both practical and effective. We report results on two under-resourced language pairs, i.e. Icelandic-English and Irish-English.- Anthology ID:
- 2020.lrec-1.450
- Volume:
- Proceedings of the Twelfth Language Resources and Evaluation Conference
- Month:
- May
- Year:
- 2020
- Address:
- Marseille, France
- Editors:
- Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 3650–3656
- Language:
- English
- URL:
- https://aclanthology.org/2020.lrec-1.450
- DOI:
- Cite (ACL):
- Arne Defauw, Tom Vanallemeersch, Koen Van Winckel, Sara Szoc, and Joachim Van den Bogaert. 2020. Being Generous with Sub-Words towards Small NMT Children. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3650–3656, Marseille, France. European Language Resources Association.
- Cite (Informal):
- Being Generous with Sub-Words towards Small NMT Children (Defauw et al., LREC 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2020.lrec-1.450.pdf