Better Neural Machine Translation by Extracting Linguistic Information from BERT

Hassan S. Shavarani, Anoop Sarkar


Abstract
Adding linguistic information (syntax or semantics) to neural machine translation (NMT) have mostly focused on using point estimates from pre-trained models. Directly using the capacity of massive pre-trained contextual word embedding models such as BERT(Devlin et al., 2019) has been marginally useful in NMT because effective fine-tuning is difficult to obtain for NMT without making training brittle and unreliable. We augment NMT by extracting dense fine-tuned vector-based linguistic information from BERT instead of using point estimates. Experimental results show that our method of incorporating linguistic information helps NMT to generalize better in a variety of training contexts and is no more difficult to train than conventional Transformer-based NMT.
Anthology ID:
2021.eacl-main.241
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2772–2783
Language:
URL:
https://aclanthology.org/2021.eacl-main.241
DOI:
10.18653/v1/2021.eacl-main.241
Bibkey:
Cite (ACL):
Hassan S. Shavarani and Anoop Sarkar. 2021. Better Neural Machine Translation by Extracting Linguistic Information from BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2772–2783, Online. Association for Computational Linguistics.
Cite (Informal):
Better Neural Machine Translation by Extracting Linguistic Information from BERT (Shavarani & Sarkar, EACL 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/2021.eacl-main.241.pdf
Code
 sfu-natlang/SFUTranslate