Abstract
Adding linguistic information (syntax or semantics) to neural machine translation (NMT) have mostly focused on using point estimates from pre-trained models. Directly using the capacity of massive pre-trained contextual word embedding models such as BERT(Devlin et al., 2019) has been marginally useful in NMT because effective fine-tuning is difficult to obtain for NMT without making training brittle and unreliable. We augment NMT by extracting dense fine-tuned vector-based linguistic information from BERT instead of using point estimates. Experimental results show that our method of incorporating linguistic information helps NMT to generalize better in a variety of training contexts and is no more difficult to train than conventional Transformer-based NMT.- Anthology ID:
- 2021.eacl-main.241
- Volume:
- Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
- Month:
- April
- Year:
- 2021
- Address:
- Online
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2772–2783
- Language:
- URL:
- https://aclanthology.org/2021.eacl-main.241
- DOI:
- 10.18653/v1/2021.eacl-main.241
- Cite (ACL):
- Hassan S. Shavarani and Anoop Sarkar. 2021. Better Neural Machine Translation by Extracting Linguistic Information from BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2772–2783, Online. Association for Computational Linguistics.
- Cite (Informal):
- Better Neural Machine Translation by Extracting Linguistic Information from BERT (Shavarani & Sarkar, EACL 2021)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2021.eacl-main.241.pdf
- Code
- sfu-natlang/SFUTranslate