Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English

Gongbo Tang, Rico Sennrich, Joakim Nivre

Abstract
Recent work has shown that deeper character-based neural machine translation (NMT) models can outperform subword-based models. However, it is still unclear what makes deeper character-based models successful. In this paper, we conduct an investigation into pure character-based models in the case of translating Finnish into English, including exploring the ability to learn word senses and morphological inflections and the attention mechanism. We demonstrate that word-level information is distributed over the entire character sequence rather than over a single character, and characters at different positions play different roles in learning linguistic knowledge. In addition, character-based models need more layers to encode word senses which explains why only deeper models outperform subword-based models. The attention distribution pattern shows that separators attract a lot of attention and we explore a sparse word-level attention to enforce character hidden states to capture the full word-level information. Experimental results show that the word-level attention with a single head results in 1.2 BLEU points drop.
Anthology ID:
2020.coling-main.375
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
4251–4262
Language:
URL:
https://aclanthology.org/2020.coling-main.375
DOI:
10.18653/v1/2020.coling-main.375
Bibkey:
Cite (ACL):
Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2020. Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4251–4262, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English (Tang et al., COLING 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2020.coling-main.375.pdf
PDF Search