Multi-layer Representation Fusion for Neural Machine Translation
Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, Jingbo Zhu
Abstract
Neural machine translation systems require a number of stacked layers for deep models. But the prediction depends on the sentence representation of the top-most layer with no access to low-level representations. This makes it more difficult to train the model and poses a risk of information loss to prediction. In this paper, we propose a multi-layer representation fusion (MLRF) approach to fusing stacked layers. In particular, we design three fusion functions to learn a better representation from the stack. Experimental results show that our approach yields improvements of 0.92 and 0.56 BLEU points over the strong Transformer baseline on IWSLT German-English and NIST Chinese-English MT tasks respectively. The result is new state-of-the-art in German-English translation.- Anthology ID:
- C18-1255
- Volume:
- Proceedings of the 27th International Conference on Computational Linguistics
- Month:
- August
- Year:
- 2018
- Address:
- Santa Fe, New Mexico, USA
- Venue:
- COLING
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3015–3026
- Language:
- URL:
- https://aclanthology.org/C18-1255
- DOI:
- Cite (ACL):
- Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingbo Zhu. 2018. Multi-layer Representation Fusion for Neural Machine Translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3015–3026, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
- Cite (Informal):
- Multi-layer Representation Fusion for Neural Machine Translation (Wang et al., COLING 2018)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/C18-1255.pdf