2020
pdf
abs
Efficient and High-Quality Neural Machine Translation with OpenNMT
Guillaume Klein
|
Dakun Zhang
|
Clément Chouteau
|
Josep Crego
|
Jean Senellart
Proceedings of the Fourth Workshop on Neural Generation and Translation
This paper describes the OpenNMT submissions to the WNGT 2020 efficiency shared task. We explore training and acceleration of Transformer models with various sizes that are trained in a teacher-student setup. We also present a custom and optimized C++ inference engine that enables fast CPU and GPU decoding with few dependencies. By combining additional optimizations and parallelization techniques, we create small, efficient, and high-quality neural machine translation models.
2018
pdf
abs
Analyzing Knowledge Distillation in Neural Machine Translation
Dakun Zhang
|
Josep Crego
|
Jean Senellart
Proceedings of the 15th International Conference on Spoken Language Translation
Knowledge distillation has recently been successfully applied to neural machine translation. It allows for building shrunk networks while the resulting systems retain most of the quality of the original model. Despite the fact that many authors report on the benefits of knowledge distillation, few have discussed the actual reasons why it works, especially in the context of neural MT. In this paper, we conduct several experiments aimed at understanding why and how distillation impacts accuracy on an English-German translation task. We show that translation complexity is actually reduced when building a distilled/synthesised bi-text when compared to the reference bi-text. We further remove noisy data from synthesised translations and merge filtered synthesised data together with original reference, thus achieving additional gains in terms of accuracy.
pdf
abs
OpenNMT System Description for WNMT 2018: 800 words/sec on a single-core CPU
Jean Senellart
|
Dakun Zhang
|
Bo Wang
|
Guillaume Klein
|
Jean-Pierre Ramatchandirin
|
Josep Crego
|
Alexander Rush
Proceedings of the 2nd Workshop on Neural Machine Translation and Generation
We present a system description of the OpenNMT Neural Machine Translation entry for the WNMT 2018 evaluation. In this work, we developed a heavily optimized NMT inference model targeting a high-performance CPU system. The final system uses a combination of four techniques, all of them lead to significant speed-ups in combination: (a) sequence distillation, (b) architecture modifications, (c) precomputation, particularly of vocabulary, and (d) CPU targeted quantization. This work achieves the fastest performance of the shared task, and led to the development of new features that have been integrated to OpenNMT and available to the community.
2017
pdf
abs
Boosting Neural Machine Translation
Dakun Zhang
|
Jungi Kim
|
Josep Crego
|
Jean Senellart
Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
Training efficiency is one of the main problems for Neural Machine Translation (NMT). Deep networks need for very large data as well as many training iterations to achieve state-of-the-art performance. This results in very high computation cost, slowing down research and industrialisation. In this paper, we propose to alleviate this problem with several training methods based on data boosting and bootstrap with no modifications to the neural network. It imitates the learning process of humans, which typically spend more time when learning “difficult” concepts than easier ones. We experiment on an English-French translation task showing accuracy improvements of up to 1.63 BLEU while saving 20% of training time.
pdf
SYSTRAN Purely Neural MT Engines for WMT2017
Yongchao Deng
|
Jungi Kim
|
Guillaume Klein
|
Catherine Kobus
|
Natalia Segal
|
Christophe Servan
|
Bo Wang
|
Dakun Zhang
|
Josep Crego
|
Jean Senellart
Proceedings of the Second Conference on Machine Translation
2015
pdf
Well-Formed Dependency to String translation with BTG Grammar
Xiaoqing Li
|
Kun Wang
|
Dakun Zhang
|
Jie Hao
Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation
2014
pdf
A Neural Reordering Model for Phrase-based Translation
Peng Li
|
Yang Liu
|
Maosong Sun
|
Tatsuya Izuha
|
Dakun Zhang
Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers
2009
pdf
A Syllable-based Name Transliteration System
Xue Jiang
|
Le Sun
|
Dakun Zhang
Proceedings of the 2009 Named Entities Workshop: Shared Task on Transliteration (NEWS 2009)
2008
pdf
A Structured Prediction Approach for Statistical Machine Translation
Dakun Zhang
|
Le Sun
|
Wenbo Li
Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II