Multi-Agent Mutual Learning at Sentence-Level and Token-Level for Neural Machine Translation

Baohao Liao, Yingbo Gao, Hermann Ney


Abstract
Mutual learning, where multiple agents learn collaboratively and teach one another, has been shown to be an effective way to distill knowledge for image classification tasks. In this paper, we extend mutual learning to the machine translation task and operate at both the sentence-level and the token-level. Firstly, we co-train multiple agents by using the same parallel corpora. After convergence, each agent selects and learns its poorly predicted tokens from other agents. The poorly predicted tokens are determined by the acceptance-rejection sampling algorithm. Our experiments show that sequential mutual learning at the sentence-level and the token-level improves the results cumulatively. Absolute improvements compared to strong baselines are obtained on various translation tasks. On the IWSLT’14 German-English task, we get a new state-of-the-art BLEU score of 37.0. We also report a competitive result, 29.9 BLEU score, on the WMT’14 English-German task.
Anthology ID:
2020.findings-emnlp.155
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1715–1724
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.155
DOI:
10.18653/v1/2020.findings-emnlp.155
Bibkey:
Cite (ACL):
Baohao Liao, Yingbo Gao, and Hermann Ney. 2020. Multi-Agent Mutual Learning at Sentence-Level and Token-Level for Neural Machine Translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1715–1724, Online. Association for Computational Linguistics.
Cite (Informal):
Multi-Agent Mutual Learning at Sentence-Level and Token-Level for Neural Machine Translation (Liao et al., Findings 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2020.findings-emnlp.155.pdf