Self-Guided Curriculum Learning for Neural Machine Translation

Lei Zhou, Liang Ding, Kevin Duh, Shinji Watanabe, Ryohei Sasano, Koichi Takeda

Abstract
In supervised learning, a well-trained model should be able to recover ground truth accurately, i.e. the predicted labels are expected to resemble the ground truth labels as much as possible. Inspired by this, we formulate a difficulty criterion based on the recovery degrees of training examples. Motivated by the intuition that after skimming through the training corpus, the neural machine translation (NMT) model “knows” how to schedule a suitable curriculum according to learning difficulty, we propose a self-guided curriculum learning strategy that encourages the NMT model to learn from easy to hard on the basis of recovery degrees. Specifically, we adopt sentence-level BLEU score as the proxy of recovery degree. Experimental results on translation benchmarks including WMT14 English-German and WMT17 Chinese-English demonstrate that our proposed method considerably improves the recovery degree, thus consistently improving the translation performance.
Anthology ID:
2021.iwslt-1.25
Volume:
Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)
Month:
August
Year:
2021
Address:
Bangkok, Thailand (online)
Editors:
Marcello Federico, Alex Waibel, Marta R. Costa-jussà, Jan Niehues, Sebastian Stuker, Elizabeth Salesky
Venue:
IWSLT
SIG:
SIGSLT
Publisher:
Association for Computational Linguistics
Note:
Pages:
206–214
Language:
URL:
https://aclanthology.org/2021.iwslt-1.25
DOI:
10.18653/v1/2021.iwslt-1.25
Bibkey:
Cite (ACL):
Lei Zhou, Liang Ding, Kevin Duh, Shinji Watanabe, Ryohei Sasano, and Koichi Takeda. 2021. Self-Guided Curriculum Learning for Neural Machine Translation. In Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021), pages 206–214, Bangkok, Thailand (online). Association for Computational Linguistics.
Cite (Informal):
Self-Guided Curriculum Learning for Neural Machine Translation (Zhou et al., IWSLT 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2021.iwslt-1.25.pdf
PDF Search