Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation

Michalis Korakakis, Andreas Vlachos

Abstract
Despite strong performance in many sequence-to-sequence tasks, autoregressive models trained with maximum likelihood estimation suffer from exposure bias, i.e. the discrepancy between the ground-truth prefixes used during training and the model-generated prefixes used at inference time. Scheduled sampling is a simple and empirically successful approach which addresses this issue by incorporating model-generated prefixes into training. However, it has been argued that it is an inconsistent training objective leading to models ignoring the prefixes altogether. In this paper, we conduct systematic experiments and find that scheduled sampling, while it ameliorates exposure bias by increasing model reliance on the input sequence, worsens performance when the prefix at inference time is correct, a form of catastrophic forgetting. We propose to use Elastic Weight Consolidation to better balance mitigating exposure bias with retaining performance. Experiments on four IWSLT’14 and WMT’14 translation datasets demonstrate that our approach alleviates catastrophic forgetting and significantly outperforms maximum likelihood estimation and scheduled sampling baselines.
Anthology ID:
2022.findings-emnlp.536
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2022
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7247–7258
Language:
URL:
https://aclanthology.org/2022.findings-emnlp.536
DOI:
10.18653/v1/2022.findings-emnlp.536
Bibkey:
Cite (ACL):
Michalis Korakakis and Andreas Vlachos. 2022. Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7247–7258, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation (Korakakis & Vlachos, Findings 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/dois-2013-emnlp/2022.findings-emnlp.536.pdf
PDF Search