Normalizing Mutual Information for Robust Adaptive Training for Translation

Youngwon Lee, Changmin Lee, Hojin Lee, Seung-won Hwang

Abstract
Despite the success of neural machine translation models, tensions between fluency of optimizing target language modeling and source-faithfulness remain as challenges. Previously, Conditional Bilingual Mutual Information (CBMI), a scoring metric for the importance of target sentences and tokens, was proposed to encourage fluent and faithful translations. The score is obtained by combining the probability from the translation model and the target language model, which is then used to assign different weights to losses from sentences and tokens. Meanwhile, we argue this metric is not properly normalized, for which we propose Normalized Pointwise Mutual Information (NPMI). NPMI utilizes an additional language model on source language to approximate the joint likelihood of source-target pair and the likelihood of the source, which is then used for normalizing the score. We showed that NPMI better captures the dependence between source-target and that NPMI-based token-level adaptive training brings improvements over baselines with empirical results from En-De, De-En, and En-Ro translation tasks.
Anthology ID:
2022.emnlp-main.547
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8008–8015
Language:
URL:
https://aclanthology.org/2022.emnlp-main.547
DOI:
Bibkey:
Cite (ACL):
Youngwon Lee, Changmin Lee, Hojin Lee, and Seung-won Hwang. 2022. Normalizing Mutual Information for Robust Adaptive Training for Translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8008–8015, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Normalizing Mutual Information for Robust Adaptive Training for Translation (Lee et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.emnlp-main.547.pdf
PDF Search