Contrastive Preference Learning for Neural Machine Translation
Jianfei He, Shichao Sun, Sen Peng, Jie Xu, Xiaohua Jia, Wenjie Li
Abstract
There exists a discrepancy between the token-level objective during training and the overall sequence-level quality that is expected from the model. This discrepancy leads to issues like exposure bias.To align the model with human expectations, sequence-level objectives are often used to fine-tune pre-trained models.In this paper, we introduce a contrastive preference model that enhances the traditional Plackett-Luce model by incorporating an indicator function. Building upon this novel preference model, we propose Contrastive Preference Learning (CPL), which uses offline samples with list-wise preferences to fine-tune a pre-trained model in Neural Machine Translation. Our experiments, conducted on three language pairs, demonstrate that CPL outperforms not only the vanilla Transformer model but also other token-level and sequence-level baselines. Furthermore, the ablation study highlights the essential role of the proposed indicator function in achieving this improvement.- Anthology ID:
- 2024.findings-naacl.174
- Volume:
- Findings of the Association for Computational Linguistics: NAACL 2024
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Kevin Duh, Helena Gomez, Steven Bethard
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2723–2735
- Language:
- URL:
- https://aclanthology.org/2024.findings-naacl.174
- DOI:
- Cite (ACL):
- Jianfei He, Shichao Sun, Sen Peng, Jie Xu, Xiaohua Jia, and Wenjie Li. 2024. Contrastive Preference Learning for Neural Machine Translation. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2723–2735, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- Contrastive Preference Learning for Neural Machine Translation (He et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2024.findings-naacl.174.pdf