Abstract
Nearest Neighbor Machine Translation (kNN-MT) has achieved great success in domain adaptation tasks by integrating pre-trained Neural Machine Translation (NMT) models with domain-specific token-level retrieval. However, the reasons underlying its success have not been thoroughly investigated. In this paper, we comprehensively analyze kNN-MT through theoretical and empirical studies. Initially, we provide new insights into the working mechanism of kNN-MT as an efficient technique to implicitly execute gradient descent on the output projection layer of NMT, indicating that it is a specific case of model fine-tuning. Subsequently, we conduct multi-domain experiments and word-level analysis to examine the differences in performance between kNN-MT and entire-model fine-tuning. Our findings suggest that: (i) Incorporating kNN-MT with adapters yields comparable translation performance to fine-tuning on in-domain test sets, while achieving better performance on out-of-domain test sets; (ii) Fine-tuning significantly outperforms kNN-MT on the recall of in-domain low-frequency words, but this gap could be bridged by optimizing the context representations with additional adapter layers.- Anthology ID:
- 2023.emnlp-main.964
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 15592–15608
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.964
- DOI:
- 10.18653/v1/2023.emnlp-main.964
- Cite (ACL):
- Ruize Gao, Zhirui Zhang, Yichao Du, Lemao Liu, and Rui Wang. 2023. Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layer. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15592–15608, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layer (Gao et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2023.emnlp-main.964.pdf