Fixing Rogue Memorization in Many-to-One Multilingual Translators of Extremely-Low-Resource Languages by Rephrasing Training Samples

Paulo Cavalin, Pedro Henrique Domingues, Claudio Pinhanez, Julio Nogima

Abstract
In this paper we study the fine-tuning of pre-trained large high-resource language models (LLMs) into many-to-one multilingual machine translators for extremely-low-resource languages such as endangered Indigenous languages. We explore those issues using datasets created from pseudo-parallel translations to English of The Bible written in 39 Brazilian Indigenous languages using mBART50 and WMT19 as pre-trained models and multiple translation metrics. We examine bilingual and multilingual models and show that, according to machine translation metrics, same-linguistic family models tend to perform best. However, we also found that many-to-one multilingual systems have a tendency to learn a “rogue” strategy of storing output strings from the training data in the LLM structure and retrieving them instead of performing actual translations. We show that rephrasing the output of the training samples seems to solve the problem.
Anthology ID:
2024.naacl-long.253
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4503–4514
Language:
URL:
https://aclanthology.org/2024.naacl-long.253
DOI:
10.18653/v1/2024.naacl-long.253
Bibkey:
Cite (ACL):
Paulo Cavalin, Pedro Henrique Domingues, Claudio Pinhanez, and Julio Nogima. 2024. Fixing Rogue Memorization in Many-to-One Multilingual Translators of Extremely-Low-Resource Languages by Rephrasing Training Samples. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4503–4514, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Fixing Rogue Memorization in Many-to-One Multilingual Translators of Extremely-Low-Resource Languages by Rephrasing Training Samples (Cavalin et al., NAACL 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2024.naacl-long.253.pdf
PDF Search