Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation

Tong Su, Xin Peng, Sarubi Thillainathan, David Guzmán, Surangika Ranathunga, En-Shiun Lee

Abstract
Parameter-efficient fine-tuning (PEFT) methods are increasingly vital in adapting large-scale pre-trained language models for diverse tasks, offering a balance between adaptability and computational efficiency. They are important in Low-Resource Language (LRL) Neural Machine Translation (NMT) to enhance translation accuracy with minimal resources. However, their practical effectiveness varies significantly across different languages. We conducted comprehensive empirical experiments with varying LRL domains and sizes to evaluate the performance of 8 PEFT methods with in total of 15 architectures using the SacreBLEU score. We showed that 6 PEFT architectures outperform the baseline for both in-domain and out-domain tests and the Houlsby+Inversion adapter has the best performance overall, proving the effectiveness of PEFT methods.
Anthology ID:
2024.findings-naacl.263
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4217–4225
Language:
URL:
https://aclanthology.org/2024.findings-naacl.263
DOI:
10.18653/v1/2024.findings-naacl.263
Bibkey:
Cite (ACL):
Tong Su, Xin Peng, Sarubi Thillainathan, David Guzmán, Surangika Ranathunga, and En-Shiun Lee. 2024. Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4217–4225, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation (Su et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl-24-ws-corrections/2024.findings-naacl.263.pdf
PDF Search