Irish-based Large Language Model with Extreme Low-Resource Settings in Machine Translation

Khanh-Tung Tran, Barry O’Sullivan, Hoang Nguyen

Abstract
Large Language Models (LLMs) have demonstrated exceptional performances in a wide range of natural language processing tasks. However, their success does not always extend to machine translation, particularly in challenging scenarios such as translating low-resource languages. This study investigates the multilingual capability of LLMs, with a case study on Irish, an extremely low-resource language, focusing on translation tasks between English and Irish. We propose a dynamic, efficient language adaptation framework for English-centric LLMs, which involves layer-specific adjustments and subsequent fine-tuning for machine translation. Our findings highlight several key insights: (1) different layers in the LLM serve distinct functions such as language understanding and task reasoning, (2) effective translation requires extensive pre-training on both source and target languages, and (3) targeted fine-tuning for machine translation leads to significant improvements of 36.7% for English to Irish and 133.4% for Irish to English compared to the previous state-of-the-art.
Anthology ID:
2024.loresmt-1.20
Volume:
Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2024)
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Atul Kr. Ojha, Chao-hong Liu, Ekaterina Vylomova, Flammie Pirinen, Jade Abbott, Jonathan Washington, Nathaniel Oco, Valentin Malykh, Varvara Logacheva, Xiaobing Zhao
Venues:
LoResMT | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
193–202
Language:
URL:
https://aclanthology.org/2024.loresmt-1.20
DOI:
10.18653/v1/2024.loresmt-1.20
Bibkey:
Cite (ACL):
Khanh-Tung Tran, Barry O’Sullivan, and Hoang Nguyen. 2024. Irish-based Large Language Model with Extreme Low-Resource Settings in Machine Translation. In Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2024), pages 193–202, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Irish-based Large Language Model with Extreme Low-Resource Settings in Machine Translation (Tran et al., LoResMT-WS 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/dois-2013-emnlp/2024.loresmt-1.20.pdf
PDF Search