@inproceedings{cao-etal-2024-exploring,
    title = "Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation",
    author = "Cao, Zhe  and
      Qu, Zhi  and
      Kamigaito, Hidetaka  and
      Watanabe, Taro",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2024.emnlp-main.1177/",
    doi = "10.18653/v1/2024.emnlp-main.1177",
    pages = "21142--21157",
    abstract = "Multilingual neural machine translation models support fine-tuning hundreds of languages simultaneously. However, fine-tuning on full parameters solely is inefficient potentially leading to negative interactions among languages. In this work, we demonstrate that the fine-tuning for a language occurs in its intrinsic language-specific subspace with a tiny fraction of entire parameters. Thus, we propose language-specific LoRA to isolate intrinsic language-specific subspaces. Furthermore, we propose architecture learning techniques and introduce a gradual pruning schedule during fine-tuning to exhaustively explore the optimal setting and the minimal intrinsic subspaces for each language, resulting in a lightweight yet effective fine-tuning procedure. The experimental results on a 12-language subset and a 30-language subset of FLORES-101 show that our methods not only outperform full-parameter fine-tuning up to 2.25 spBLEU scores but also reduce trainable parameters to 0.4{\%} for high and medium-resource languages and 1.6{\%} for low-resource ones."
}