X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity
Taejun Yun, Jinhyeon Kim, Deokyeong Kang, Seonghoon Lim, Jihoon Kim, Taeuk Kim
Abstract
Cross-lingual transfer (XLT) is an emergent ability of multilingual language models that preserves their performance on a task to a significant extent when evaluated in languages that were not included in the fine-tuning process. While English, due to its widespread usage, is typically regarded as the primary language for model adaption in various tasks, recent studies have revealed that the efficacy of XLT can be amplified by selecting the most appropriate source languages based on specific conditions. In this work, we propose the utilization of sub-network similarity between two languages as a proxy for predicting the compatibility of the languages in the context of XLT. Our approach is model-oriented, better reflecting the inner workings of foundation models. In addition, it requires only a moderate amount of raw text from candidate languages, distinguishing it from the majority of previous methods that rely on external resources. In experiments, we demonstrate that our method is more effective than baselines across diverse tasks. Specifically, it shows proficiency in ranking candidates for zero-shot XLT, achieving an improvement of 4.6% on average in terms of NDCG@3. We also provide extensive analyses that confirm the utility of sub-networks for XLT prediction.- Anthology ID:
- 2023.findings-emnlp.875
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 13131–13144
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2023.findings-emnlp.875/
- DOI:
- 10.18653/v1/2023.findings-emnlp.875
- Cite (ACL):
- Taejun Yun, Jinhyeon Kim, Deokyeong Kang, Seonghoon Lim, Jihoon Kim, and Taeuk Kim. 2023. X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13131–13144, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity (Yun et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2023.findings-emnlp.875.pdf