Probing the Emergence of Cross-lingual Alignment during LLM Training

Hetong Wang, Pasquale Minervini, Edoardo Ponti

Abstract
Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance. We speculate that this is predicated on their ability to align languages without explicit supervision from parallel sentences. While representations of translationally equivalent sentences in different languages are known to be similar after convergence, however, it remains unclear how such cross-lingual alignment emerges during pre-training of LLMs. Our study leverages intrinsic probing techniques, which identify which subsets of neurons encode linguistic features, to correlate the degree of cross-lingual neuron overlap with the zero-shot cross-lingual transfer performance for a given model. In particular, we rely on checkpoints of BLOOM, a multilingual autoregressive LLM, across different training steps and model scales. We observe a high correlation between neuron overlap and downstream performance, which supports our hypothesis on the conditions leading to effective cross-lingual transfer. Interestingly, we also detect a degradation of both implicit alignment and multilingual abilities in certain phases of the pre-training process, providing new insights into the multilingual pretraining dynamics.
Anthology ID:
2024.findings-acl.724
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12159–12173
Language:
URL:
https://aclanthology.org/2024.findings-acl.724
DOI:
10.18653/v1/2024.findings-acl.724
Bibkey:
Cite (ACL):
Hetong Wang, Pasquale Minervini, and Edoardo Ponti. 2024. Probing the Emergence of Cross-lingual Alignment during LLM Training. In Findings of the Association for Computational Linguistics ACL 2024, pages 12159–12173, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
Probing the Emergence of Cross-lingual Alignment during LLM Training (Wang et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2024.findings-acl.724.pdf
PDF Search