Abstract
For multilingual training, we present CrossInit, an initialization method that initializes embeddings into similar geometrical structures across languages in an unsupervised manner. CrossInit leverages a common cognitive linguistic mechanism, Zipf’s law, which indicates that similar concepts across languages have similar word ranks or frequencies in their monolingual corpora. Instead of considering point-to-point alignments based on ranks, CrossInit considers the same span of consecutive ranks in each language as the Positive pairs for alignment, while others out of the span are used as Negative pairs. CrossInit then employs Contrastive Learning to iteratively refine randomly initialized embeddings for similar geometrical structures across languages. Our experiments on Unsupervised NMT, XNLI, and MLQA showed significant gains in low-resource and dissimilar languages after applying CrossInit.- Anthology ID:
- 2024.findings-acl.358
- Original:
- 2024.findings-acl.358v1
- Version 2:
- 2024.findings-acl.358v2
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2024
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Lun-Wei Ku, Andre Martins, Vivek Srikumar
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5997–6007
- Language:
- URL:
- https://aclanthology.org/2024.findings-acl.358
- DOI:
- 10.18653/v1/2024.findings-acl.358
- Cite (ACL):
- Xi Ai and Zhiyong Huang. 2024. Zero-shot Cross-lingual Alignment for Embedding Initialization. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5997–6007, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Zero-shot Cross-lingual Alignment for Embedding Initialization (Ai & Huang, Findings 2024)
- PDF:
- https://preview.aclanthology.org/autopr/2024.findings-acl.358.pdf