Abstract
With the development of multilingual pre-trained language models (mPLMs), zero-shot cross-lingual transfer shows great potential. To further improve the performance of cross-lingual transfer, many studies have explored representation misalignment caused by morphological differences but neglected the misalignment caused by the anisotropic distribution of contextual representations. In this work, we propose enhanced isotropy and constrained code-switching for zero-shot cross-lingual transfer to alleviate the problem of misalignment caused by the anisotropic representations and maintain syntactic structural knowledge. Extensive experiments on three zero-shot cross-lingual transfer tasks demonstrate that our method gains significant improvements over strong mPLM backbones and further improves the state-of-the-art methods.- Anthology ID:
- 2023.findings-emnlp.545
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 8104–8118
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.545
- DOI:
- 10.18653/v1/2023.findings-emnlp.545
- Cite (ACL):
- Yixin Ji, Jikai Wang, Juntao Li, Hai Ye, and Min Zhang. 2023. Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8104–8118, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models (Ji et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2023.findings-emnlp.545.pdf