Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization
Ningyu Xu, Qi Zhang, Jingting Ye, Menghan Zhang, Xuanjing Huang
Abstract
Large language models (LLMs) have exhibited considerable cross-lingual generalization abilities, whereby they implicitly transfer knowledge across languages. However, the transfer is not equally successful for all languages, especially for low-resource ones, which poses an ongoing challenge. It is unclear whether we have reached the limits of implicit cross-lingual generalization and if explicit knowledge transfer is viable. In this paper, we investigate the potential for explicitly aligning conceptual correspondence between languages to enhance cross-lingual generalization. Using the syntactic aspect of language as a testbed, our analyses of 43 languages reveal a high degree of alignability among the spaces of structural concepts within each language for both encoder-only and decoder-only LLMs. We then propose a meta-learning-based method to learn to align conceptual spaces of different languages, which facilitates zero-shot and few-shot generalization in concept classification and also offers insights into the cross-lingual in-context learning phenomenon. Experiments on syntactic analysis tasks show that our approach achieves competitive results with state-of-the-art methods and narrows the performance gap between languages, particularly benefiting those with limited resources.- Anthology ID:
- 2023.findings-emnlp.931
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 13951–13976
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.931
- DOI:
- 10.18653/v1/2023.findings-emnlp.931
- Cite (ACL):
- Ningyu Xu, Qi Zhang, Jingting Ye, Menghan Zhang, and Xuanjing Huang. 2023. Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13951–13976, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization (Xu et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/ingest-2024-clasp/2023.findings-emnlp.931.pdf