A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning

Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, Rong Tian, Yiren Chen


Abstract
Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries, which are expensive and impractical for low-resource languages. To disengage from these dependencies, researchers have explored training multilingual models on English-only resources and transferring them to low-resource languages. However, its effect is limited by the gap between embedding clusters of different languages. To address this issue, we propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embeddings without semantic loss, thereby improving cross-lingual transferability. Experimental results on mBERT and XLM-R demonstrate that our method significantly outperforms previous works on the zero-shot cross-lingual text classification task and can obtain a better multilingual alignment.
Anthology ID:
2022.coling-1.385
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
4372–4380
Language:
URL:
https://aclanthology.org/2022.coling-1.385
DOI:
Bibkey:
Cite (ACL):
Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, Rong Tian, and Yiren Chen. 2022. A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4372–4380, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning (Ding et al., COLING 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.coling-1.385.pdf
Code
 kb-ding/ear
Data
XNLI