Abstract
Large scale multilingual pre-trained language models have shown promising results in zero- and few-shot cross-lingual tasks. However, recent studies have shown their lack of generalizability when the languages are structurally dissimilar. In this work, we propose a novel fine-tuning method based on co-training that aims to learn more generalized semantic equivalences as a complementary to multilingual language modeling using the unlabeled data in the target language. We also propose an adaption method based on contrastive learning to better capture the semantic relationship in the parallel data, when a few translation pairs are available. To show our method’s effectiveness, we conduct extensive experiments on cross-lingual inference and review classification tasks across various languages. We report significant gains compared to directly fine-tuning multilingual pre-trained models and other semi-supervised alternatives.- Anthology ID:
- 2021.emnlp-main.668
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 8492–8501
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.668
- DOI:
- 10.18653/v1/2021.emnlp-main.668
- Cite (ACL):
- Tao Yu and Shafiq Joty. 2021. Effective Fine-Tuning Methods for Cross-lingual Adaptation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8492–8501, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Effective Fine-Tuning Methods for Cross-lingual Adaptation (Yu & Joty, EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2021.emnlp-main.668.pdf
- Data
- XNLI