Multi-task pre-finetuning for zero-shot cross lingual transfer
Abstract
Building machine learning models for low resource languages is extremely challenging due to the lack of available training data (either un-annotated or annotated). To support such scenarios, zero-shot cross lingual transfer is used where the machine learning model is trained on a resource rich language and is directly tested on the resource poor language. In this paper, we present a technique which improves the performance of zero-shot cross lingual transfer. Our method performs multi-task pre-finetuning on a resource rich language using a multilingual pre-trained model. The pre-finetuned model is then tested in a zero-shot manner on the resource poor languages. We test the performance of our method on 8 languages and for two tasks, namely, Intent Classification (IC) & Named Entity Recognition (NER) using the MultiAtis++ dataset. The results showed that our method improves IC performance in 7 out of 8 languages and NER performance in 4 languages. Our method also leads to faster convergence during finetuning. The usage of pre-finetuning demonstrates a data efficient way for supporting new languages and geographies across the world.- Anthology ID:
- 2021.icon-main.57
- Volume:
- Proceedings of the 18th International Conference on Natural Language Processing (ICON)
- Month:
- December
- Year:
- 2021
- Address:
- National Institute of Technology Silchar, Silchar, India
- Editors:
- Sivaji Bandyopadhyay, Sobha Lalitha Devi, Pushpak Bhattacharyya
- Venue:
- ICON
- SIG:
- Publisher:
- NLP Association of India (NLPAI)
- Note:
- Pages:
- 474–480
- Language:
- URL:
- https://aclanthology.org/2021.icon-main.57
- DOI:
- Cite (ACL):
- Moukthika Yerramilli, Pritam Varma, and Anurag Dwarakanath. 2021. Multi-task pre-finetuning for zero-shot cross lingual transfer. In Proceedings of the 18th International Conference on Natural Language Processing (ICON), pages 474–480, National Institute of Technology Silchar, Silchar, India. NLP Association of India (NLPAI).
- Cite (Informal):
- Multi-task pre-finetuning for zero-shot cross lingual transfer (Yerramilli et al., ICON 2021)
- PDF:
- https://preview.aclanthology.org/teach-a-man-to-fish/2021.icon-main.57.pdf
- Data
- GLUE