Cross-lingual Sentence Embedding using Multi-Task Learning
Koustava Goswami, Sourav Dutta, Haytham Assem, Theodorus Fransen, John P. McCrae
Abstract
Multilingual sentence embeddings capture rich semantic information not only for measuring similarity between texts but also for catering to a broad range of downstream cross-lingual NLP tasks. State-of-the-art multilingual sentence embedding models require large parallel corpora to learn efficiently, which confines the scope of these models. In this paper, we propose a novel sentence embedding framework based on an unsupervised loss function for generating effective multilingual sentence embeddings, eliminating the need for parallel corpora. We capture semantic similarity and relatedness between sentences using a multi-task loss function for training a dual encoder model mapping different languages onto the same vector space. We demonstrate the efficacy of an unsupervised as well as a weakly supervised variant of our framework on STS, BUCC and Tatoeba benchmark tasks. The proposed unsupervised sentence embedding framework outperforms even supervised state-of-the-art methods for certain under-resourced languages on the Tatoeba dataset and on a monolingual benchmark. Further, we show enhanced zero-shot learning capabilities for more than 30 languages, with the model being trained on only 13 languages. Our model can be extended to a wide range of languages from any language family, as it overcomes the requirement of parallel corpora for training.- Anthology ID:
- 2021.emnlp-main.716
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 9099–9113
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.716
- DOI:
- 10.18653/v1/2021.emnlp-main.716
- Cite (ACL):
- Koustava Goswami, Sourav Dutta, Haytham Assem, Theodorus Fransen, and John P. McCrae. 2021. Cross-lingual Sentence Embedding using Multi-Task Learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9099–9113, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Cross-lingual Sentence Embedding using Multi-Task Learning (Goswami et al., EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2021.emnlp-main.716.pdf
- Data
- BUCC, SST, SST-2, SentEval, XNLI