Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders
Ivan Vulić, Goran Glavaš, Fangyu Liu, Nigel Collier, Edoardo Maria Ponti, Anna Korhonen
Abstract
Pretrained multilingual language models (LMs) can be successfully transformed into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional fine-tuning or model distillation with parallel data. However, it remains unclear how to best leverage them to represent sub-sentence lexical items (i.e., words and phrases) in cross-lingual lexical tasks. In this work, we probe SEs for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a simple yet efficient method for exposing the cross-lingual lexical knowledge by means of additional fine-tuning through inexpensive contrastive learning that requires only a small amount of word translation pairs. Using bilingual lexical induction (BLI), cross-lingual lexical semantic similarity, and cross-lingual entity linking as lexical probing tasks, we report substantial gains on standard benchmarks (e.g., +10 Precision@1 points in BLI). The results indicate that the SEs such as LaBSE can be ‘rewired’ into effective cross-lingual lexical encoders via the contrastive learning procedure, and that it is possible to expose more cross-lingual lexical knowledge compared to using them as off-the-shelf SEs. This way, we also provide an effective tool for harnessing ‘covert’ multilingual lexical knowledge hidden in multilingual sentence encoders.- Anthology ID:
- 2023.eacl-main.153
- Volume:
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2089–2105
- Language:
- URL:
- https://aclanthology.org/2023.eacl-main.153
- DOI:
- 10.18653/v1/2023.eacl-main.153
- Cite (ACL):
- Ivan Vulić, Goran Glavaš, Fangyu Liu, Nigel Collier, Edoardo Maria Ponti, and Anna Korhonen. 2023. Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2089–2105, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders (Vulić et al., EACL 2023)
- PDF:
- https://preview.aclanthology.org/revert-3132-ingestion-checklist/2023.eacl-main.153.pdf