Investigating Cross-Lingual Alignment Methods for Contextualized Embeddings with Token-Level Evaluation

Qianchu Liu, Diana McCarthy, Ivan Vulić, Anna Korhonen

[How to correct problems with metadata yourself]

Abstract
In this paper, we present a thorough investigation on methods that align pre-trained contextualized embeddings into shared cross-lingual context-aware embedding space, providing strong reference benchmarks for future context-aware crosslingual models. We propose a novel and challenging task, Bilingual Token-level Sense Retrieval (BTSR). It specifically evaluates the accurate alignment of words with the same meaning in cross-lingual non-parallel contexts, currently not evaluated by existing tasks such as Bilingual Contextual Word Similarity and Sentence Retrieval. We show how the proposed BTSR task highlights the merits of different alignment methods. In particular, we find that using context average type-level alignment is effective in transferring monolingual contextualized embeddings cross-lingually especially in non-parallel contexts, and at the same time improves the monolingual space. Furthermore, aligning independently trained models yields better performance than aligning multilingual embeddings with shared vocabulary.
Anthology ID:
K19-1004
Volume:
Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Mohit Bansal, Aline Villavicencio
Venue:
CoNLL
SIG:
SIGNLL
Publisher:
Association for Computational Linguistics
Note:
Pages:
33–43
Language:
URL:
https://aclanthology.org/K19-1004
DOI:
10.18653/v1/K19-1004
Bibkey:
Cite (ACL):
Qianchu Liu, Diana McCarthy, Ivan Vulić, and Anna Korhonen. 2019. Investigating Cross-Lingual Alignment Methods for Contextualized Embeddings with Token-Level Evaluation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 33–43, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Investigating Cross-Lingual Alignment Methods for Contextualized Embeddings with Token-Level Evaluation (Liu et al., CoNLL 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/teach-a-man-to-fish/K19-1004.pdf
PDF Search