@inproceedings{wada-etal-2021-learning,
    title = "Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora",
    author = "Wada, Takashi  and
      Iwata, Tomoharu  and
      Matsumoto, Yuji  and
      Baldwin, Timothy  and
      Lau, Jey Han",
    editor = "Ataman, Duygu  and
      Birch, Alexandra  and
      Conneau, Alexis  and
      Firat, Orhan  and
      Ruder, Sebastian  and
      Sahin, Gozde Gul",
    booktitle = "Proceedings of the 1st Workshop on Multilingual Representation Learning",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.mrl-1.2",
    doi = "10.18653/v1/2021.mrl-1.2",
    pages = "16--31",
    abstract = "We propose a new approach for learning contextualised cross-lingual word embeddings based on a small parallel corpus (e.g. a few hundred sentence pairs). Our method obtains word embeddings via an LSTM encoder-decoder model that simultaneously translates and reconstructs an input sentence. Through sharing model parameters among different languages, our model jointly trains the word embeddings in a common cross-lingual space. We also propose to combine word and subword embeddings to make use of orthographic similarities across different languages. We base our experiments on real-world data from endangered languages, namely Yongning Na, Shipibo-Konibo, and Griko. Our experiments on bilingual lexicon induction and word alignment tasks show that our model outperforms existing methods by a large margin for most language pairs. These results demonstrate that, contrary to common belief, an encoder-decoder translation model is beneficial for learning cross-lingual representations even in extremely low-resource conditions. Furthermore, our model also works well on high-resource conditions, achieving state-of-the-art performance on a German-English word-alignment task.",
}