Addressing Low-Resource Scenarios with Character-aware Embeddings

Sean Papay, Sebastian Padó, Ngoc Thang Vu


Abstract
Most modern approaches to computing word embeddings assume the availability of text corpora with billions of words. In this paper, we explore a setup where only corpora with millions of words are available, and many words in any new text are out of vocabulary. This setup is both of practical interests – modeling the situation for specific domains and low-resource languages – and of psycholinguistic interest, since it corresponds much more closely to the actual experiences and challenges of human language learning and use. We compare standard skip-gram word embeddings with character-based embeddings on word relatedness prediction. Skip-grams excel on large corpora, while character-based embeddings do well on small corpora generally and rare and complex words specifically. The models can be combined easily.
Anthology ID:
W18-1204
Volume:
Proceedings of the Second Workshop on Subword/Character LEvel Models
Month:
June
Year:
2018
Address:
New Orleans
Editors:
Manaal Faruqui, Hinrich Schütze, Isabel Trancoso, Yulia Tsvetkov, Yadollah Yaghoobzadeh
Venue:
SCLeM
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
32–37
Language:
URL:
https://aclanthology.org/W18-1204
DOI:
10.18653/v1/W18-1204
Bibkey:
Cite (ACL):
Sean Papay, Sebastian Padó, and Ngoc Thang Vu. 2018. Addressing Low-Resource Scenarios with Character-aware Embeddings. In Proceedings of the Second Workshop on Subword/Character LEvel Models, pages 32–37, New Orleans. Association for Computational Linguistics.
Cite (Informal):
Addressing Low-Resource Scenarios with Character-aware Embeddings (Papay et al., SCLeM 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-1/W18-1204.pdf