Subword-level Composition Functions for Learning Word Embeddings

Bofang Li, Aleksandr Drozd, Tao Liu, Xiaoyong Du


Abstract
Subword-level information is crucial for capturing the meaning and morphology of words, especially for out-of-vocabulary entries. We propose CNN- and RNN-based subword-level composition functions for learning word embeddings, and systematically compare them with popular word-level and subword-level models (Skip-Gram and FastText). Additionally, we propose a hybrid training scheme in which a pure subword-level model is trained jointly with a conventional word-level embedding model based on lookup-tables. This increases the fitness of all types of subword-level word embeddings; the word-level embeddings can be discarded after training, leaving only compact subword-level representation with much smaller data volume. We evaluate these embeddings on a set of intrinsic and extrinsic tasks, showing that subword-level models have advantage on tasks related to morphology and datasets with high OOV rate, and can be combined with other types of embeddings.
Anthology ID:
W18-1205
Volume:
Proceedings of the Second Workshop on Subword/Character LEvel Models
Month:
June
Year:
2018
Address:
New Orleans
Venue:
SCLeM
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
38–48
Language:
URL:
https://aclanthology.org/W18-1205
DOI:
10.18653/v1/W18-1205
Bibkey:
Cite (ACL):
Bofang Li, Aleksandr Drozd, Tao Liu, and Xiaoyong Du. 2018. Subword-level Composition Functions for Learning Word Embeddings. In Proceedings of the Second Workshop on Subword/Character LEvel Models, pages 38–48, New Orleans. Association for Computational Linguistics.
Cite (Informal):
Subword-level Composition Functions for Learning Word Embeddings (Li et al., SCLeM 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/W18-1205.pdf
Data
IMDb Movie Reviews