Beyond Context: A New Perspective for Word Embeddings

Yichu Zhou, Vivek Srikumar


Abstract
Most word embeddings today are trained by optimizing a language modeling goal of scoring words in their context, modeled as a multi-class classification problem. In this paper, we argue that, despite the successes of this assumption, it is incomplete: in addition to its context, orthographical or morphological aspects of words can offer clues about their meaning. We define a new modeling framework for training word embeddings that captures this intuition. Our framework is based on the well-studied problem of multi-label classification and, consequently, exposes several design choices for featurizing words and contexts, loss functions for training and score normalization. Indeed, standard models such as CBOW and fasttext are specific choices along each of these axes. We show via experiments that by combining feature engineering with embedding learning, our method can outperform CBOW using only 10% of the training data in both the standard word embedding evaluations and also text classification experiments.
Anthology ID:
S19-1003
Volume:
Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Venues:
SemEval | *SEM
SIGs:
SIGLEX | SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
22–32
Language:
URL:
https://aclanthology.org/S19-1003
DOI:
10.18653/v1/S19-1003
Bibkey:
Cite (ACL):
Yichu Zhou and Vivek Srikumar. 2019. Beyond Context: A New Perspective for Word Embeddings. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 22–32, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
Beyond Context: A New Perspective for Word Embeddings (Zhou & Srikumar, SemEval-*SEM 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/S19-1003.pdf