Abstract
Most word embeddings today are trained by optimizing a language modeling goal of scoring words in their context, modeled as a multi-class classification problem. In this paper, we argue that, despite the successes of this assumption, it is incomplete: in addition to its context, orthographical or morphological aspects of words can offer clues about their meaning. We define a new modeling framework for training word embeddings that captures this intuition. Our framework is based on the well-studied problem of multi-label classification and, consequently, exposes several design choices for featurizing words and contexts, loss functions for training and score normalization. Indeed, standard models such as CBOW and fasttext are specific choices along each of these axes. We show via experiments that by combining feature engineering with embedding learning, our method can outperform CBOW using only 10% of the training data in both the standard word embedding evaluations and also text classification experiments.- Anthology ID:
- S19-1003
- Volume:
- Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)
- Month:
- June
- Year:
- 2019
- Address:
- Minneapolis, Minnesota
- Editors:
- Rada Mihalcea, Ekaterina Shutova, Lun-Wei Ku, Kilian Evang, Soujanya Poria
- Venue:
- *SEM
- SIGs:
- SIGSEM | SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 22–32
- Language:
- URL:
- https://aclanthology.org/S19-1003
- DOI:
- 10.18653/v1/S19-1003
- Cite (ACL):
- Yichu Zhou and Vivek Srikumar. 2019. Beyond Context: A New Perspective for Word Embeddings. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 22–32, Minneapolis, Minnesota. Association for Computational Linguistics.
- Cite (Informal):
- Beyond Context: A New Perspective for Word Embeddings (Zhou & Srikumar, *SEM 2019)
- PDF:
- https://preview.aclanthology.org/ingest-bitext-workshop/S19-1003.pdf