attr2vec: Jointly Learning Word and Contextual Attribute Embeddings with Factorization Machines

Fabio Petroni, Vassilis Plachouras, Timothy Nugent, Jochen L. Leidner


Abstract
The widespread use of word embeddings is associated with the recent successes of many natural language processing (NLP) systems. The key approach of popular models such as word2vec and GloVe is to learn dense vector representations from the context of words. More recently, other approaches have been proposed that incorporate different types of contextual information, including topics, dependency relations, n-grams, and sentiment. However, these models typically integrate only limited additional contextual information, and often in ad hoc ways. In this work, we introduce attr2vec, a novel framework for jointly learning embeddings for words and contextual attributes based on factorization machines. We perform experiments with different types of contextual information. Our experimental results on a text classification task demonstrate that using attr2vec to jointly learn embeddings for words and Part-of-Speech (POS) tags improves results compared to learning the embeddings independently. Moreover, we use attr2vec to train dependency-based embeddings and we show that they exhibit higher similarity between functionally related words compared to traditional approaches.
Anthology ID:
N18-1042
Volume:
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
453–462
Language:
URL:
https://aclanthology.org/N18-1042
DOI:
10.18653/v1/N18-1042
Bibkey:
Cite (ACL):
Fabio Petroni, Vassilis Plachouras, Timothy Nugent, and Jochen L. Leidner. 2018. attr2vec: Jointly Learning Word and Contextual Attribute Embeddings with Factorization Machines. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 453–462, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
attr2vec: Jointly Learning Word and Contextual Attribute Embeddings with Factorization Machines (Petroni et al., NAACL 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/N18-1042.pdf