Predicting and interpreting embeddings for out of vocabulary words in downstream tasks

Nicolas Garneau, Jean-Samuel Leboeuf, Luc Lamontagne


Abstract
We propose a novel way to handle out of vocabulary (OOV) words in downstream natural language processing (NLP) tasks. We implement a network that predicts useful embeddings for OOV words based on their morphology and on the context in which they appear. Our model also incorporates an attention mechanism indicating the focus allocated to the left context words, the right context words or the word’s characters, hence making the prediction more interpretable. The model is a “drop-in” module that is jointly trained with the downstream task’s neural network, thus producing embeddings specialized for the task at hand. When the task is mostly syntactical, we observe that our model aims most of its attention on surface form characters. On the other hand, for tasks more semantical, the network allocates more attention to the surrounding words. In all our tests, the module helps the network to achieve better performances in comparison to the use of simple random embeddings.
Anthology ID:
W18-5439
Volume:
Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
Month:
November
Year:
2018
Address:
Brussels, Belgium
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
331–333
Language:
URL:
https://aclanthology.org/W18-5439
DOI:
10.18653/v1/W18-5439
Bibkey:
Cite (ACL):
Nicolas Garneau, Jean-Samuel Leboeuf, and Luc Lamontagne. 2018. Predicting and interpreting embeddings for out of vocabulary words in downstream tasks. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 331–333, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Predicting and interpreting embeddings for out of vocabulary words in downstream tasks (Garneau et al., EMNLP 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/auto-file-uploads/W18-5439.pdf