Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models
Abstract
Researchers have recently demonstrated that tying the neural weights between the input look-up table and the output classification layer can improve training and lower perplexity on sequence learning tasks such as language modelling. Such a procedure is possible due to the design of the softmax classification layer, which previous work has shown to comprise a viable set of semantic representations for the model vocabulary, and these these output embeddings are known to perform well on word similarity benchmarks. In this paper, we make meaningful comparisons between the input and output embeddings and other SOTA distributional models to gain a better understanding of the types of information they represent. We also construct a new set of word embeddings using the output embeddings to create locally-optimal approximations for the intermediate representations from the language model. These locally-optimal embeddings demonstrate excellent performance across all our evaluations.- Anthology ID:
- 2020.conll-1.36
- Volume:
- Proceedings of the 24th Conference on Computational Natural Language Learning
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Raquel Fernández, Tal Linzen
- Venue:
- CoNLL
- SIG:
- SIGNLL
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 442–454
- Language:
- URL:
- https://aclanthology.org/2020.conll-1.36
- DOI:
- 10.18653/v1/2020.conll-1.36
- Cite (ACL):
- Steven Derby, Paul Miller, and Barry Devereux. 2020. Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 442–454, Online. Association for Computational Linguistics.
- Cite (Informal):
- Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models (Derby et al., CoNLL 2020)
- PDF:
- https://preview.aclanthology.org/ingest-2024-clasp/2020.conll-1.36.pdf
- Code
- stevend94/conll2020
- Data
- MPQA Opinion Corpus, Penn Treebank, SICK, SST, SST-2, SST-5