A Neural Model for Compositional Word Embeddings and Sentence Processing

Shalom Lappin, Jean-Philippe Bernardy


Abstract
We propose a new neural model for word embeddings, which uses Unitary Matrices as the primary device for encoding lexical information. It uses simple matrix multiplication to derive matrices for large units, yielding a sentence processing model that is strictly compositional, does not lose information over time steps, and is transparent, in the sense that word embeddings can be analysed regardless of context. This model does not employ activation functions, and so the network is fully accessible to analysis by the methods of linear algebra at each point in its operation on an input sequence. We test it in two NLP agreement tasks and obtain rule like perfect accuracy, with greater stability than current state-of-the-art systems. Our proposed model goes some way towards offering a class of computationally powerful deep learning systems that can be fully understood and compared to human cognitive processes for natural language learning and representation.
Anthology ID:
2022.cmcl-1.2
Volume:
Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Emmanuele Chersoni, Nora Hollenstein, Cassandra Jacobs, Yohei Oseki, Laurent Prévot, Enrico Santus
Venue:
CMCL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12–22
Language:
URL:
https://aclanthology.org/2022.cmcl-1.2
DOI:
10.18653/v1/2022.cmcl-1.2
Bibkey:
Cite (ACL):
Shalom Lappin and Jean-Philippe Bernardy. 2022. A Neural Model for Compositional Word Embeddings and Sentence Processing. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 12–22, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
A Neural Model for Compositional Word Embeddings and Sentence Processing (Lappin & Bernardy, CMCL 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl24-info/2022.cmcl-1.2.pdf
Video:
 https://preview.aclanthology.org/naacl24-info/2022.cmcl-1.2.mp4