Abstract
While static word embedding models are known to represent linguistic analogies as parallel lines in high-dimensional space, the underlying mechanism as to why they result in such geometric structures remains obscure. We find that an elementary contrastive-style method employed over distributional information performs competitively with popular word embedding models on analogy recovery tasks, while achieving dramatic speedups in training time. Further, we demonstrate that a contrastive loss is sufficient to create these parallel structures in word embeddings, and establish a precise relationship between the co-occurrence statistics and the geometric structure of the resulting word embeddings.- Anthology ID:
- 2023.repl4nlp-1.14
- Volume:
- Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Burcu Can, Maximilian Mozes, Samuel Cahyawijaya, Naomi Saphra, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Chen Zhao, Isabelle Augenstein, Anna Rogers, Kyunghyun Cho, Edward Grefenstette, Lena Voita
- Venue:
- RepL4NLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 164–173
- Language:
- URL:
- https://aclanthology.org/2023.repl4nlp-1.14
- DOI:
- 10.18653/v1/2023.repl4nlp-1.14
- Cite (ACL):
- Narutatsu Ri, Fei-Tzin Lee, and Nakul Verma. 2023. Contrastive Loss is All You Need to Recover Analogies as Parallel Lines. In Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023), pages 164–173, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Contrastive Loss is All You Need to Recover Analogies as Parallel Lines (Ri et al., RepL4NLP 2023)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2023.repl4nlp-1.14.pdf