Abstract
In this paper we define a measure of dependency between two random variables, based on the Jensen-Shannon (JS) divergence between their joint distribution and the product of their marginal distributions. Then, we show that word2vec’s skip-gram with negative sampling embedding algorithm finds the optimal low-dimensional approximation of this JS dependency measure between the words and their contexts. The gap between the optimal score and the low-dimensional approximation is demonstrated on a standard text corpus.- Anthology ID:
- P17-2026
- Volume:
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- July
- Year:
- 2017
- Address:
- Vancouver, Canada
- Editors:
- Regina Barzilay, Min-Yen Kan
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 167–171
- Language:
- URL:
- https://aclanthology.org/P17-2026
- DOI:
- 10.18653/v1/P17-2026
- Cite (ACL):
- Oren Melamud and Jacob Goldberger. 2017. Information-Theory Interpretation of the Skip-Gram Negative-Sampling Objective Function. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 167–171, Vancouver, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Information-Theory Interpretation of the Skip-Gram Negative-Sampling Objective Function (Melamud & Goldberger, ACL 2017)
- PDF:
- https://preview.aclanthology.org/fix-dup-bibkey/P17-2026.pdf