Romain Couillet


2020

pdf bib
Word Representations Concentrate and This is Good News!
Romain Couillet | Yagmur Gizem Cinar | Eric Gaussier | Muhammad Imran
Proceedings of the 24th Conference on Computational Natural Language Learning

This article establishes that, unlike the legacy tf*idf representation, recent natural language representations (word embedding vectors) tend to exhibit a so-called concentration of measure phenomenon, in the sense that, as the representation size p and database size n are both large, their behavior is similar to that of large dimensional Gaussian random vectors. This phenomenon may have important consequences as machine learning algorithms for natural language data could be amenable to improvement, thereby providing new theoretical insights into the field of natural language processing.