Generalised Spherical Text Embedding
Souvik Banerjee, Bamdev Mishra, Pratik Jawanpuria, Manish Shrivastava Shrivastava
Abstract
This paper aims to provide an unsupervised modelling approach that allows for a more flexible representation of text embeddings. It jointly encodes the words and the paragraphs as individual matrices of arbitrary column dimension with unit Frobenius norm. The representation is also linguistically motivated with the introduction of a metric for the ambient space in which we train the embeddings that calculates the similarity between matrices of unequal number of columns. Thus, the proposed modelling and the novel similarity metric exploits the matrix structure of embeddings. We then go on to show that the same matrices can be reshaped into vectors of unit norm and transform our problem into an optimization problem in a spherical manifold for optimization simplicity. Given the total number of matrices we are dealing with, which is equal to the vocab size plus the total number of documents in the corpus, this makes the training of an otherwise expensive non-linear model extremely efficient. We also quantitatively verify the quality of our text embeddings by showing that they demonstrate improved results in document classification, document clustering and semantic textual similarity benchmark tests.- Anthology ID:
- 2022.icon-main.11
- Volume:
- Proceedings of the 19th International Conference on Natural Language Processing (ICON)
- Month:
- December
- Year:
- 2022
- Address:
- New Delhi, India
- Editors:
- Md. Shad Akhtar, Tanmoy Chakraborty
- Venue:
- ICON
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 80–85
- Language:
- URL:
- https://aclanthology.org/2022.icon-main.11
- DOI:
- Cite (ACL):
- Souvik Banerjee, Bamdev Mishra, Pratik Jawanpuria, and Manish Shrivastava Shrivastava. 2022. Generalised Spherical Text Embedding. In Proceedings of the 19th International Conference on Natural Language Processing (ICON), pages 80–85, New Delhi, India. Association for Computational Linguistics.
- Cite (Informal):
- Generalised Spherical Text Embedding (Banerjee et al., ICON 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.icon-main.11.pdf