Abstract
In some memory-constrained settings like IoT devices and over-the-network data pipelines, it can be advantageous to have smaller contextual embeddings. We investigate the efficacy of projecting contextual embedding data (BERT) onto a manifold, and using nonlinear dimensionality reduction techniques to compress these embeddings. In particular, we propose a novel post-processing approach, applying a combination of Isomap and PCA. We find that the geodesic distance estimations, estimates of the shortest path on a Riemannian manifold, from Isomap’s k-Nearest Neighbors graph bolstered the performance of the compressed embeddings to be comparable to the original BERT embeddings. On one dataset, we find that despite a 12-fold dimensionality reduction, the compressed embeddings performed within 0.1% of the original BERT embeddings on a downstream classification task. In addition, we find that this approach works particularly well on tasks reliant on syntactic data, when compared with linear dimensionality reduction. These results show promise for a novel geometric approach to achieve lower dimensional text embeddings from existing transformers and pave the way for data-specific and application-specific embedding compressions.- Anthology ID:
- 2021.textgraphs-1.15
- Volume:
- Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15)
- Month:
- June
- Year:
- 2021
- Address:
- Mexico City, Mexico
- Venue:
- TextGraphs
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 144–149
- Language:
- URL:
- https://aclanthology.org/2021.textgraphs-1.15
- DOI:
- 10.18653/v1/2021.textgraphs-1.15
- Cite (ACL):
- Rishi Jha and Kai Mihata. 2021. On Geodesic Distances and Contextual Embedding Compression for Text Classification. In Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15), pages 144–149, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- On Geodesic Distances and Contextual Embedding Compression for Text Classification (Jha & Mihata, TextGraphs 2021)
- PDF:
- https://preview.aclanthology.org/starsem-semeval-split/2021.textgraphs-1.15.pdf
- Code
- kaimihata/geo-bert
- Data
- CoLA, GLUE, SST