@inproceedings{jha-mihata-2021-geodesic,
    title = "On Geodesic Distances and Contextual Embedding Compression for Text Classification",
    author = "Jha, Rishi  and
      Mihata, Kai",
    editor = "Panchenko, Alexander  and
      Malliaros, Fragkiskos D.  and
      Logacheva, Varvara  and
      Jana, Abhik  and
      Ustalov, Dmitry  and
      Jansen, Peter",
    booktitle = "Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15)",
    month = jun,
    year = "2021",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2021.textgraphs-1.15/",
    doi = "10.18653/v1/2021.textgraphs-1.15",
    pages = "144--149",
    abstract = "In some memory-constrained settings like IoT devices and over-the-network data pipelines, it can be advantageous to have smaller contextual embeddings. We investigate the efficacy of projecting contextual embedding data (BERT) onto a manifold, and using nonlinear dimensionality reduction techniques to compress these embeddings. In particular, we propose a novel post-processing approach, applying a combination of Isomap and PCA. We find that the geodesic distance estimations, estimates of the shortest path on a Riemannian manifold, from Isomap`s k-Nearest Neighbors graph bolstered the performance of the compressed embeddings to be comparable to the original BERT embeddings. On one dataset, we find that despite a 12-fold dimensionality reduction, the compressed embeddings performed within 0.1{\%} of the original BERT embeddings on a downstream classification task. In addition, we find that this approach works particularly well on tasks reliant on syntactic data, when compared with linear dimensionality reduction. These results show promise for a novel geometric approach to achieve lower dimensional text embeddings from existing transformers and pave the way for data-specific and application-specific embedding compressions."
}