BioSimCSE: BioMedical Sentence Embeddings using Contrastive learning
Kamal raj Kanakarajan, Bhuvana Kundumani, Abhijith Abraham, Malaikannan Sankarasubbu
Abstract
Sentence embeddings in the form of fixed-size vectors that capture the information in the sentence as well as the context are critical components of Natural Language Processing systems. With transformer model based sentence encoders outperforming the other sentence embedding methods in the general domain, we explore the transformer based architectures to generate dense sentence embeddings in the biomedical domain. In this work, we present BioSimCSE, where we train sentence embeddings with domain specific transformer based models with biomedical texts. We assess our model’s performance with zero-shot and fine-tuned settings on Semantic Textual Similarity (STS) and Recognizing Question Entailment (RQE) tasks. Our BioSimCSE model using BioLinkBERT achieves state of the art (SOTA) performance on both tasks.- Anthology ID:
- 2022.louhi-1.10
- Volume:
- Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Editors:
- Alberto Lavelli, Eben Holderness, Antonio Jimeno Yepes, Anne-Lyse Minard, James Pustejovsky, Fabio Rinaldi
- Venue:
- Louhi
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 81–86
- Language:
- URL:
- https://aclanthology.org/2022.louhi-1.10
- DOI:
- 10.18653/v1/2022.louhi-1.10
- Cite (ACL):
- Kamal raj Kanakarajan, Bhuvana Kundumani, Abhijith Abraham, and Malaikannan Sankarasubbu. 2022. BioSimCSE: BioMedical Sentence Embeddings using Contrastive learning. In Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), pages 81–86, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- BioSimCSE: BioMedical Sentence Embeddings using Contrastive learning (Kanakarajan et al., Louhi 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2022.louhi-1.10.pdf