BioELECTRA:Pretrained Biomedical text Encoder using Discriminators

Kamal raj Kanakarajan, Bhuvana Kundumani, Malaikannan Sankarasubbu

Abstract
Recent advancements in pretraining strategies in NLP have shown a significant improvement in the performance of models on various text mining tasks. We apply ‘replaced token detection’ pretraining technique proposed by ELECTRA and pretrain a biomedical language model from scratch using biomedical text and vocabulary. We introduce BioELECTRA, a biomedical domain-specific language encoder model that adapts ELECTRA for the Biomedical domain. WE evaluate our model on the BLURB and BLUE biomedical NLP benchmarks. BioELECTRA outperforms the previous models and achieves state of the art (SOTA) on all the 13 datasets in BLURB benchmark and on all the 4 Clinical datasets from BLUE Benchmark across 7 different NLP tasks. BioELECTRA pretrained on PubMed and PMC full text articles performs very well on Clinical datasets as well. BioELECTRA achieves new SOTA 86.34%(1.39% accuracy improvement) on MedNLI and 64% (2.98% accuracy improvement) on PubMedQA dataset.
Anthology ID:
2021.bionlp-1.16
Volume:
Proceedings of the 20th Workshop on Biomedical Language Processing
Month:
June
Year:
2021
Address:
Online
Editors:
Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, Junichi Tsujii
Venue:
BioNLP
SIG:
SIGBIOMED
Publisher:
Association for Computational Linguistics
Note:
Pages:
143–154
Language:
URL:
https://aclanthology.org/2021.bionlp-1.16
DOI:
10.18653/v1/2021.bionlp-1.16
Bibkey:
Cite (ACL):
Kamal raj Kanakarajan, Bhuvana Kundumani, and Malaikannan Sankarasubbu. 2021. BioELECTRA:Pretrained Biomedical text Encoder using Discriminators. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 143–154, Online. Association for Computational Linguistics.
Cite (Informal):
BioELECTRA:Pretrained Biomedical text Encoder using Discriminators (Kanakarajan et al., BioNLP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2021.bionlp-1.16.pdf
Data
BC5CDR, BIOSSES, BLUE, BLURB, BioASQ, ChemProt, DDI, HOC, MedNLI, NCBI Disease, PubMed PICO Element Detection Dataset, PubMedQA, SQuAD
PDF Search