Prompt Discriminative Language Models for Domain Adaptation
Keming Lu, Peter Potash, Xihui Lin, Yuwen Sun, Zihan Qian, Zheng Yuan, Tristan Naumann, Tianxi Cai, Junwei Lu
Abstract
Prompt tuning offers an efficient approach to domain adaptation for pretrained language models, which predominantly focus on masked language modeling or generative objectives. However, the potential of discriminative language models in biomedical tasks remains underexplored.To bridge this gap, we develop BioDLM, a method tailored for biomedical domain adaptation of discriminative language models that incorporates prompt-based continual pretraining and prompt tuning for downstream tasks. BioDLM aims to maximize the potential of discriminative language models in low-resource scenarios by reformulating these tasks as span-level corruption detection, thereby enhancing performance on domain-specific tasks and improving the efficiency of continual pertaining. In this way, BioDLM provides a data-efficient domain adaptation method for discriminative language models, effectively enhancing performance on discriminative tasks within the biomedical domain.- Anthology ID:
- 2023.clinicalnlp-1.30
- Volume:
- Proceedings of the 5th Clinical Natural Language Processing Workshop
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Tristan Naumann, Asma Ben Abacha, Steven Bethard, Kirk Roberts, Anna Rumshisky
- Venue:
- ClinicalNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 247–258
- Language:
- URL:
- https://aclanthology.org/2023.clinicalnlp-1.30
- DOI:
- 10.18653/v1/2023.clinicalnlp-1.30
- Cite (ACL):
- Keming Lu, Peter Potash, Xihui Lin, Yuwen Sun, Zihan Qian, Zheng Yuan, Tristan Naumann, Tianxi Cai, and Junwei Lu. 2023. Prompt Discriminative Language Models for Domain Adaptation. In Proceedings of the 5th Clinical Natural Language Processing Workshop, pages 247–258, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Prompt Discriminative Language Models for Domain Adaptation (Lu et al., ClinicalNLP 2023)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2023.clinicalnlp-1.30.pdf