Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints
Omid Rohanian, Hannah Jauncey, Mohammadmahdi Nouriborji, Vinod Kumar, Bronner P. Gonalves, Christiana Kartsonaki, Isaric Clinical Characterisation Group, Laura Merson, David Clifton
Abstract
Processing information locked within clinical health records is a challenging task that remains an active area of research in biomedical NLP. In this work, we evaluate a broad set of machine learning techniques ranging from simple RNNs to specialised transformers such as BioBERT on a dataset containing clinical notes along with a set of annotations indicating whether a sample is cancer-related or not. Furthermore, we specifically employ efficient fine-tuning methods from NLP, namely, bottleneck adapters and prompt tuning, to adapt the models to our specialised task. Our evaluations suggest that fine-tuning a frozen BERT model pre-trained on natural language and with bottleneck adapters outperforms all other strategies, including full fine-tuning of the specialised BioBERT model. Based on our findings, we suggest that using bottleneck adapters in low-resource situations with limited access to labelled data or processing capacity could be a viable strategy in biomedical text mining.- Anthology ID:
- 2023.bionlp-1.5
- Volume:
- The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Dina Demner-fushman, Sophia Ananiadou, Kevin Cohen
- Venue:
- BioNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 62–78
- Language:
- URL:
- https://aclanthology.org/2023.bionlp-1.5
- DOI:
- 10.18653/v1/2023.bionlp-1.5
- Cite (ACL):
- Omid Rohanian, Hannah Jauncey, Mohammadmahdi Nouriborji, Vinod Kumar, Bronner P. Gonalves, Christiana Kartsonaki, Isaric Clinical Characterisation Group, Laura Merson, and David Clifton. 2023. Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 62–78, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints (Rohanian et al., BioNLP 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2023.bionlp-1.5.pdf