Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting

Marco Naguib, Xavier Tannier, Aurélie Névéol

Abstract
Large language models (LLMs) have become the preferred solution for many natural language processing tasks. In low-resource environments such as specialized domains, their few-shot capabilities are expected to deliver high performance. Named Entity Recognition (NER) is a critical task in information extraction that is not covered in recent LLM benchmarks. There is a need for better understanding the performance of LLMs for NER in a variety of settings including languages other than English. This study aims to evaluate generative LLMs, employed through prompt engineering, for few-shot clinical NER. We compare 13 auto-regressive models using prompting and 16 masked models using fine-tuning on 14 NER datasets covering English, French and Spanish. While prompt-based auto-regressive models achieve competitive F1 for general NER, they are outperformed within the clinical domain by lighter biLSTM-CRF taggers based on masked models. Additionally, masked models exhibit lower environmental impact compared to auto-regressive models. Findings are consistent across the three languages studied, which suggests that LLM prompting is not yet suited for NER production in the clinical domain.
Anthology ID:
2024.findings-emnlp.400
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2024
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6829–6852
Language:
URL:
https://preview.aclanthology.org/remove-affiliations/2024.findings-emnlp.400/
DOI:
10.18653/v1/2024.findings-emnlp.400
Bibkey:
Cite (ACL):
Marco Naguib, Xavier Tannier, and Aurélie Névéol. 2024. Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6829–6852, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting (Naguib et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/remove-affiliations/2024.findings-emnlp.400.pdf
PDF Search Fix data