Large Language Models Provide Human-Level Medical Text Snippet Labeling
Ibtihel Amara, Haiyang Yu, Fan Zhang, Yuchen Liu, Benny Li, Chang Liu, Rupesh Kartha, Akshay Goel
Abstract
This study evaluates the proficiency of Large Language Models (LLMs) in accurately labeling clinical document excerpts. Our focus is on the assignment of potential or confirmed diagnoses and medical procedures to snippets of medical text sourced from unstructured clinical patient records. We explore how the performance of LLMs compare against human annotators in classifying these excerpts. Employing a few-shot, chain-of-thought prompting approach with the MIMIC-III dataset, Med-PaLM 2 showcases annotation accuracy comparable to human annotators, achieving a notable precision rate of approximately 92% relative to the gold standard labels established by human experts.- Anthology ID:
- 2024.clinicalnlp-1.15
- Volume:
- Proceedings of the 6th Clinical Natural Language Processing Workshop
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Tristan Naumann, Asma Ben Abacha, Steven Bethard, Kirk Roberts, Danielle Bitterman
- Venues:
- ClinicalNLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 185–195
- Language:
- URL:
- https://aclanthology.org/2024.clinicalnlp-1.15
- DOI:
- Cite (ACL):
- Ibtihel Amara, Haiyang Yu, Fan Zhang, Yuchen Liu, Benny Li, Chang Liu, Rupesh Kartha, and Akshay Goel. 2024. Large Language Models Provide Human-Level Medical Text Snippet Labeling. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 185–195, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- Large Language Models Provide Human-Level Medical Text Snippet Labeling (Amara et al., ClinicalNLP-WS 2024)
- PDF:
- https://preview.aclanthology.org/jeptaln-2024-ingestion/2024.clinicalnlp-1.15.pdf