Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints
Aryo Gema, Chaeeun Lee, Pasquale Minervini, Luke Daines, T. Simpson, Beatrice Alex
Abstract
The MEDIQA-CORR 2024 shared task aims to assess the ability of Large Language Models (LLMs) to identify and correct medical errors in clinical notes. In this study, we evaluate the capability of general LLMs, specifically GPT-3.5 and GPT-4, to identify and correct medical errors with multiple prompting strategies. Recognising the limitation of LLMs in generating accurate corrections only via prompting strategies, we propose incorporating error-span predictions from a smaller, fine-tuned model in two ways: 1) by presenting it as a hint in the prompt and 2) by framing it as multiple-choice questions from which the LLM can choose the best correction. We found that our proposed prompting strategies significantly improve the LLM’s ability to generate corrections. Our best-performing solution with 8-shot + CoT + hints ranked sixth in the shared task leaderboard. Additionally, our comprehensive analyses show the impact of the location of the error sentence, the prompted role, and the position of the multiple-choice option on the accuracy of the LLM. This prompts further questions about the readiness of LLM to be implemented in real-world clinical settings.- Anthology ID:
- 2024.clinicalnlp-1.49
- Volume:
- Proceedings of the 6th Clinical Natural Language Processing Workshop
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Tristan Naumann, Asma Ben Abacha, Steven Bethard, Kirk Roberts, Danielle Bitterman
- Venues:
- ClinicalNLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 488–501
- Language:
- URL:
- https://aclanthology.org/2024.clinicalnlp-1.49
- DOI:
- Cite (ACL):
- Aryo Gema, Chaeeun Lee, Pasquale Minervini, Luke Daines, T. Simpson, and Beatrice Alex. 2024. Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 488–501, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints (Gema et al., ClinicalNLP-WS 2024)
- PDF:
- https://preview.aclanthology.org/jeptaln-2024-ingestion/2024.clinicalnlp-1.49.pdf