SummQA at MEDIQA-Chat 2023: In-Context Learning with GPT-4 for Medical Summarization

Yash Mathur, Sanketh Rangreji, Raghav Kapoor, Medha Palavalli, Amanda Bertsch, Matthew Gormley

Abstract
Medical dialogue summarization is challenging due to the unstructured nature of medical conversations, the use of medical terminologyin gold summaries, and the need to identify key information across multiple symptom sets. We present a novel system for the Dialogue2Note Medical Summarization tasks in the MEDIQA 2023 Shared Task. Our approach for sectionwise summarization (Task A) is a two-stage process of selecting semantically similar dialogues and using the top-k similar dialogues as in-context examples for GPT-4. For full-note summarization (Task B), we use a similar solution with k=1. We achieved 3rd place in Task A (2nd among all teams), 4th place in Task B Division Wise Summarization (2nd among all teams), 15th place in Task A Section Header Classification (9th among all teams), and 8th place among all teams in Task B. Our results highlight the effectiveness of few-shot prompting for this task, though we also identify several weaknesses of prompting-based approaches. We compare GPT-4 performance with several finetuned baselines. We find that GPT-4 summaries are more abstractive and shorter. We make our code publicly available.
Anthology ID:
2023.clinicalnlp-1.51
Volume:
Proceedings of the 5th Clinical Natural Language Processing Workshop
Month:
July
Year:
2023
Address:
Toronto, Canada
Venue:
ClinicalNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
490–502
Language:
URL:
https://aclanthology.org/2023.clinicalnlp-1.51
DOI:
Bibkey:
Cite (ACL):
Yash Mathur, Sanketh Rangreji, Raghav Kapoor, Medha Palavalli, Amanda Bertsch, and Matthew Gormley. 2023. SummQA at MEDIQA-Chat 2023: In-Context Learning with GPT-4 for Medical Summarization. In Proceedings of the 5th Clinical Natural Language Processing Workshop, pages 490–502, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
SummQA at MEDIQA-Chat 2023: In-Context Learning with GPT-4 for Medical Summarization (Mathur et al., ClinicalNLP 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/paclic-22-ingestion/2023.clinicalnlp-1.51.pdf
PDF Search