Multi-Task Training with In-Domain Language Models for Diagnostic Reasoning
Brihat Sharma, Yanjun Gao, Timothy Miller, Matthew Churpek, Majid Afshar, Dmitriy Dligach
Abstract
Generative artificial intelligence (AI) is a promising direction for augmenting clinical diagnostic decision support and reducing diagnostic errors, a leading contributor to medical errors. To further the development of clinical AI systems, the Diagnostic Reasoning Benchmark (DR.BENCH) was introduced as a comprehensive generative AI framework, comprised of six tasks representing key components in clinical reasoning. We present a comparative analysis of in-domain versus out-of-domain language models as well as multi-task versus single task training with a focus on the problem summarization task in DR.BENCH. We demonstrate that a multi-task, clinically-trained language model outperforms its general domain counterpart by a large margin, establishing a new state-of-the-art performance, with a ROUGE-L score of 28.55. This research underscores the value of domain-specific training for optimizing clinical diagnostic reasoning tasks.- Anthology ID:
- 2023.clinicalnlp-1.10
- Volume:
- Proceedings of the 5th Clinical Natural Language Processing Workshop
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Venue:
- ClinicalNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 78–85
- Language:
- URL:
- https://aclanthology.org/2023.clinicalnlp-1.10
- DOI:
- Cite (ACL):
- Brihat Sharma, Yanjun Gao, Timothy Miller, Matthew Churpek, Majid Afshar, and Dmitriy Dligach. 2023. Multi-Task Training with In-Domain Language Models for Diagnostic Reasoning. In Proceedings of the 5th Clinical Natural Language Processing Workshop, pages 78–85, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Multi-Task Training with In-Domain Language Models for Diagnostic Reasoning (Sharma et al., ClinicalNLP 2023)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2023.clinicalnlp-1.10.pdf