Guergana K Savova
2025
Overview of the 2025 Shared Task on Chemotherapy Treatment Timeline Extraction
Jiarui Yao
|
Harry Hochheiser
|
WonJin Yoon
|
Eli T Goldner
|
Guergana K Savova
Proceedings of the 7th Clinical Natural Language Processing Workshop
Extracting patient treatment timelines from clinical notes is a complex task involving identification of relevant events, temporal expressions, and temporal relations in individual documents and developing cross-document summaries. The 2025 Shared Task on Chemotherapy Treatment Timeline Extraction builds upon the initial 2024 challenge, using data from 57,530 breast and ovarian cancer patients and 15,946 melanoma patients. Participants were provided with a subset annotated for treatment entities, temporal expressions, temporal relations, and timelines for each patient. This training data was used to addressed two subtasks. Subtask 1 focused on extracting temporal relations and creating timelines, given documents and gold-standard events and temporal expressions. Sutask 2 involved development of an end-to-end system involving extraction of entities, temporal expressions, and relations, and construction of timelines, given only the Electronic Health Record notes. Five teams participated, submitting eight entries for Subtask 1 and twelve for Subtask 2. Supervised fine-tuning remains a productive approach albeit with a shift of supervised fine-tuning of very large language models compared to the 2024 task edition. Even with the much more “strict” evaluation metric, the best results are comparable to the best less strict 2024 relaxed-to-month results.
WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation
João Matos
|
Shan Chen
|
Siena Kathleen V. Placino
|
Yingya Li
|
Juan Carlos Climent Pardo
|
Daphna Idan
|
Takeshi Tohyama
|
David Restrepo
|
Luis Filipe Nakayama
|
José María Millet Pascual-Leone
|
Guergana K Savova
|
Hugo Aerts
|
Leo Anthony Celi
|
An-Kwok Ian Wong
|
Danielle Bitterman
|
Jack Gallifant
Findings of the Association for Computational Linguistics: NAACL 2025
Multimodal/vision language models (VLMs) are increasingly being deployed in healthcare settings worldwide, necessitating robust benchmarks to ensure their safety, efficacy, and fairness. Multiple-choice question and answer (QA) datasets derived from national medical examinations have long served as valuable evaluation tools, but existing datasets are largely text-only and available in a limited subset of languages and countries. To address these challenges, we present WorldMedQA-V, an updated multilingual, multimodal benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V includes 568 labeled multiple-choice QAs paired with 568 medical images from four countries (Brazil, Israel, Japan, and Spain), covering original languages and validated English translations by native clinicians, respectively. Baseline performance for common open- and closed-source models are provided in the local language and English translations, and with and without images provided to the model. The WorldMedQA-V benchmark aims to better match AI systems to the diverse healthcare environments in which they are deployed, fostering more equitable, effective, and representative applications.
Search
Fix author
Co-authors
- Hugo Aerts 1
- Danielle Bitterman 1
- Leo Anthony Celi 1
- Shan Chen 1
- Jack Gallifant 1
- show all...