@inproceedings{matos-etal-2025-worldmedqa,
    title = "{W}orld{M}ed{QA}-{V}: a multilingual, multimodal medical examination dataset for multimodal language models evaluation",
    author = "Matos, Jo{\~a}o  and
      Chen, Shan  and
      Placino, Siena Kathleen V.  and
      Li, Yingya  and
      Pardo, Juan Carlos Climent  and
      Idan, Daphna  and
      Tohyama, Takeshi  and
      Restrepo, David  and
      Nakayama, Luis Filipe  and
      Pascual-Leone, Jos{\'e} Mar{\'i}a Millet  and
      Savova, Guergana K  and
      Aerts, Hugo  and
      Celi, Leo Anthony  and
      Wong, An-Kwok Ian  and
      Bitterman, Danielle  and
      Gallifant, Jack",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.402/",
    pages = "7203--7216",
    ISBN = "979-8-89176-195-7",
    abstract = "Multimodal/vision language models (VLMs) are increasingly being deployed in healthcare settings worldwide, necessitating robust benchmarks to ensure their safety, efficacy, and fairness. Multiple-choice question and answer (QA) datasets derived from national medical examinations have long served as valuable evaluation tools, but existing datasets are largely text-only and available in a limited subset of languages and countries. To address these challenges, we present WorldMedQA-V, an updated multilingual, multimodal benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V includes 568 labeled multiple-choice QAs paired with 568 medical images from four countries (Brazil, Israel, Japan, and Spain), covering original languages and validated English translations by native clinicians, respectively. Baseline performance for common open- and closed-source models are provided in the local language and English translations, and with and without images provided to the model. The WorldMedQA-V benchmark aims to better match AI systems to the diverse healthcare environments in which they are deployed, fostering more equitable, effective, and representative applications."
}