@inproceedings{nimo-etal-2025-afrimed,
    title = "{A}fri{M}ed-{QA}: A Pan-{A}frican, Multi-Specialty, Medical Question-Answering Benchmark Dataset",
    author = {Nimo, Charles  and
      Olatunji, Tobi  and
      Owodunni, Abraham Toluwase  and
      Abdullahi, Tassallah  and
      Ayodele, Emmanuel  and
      Sanni, Mardhiyah  and
      Aka, Ezinwanne C.  and
      Omofoye, Folafunmi  and
      Yuehgoh, Foutse  and
      Faniran, Timothy  and
      Dossou, Bonaventure F. P.  and
      Yekini, Moshood O.  and
      Kemp, Jonas  and
      Heller, Katherine A  and
      Omeke, Jude Chidubem  and
      Md, Chidi Asuzu  and
      Etori, Naome A  and
      Ndiaye, A{\"i}m{\'e}rou  and
      Okoh, Ifeoma  and
      Ocansey, Evans Doe  and
      Kinara, Wendy  and
      Best, Michael L.  and
      Essa, Irfan  and
      Moore, Stephen Edward  and
      Fourie, Chris  and
      Asiedu, Mercy Nyamewaa},
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.96/",
    pages = "1948--1973",
    ISBN = "979-8-89176-251-0",
    abstract = "Recent advancements in large language model (LLM) performance on medical multiplechoice question (MCQ) benchmarks have stimulated interest from healthcare providers and patients globally. Particularly in low-andmiddle-income countries (LMICs) facing acute physician shortages and lack of specialists, LLMs offer a potentially scalable pathway to enhance healthcare access and reduce costs. However, their effectiveness in the Global South, especially across the African continent, remains to be established. In this work, we introduce AfriMed-QA , the first largescale Pan-African English multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open and closed-ended) sourced from over 60 medical schools across 16 countries, covering 32 medical specialties. We further evaluate 30 LLMs across multiple axes including correctness and demographic bias. Our findings show significant performance variation across specialties and geographies, MCQ performance clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general models and smaller edge-friendly LLMs struggle to achieve a passing score. Interestingly, human evaluations show a consistent consumer preference for LLM answers and explanations when compared with clinician answers."
}