@inproceedings{olufemi-etal-2025-challenging,
    title = "Challenging Multimodal {LLM}s with {A}frican Standardized Exams: A Document {VQA} Evaluation",
    author = "Olufemi, Victor Tolulope  and
      Babatunde, Oreoluwa Boluwatife  and
      Bolarinwa, Emmanuel  and
      Moshood, Kausar Yetunde",
    editor = "Lignos, Constantine  and
      Abdulmumin, Idris  and
      Adelani, David",
    booktitle = "Proceedings of the Sixth Workshop on African Natural Language Processing (AfricaNLP 2025)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.africanlp-1.22/",
    doi = "10.18653/v1/2025.africanlp-1.22",
    pages = "150--157",
    ISBN = "979-8-89176-257-2",
    abstract = "Despite rapid advancements in multimodal large language models (MLLMs), their ability to process low-resource African languages in document-based visual question answering (VQA) tasks remains limited. This paper evaluates three state-of-the-art MLLMs{---}GPT-4o, Claude-3.5 Haiku, and Gemini-1.5 Pro{---}on WAEC/NECO standardized exam questions in Yoruba, Igbo, and Hausa. We curate a dataset of multiple-choice questions from exam images and compare model accuracies across two prompting strategies: (1) using English prompts for African language questions, and (2) using native-language prompts. While GPT-4o achieves over 90{\%} accuracy for English, performance drops below 40{\%} for African languages, highlighting severe data imbalance in model training. Notably, native-language prompting improves accuracy for most models, yet no system approaches human-level performance, which reaches over 50{\%} in Yoruba, Igbo, and Hausa. These findings emphasize the need for diverse training data, fine-tuning, and dedicated benchmarks that address the linguistic intricacies of African languages in multimodal tasks, paving the way for more equitable and effective AI systems in education."
}