@inproceedings{bayes-etal-2026-uhura,
    title = "Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource {A}frican Languages",
    author = "Bayes, Edward Thomas  and
      Azime, Israel Abebe  and
      Alabi, Jesujoba  and
      Kgomo, Jonas  and
      Eloundou, Tyna  and
      Proehl, Elizabeth  and
      Chen, Kai  and
      Khadir, Imaan  and
      Etori, Naome A.  and
      Muhammad, Shamsuddeen Hassan  and
      Mpanza, Choice  and
      Thete, Igneciah Pocia IP  and
      Klakow, Dietrich  and
      Adelani, David Ifeoluwa",
    editor = "Piperidis, Stelios  and
      Bel, N{\'u}ria  and
      van den Heuvel, Henk  and
      Ide, Nancy  and
      Krek, Simon  and
      Toral, Antonio",
    booktitle = "Proceedings of the Fifteenth Language Resources and Evaluation Conference",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resource Association",
    url = "https://preview.aclanthology.org/ingest-nlpsi/2026.lrec-1.115/",
    doi = "10.63317/43x6rqwpycuo",
    pages = "1485--1504",
    abstract = "Evaluations of Large Language Models (LLMs) on knowledge-intensive tasks and factual accuracy often focus on high-resource languages primarily because datasets for low-resource languages (LRLs) are scarce. In this paper, we present Uhura{---}a new benchmark that focuses on two tasks in six typologically-diverse African languages, created via human translation of existing English benchmarks. The first dataset, Uhura-ARC-Easy, is composed of multiple-choice science questions. The second, Uhura-TruthfulQA, is a safety benchmark testing the truthfulness of models on topics including health, law, finance, and politics. We highlight the challenges creating benchmarks with highly technical content for LRLs and outline mitigation strategies. Our evaluation reveals a significant performance gap between proprietary models such as GPT-4o and o1-preview, and Claude models, and open-source models like LLaMA and Gemma. Additionally, all models perform better in English than in African languages. These results indicate that LLMs struggle with answering scientific questions and are more prone to generating false claims in low-resource African languages. Our findings underscore the necessity for continuous improvement of multilingual LLM capabilities in LRL settings to ensure safe and reliable use in real-world contexts. We open-source the Uhura Benchmark and Uhura Platform to foster further research and development in NLP for LRLs."
}