@inproceedings{adelani-etal-2025-irokobench,
    title = "{I}roko{B}ench: A New Benchmark for {A}frican Languages in the Age of Large Language Models",
    author = "Adelani, David Ifeoluwa  and
      Ojo, Jessica  and
      Azime, Israel Abebe  and
      Zhuang, Jian Yun  and
      Alabi, Jesujoba Oluwadara  and
      He, Xuanli  and
      Ochieng, Millicent  and
      Hooker, Sara  and
      Bukula, Andiswa  and
      Lee, En-Shiun Annie  and
      Chukwuneke, Chiamaka Ijeoma  and
      Buzaaba, Happy  and
      Sibanda, Blessing Kudzaishe  and
      Kalipe, Godson Koffi  and
      Mukiibi, Jonathan  and
      Kabongo Kabenamualu, Salomon  and
      Yuehgoh, Foutse  and
      Setaka, Mmasibidi  and
      Ndolela, Lolwethu  and
      Odu, Nkiruka  and
      Mabuya, Rooweither  and
      Osei, Salomey  and
      Muhammad, Shamsuddeen Hassan  and
      Samb, Sokhar  and
      Guge, Tadesse Kebede  and
      Sherman, Tombekai Vangoni  and
      Stenetorp, Pontus",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.139/",
    pages = "2732--2757",
    ISBN = "979-8-89176-189-6",
    abstract = "Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (e.g. African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoBench{---}a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages covering three tasks: natural language inference(AfriXNLI), mathematical reasoning(AfriMGSM), and multi-choice knowledge-based QA(AfriMMLU). We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings(where test sets are translated into English) across 10 open and four proprietary LLMs. Our evaluation reveals a significant performance gap between high-resource languages (such as English and French) and low-resource African languages. We observe a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63{\%} of the best-performing proprietary model GPT-4o performance. Machine translating the test set to English before evaluation helped to close the gap for larger models that are English-centric, like Gemma 2 27B and LLaMa 3.1 70B. These findings suggest that more efforts are needed to develop and adapt LLMs for African languages."
}