@inproceedings{togmanov-etal-2025-kazmmlu,
    title = "{K}az{MMLU}: Evaluating Language Models on {K}azakh, {R}ussian, and Regional Knowledge of {K}azakhstan",
    author = "Togmanov, Mukhammed  and
      Mukhituly, Nurdaulet  and
      Turmakhan, Diana  and
      Mansurov, Jonibek  and
      Goloburda, Maiya  and
      Sakip, Akhmed  and
      Xie, Zhuohan  and
      Wang, Yuxia  and
      Syzdykov, Bekassyl  and
      Laiyk, Nurkhan  and
      Aji, Alham Fikri  and
      Kochmar, Ekaterina  and
      Nakov, Preslav  and
      Koto, Fajri",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.701/",
    pages = "14403--14416",
    ISBN = "979-8-89176-251-0",
    abstract = "Despite having a population of twenty million, Kazakhstan{'}s culture and language remain underrepresented in the field of natural language processing. Although large language models (LLMs) continue to advance worldwide, progress in Kazakh language has been limited, as seen in the scarcity of dedicated models and benchmark evaluations. To address this gap, we introduce KazMMLU, the first MMLU-style dataset specifically designed for Kazakh language. KazMMLU comprises 23,000 questions that cover various educational levels, including STEM, humanities, and social sciences, sourced from authentic educational materials and manually validated by native speakers and educators. The dataset includes 10,969 Kazakh questions and 12,031 Russian questions, reflecting Kazakhstan{'}s bilingual education system and rich local context. Our evaluation of several state-of-the-art multilingual models (Llama3.1, Qwen-2.5, GPT-4, and DeepSeek V3) demonstrates substantial room for improvement, as even the best-performing models struggle to achieve competitive performance in Kazakh and Russian. These findings highlight significant performance gaps compared to high-resource languages. We hope that our dataset will enable further research and development of Kazakh-centric LLMs."
}