@inproceedings{isbarov-etal-2025-tumlu,
    title = "{TUMLU}: A Unified and Native Language Understanding Benchmark for {T}urkic Languages",
    author = {Isbarov, Jafar  and
      Akhundjanova, Arofat  and
      Hajili, Mammad  and
      Huseynova, Kavsar  and
      Gaynullin, Dmitry  and
      Rzayev, Anar  and
      Tursun, Osman  and
      Turdubaeva, Aizirek  and
      Saetov, Ilshat  and
      Kharisov, Rinat  and
      Belginova, Saule  and
      Kenbayeva, Ariana  and
      Alisheva, Amina  and
      K{\"o}ksal, Abdullatif  and
      Rustamov, Samir  and
      Ataman, Duygu},
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1112/",
    pages = "22816--22838",
    ISBN = "979-8-89176-251-0",
    abstract = "Being able to thoroughly assess massive multi-task language understanding (MMLU) capabilities is essential for advancing the applicability of multilingual language models. However, preparing such benchmarks in high quality native language is often costly and therefore limits the representativeness of evaluation datasets. While recent efforts focused on building more inclusive MMLU benchmarks, these are conventionally built using machine translation from high-resource languages, which may introduce errors and fail to account for the linguistic and cultural intricacies of the target languages. In this paper, we address the lack of native language MMLU benchmark especially in the under-represented Turkic language family with distinct morphosyntactic and cultural characteristics. We propose two benchmarks for Turkic language MMLU: TUMLU is a comprehensive, multilingual, and natively developed language understanding benchmark specifically designed for Turkic languages. It consists of middle- and high-school level questions spanning 11 academic subjects in Azerbaijani, Crimean Tatar, Karakalpak, Kazakh, Kyrgyz, Tatar, Turkish, Uyghur, and Uzbek. We also present TUMLU-mini, a more concise, balanced, and manually verified subset of the dataset. Using this dataset, we systematically evaluate a diverse range of open and proprietary multilingual large language models (LLMs), including Claude, Gemini, GPT, and LLaMA, offering an in-depth analysis of their performance across different languages, subjects, and alphabets. To promote further research and development in multilingual language understanding, we release TUMLU-mini and all corresponding evaluation scripts."
}