@inproceedings{xuan-etal-2025-mmlu,
    title = "{MMLU}-{P}ro{X}: A Multilingual Benchmark for Advanced Large Language Model Evaluation",
    author = "Xuan, Weihao  and
      Yang, Rui  and
      Qi, Heli  and
      Zeng, Qingcheng  and
      Xiao, Yunze  and
      Feng, Aosong  and
      Liu, Dairui  and
      Xing, Yun  and
      Wang, Junjue  and
      Gao, Fan  and
      Lu, Jinghui  and
      Jiang, Yuang  and
      Li, Huitao  and
      Li, Xin  and
      Yu, Kunyu  and
      Dong, Ruihai  and
      Gu, Shangding  and
      Li, Yuekang  and
      Xie, Xiaofei  and
      Juefei-Xu, Felix  and
      Khomh, Foutse  and
      Yoshie, Osamu  and
      Chen, Qingyu  and
      Teodoro, Douglas  and
      Liu, Nan  and
      Goebel, Randy  and
      Ma, Lei  and
      Marrese-Taylor, Edison  and
      Lu, Shijian  and
      Iwasawa, Yusuke  and
      Matsuo, Yutaka  and
      Li, Irene",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.79/",
    pages = "1513--1532",
    ISBN = "979-8-89176-332-6",
    abstract = "Existing large language model (LLM) evaluation benchmarks primarily focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-lingual reasoning abilities. This dual limitation makes it challenging to assess LLMs' performance in the multilingual setting comprehensively. To fill this gap, we introduce MMLU-ProX, a comprehensive benchmark covering 29 languages, built on an English benchmark. Each language version consists of 11,829 identical questions, enabling direct cross-lingual comparisons. Additionally, to meet efficient evaluation needs, we provide a lite version containing 658 questions per language. To ensure the high quality of MMLU-ProX, we employ a rigorous development process that involves multiple powerful LLMs for translation, followed by expert review to ensure accurate expression, consistent terminology, and cultural relevance. Building on this, we systematically evaluate 36 state-of-the-art LLMs, including reasoning-enhanced and multilingual-optimized LLMs. The results reveal significant disparities in the multilingual capabilities of LLMs: While they perform well in high-resource languages, their performance declines markedly in low-resource languages, particularly for African languages. Through MMLU-ProX, we aim to advance the development of more inclusive AI systems and promote equitable access to technology across global contexts."
}