Shangding Gu
2025
MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
Weihao Xuan
|
Rui Yang
|
Heli Qi
|
Qingcheng Zeng
|
Yunze Xiao
|
Aosong Feng
|
Dairui Liu
|
Yun Xing
|
Junjue Wang
|
Fan Gao
|
Jinghui Lu
|
Yuang Jiang
|
Huitao Li
|
Xin Li
|
Kunyu Yu
|
Ruihai Dong
|
Shangding Gu
|
Yuekang Li
|
Xiaofei Xie
|
Felix Juefei-Xu
|
Foutse Khomh
|
Osamu Yoshie
|
Qingyu Chen
|
Douglas Teodoro
|
Nan Liu
|
Randy Goebel
|
Lei Ma
|
Edison Marrese-Taylor
|
Shijian Lu
|
Yusuke Iwasawa
|
Yutaka Matsuo
|
Irene Li
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Existing large language model (LLM) evaluation benchmarks primarily focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-lingual reasoning abilities. This dual limitation makes it challenging to assess LLMs’ performance in the multilingual setting comprehensively. To fill this gap, we introduce MMLU-ProX, a comprehensive benchmark covering 29 languages, built on an English benchmark. Each language version consists of 11,829 identical questions, enabling direct cross-lingual comparisons. Additionally, to meet efficient evaluation needs, we provide a lite version containing 658 questions per language. To ensure the high quality of MMLU-ProX, we employ a rigorous development process that involves multiple powerful LLMs for translation, followed by expert review to ensure accurate expression, consistent terminology, and cultural relevance. Building on this, we systematically evaluate 36 state-of-the-art LLMs, including reasoning-enhanced and multilingual-optimized LLMs. The results reveal significant disparities in the multilingual capabilities of LLMs: While they perform well in high-resource languages, their performance declines markedly in low-resource languages, particularly for African languages. Through MMLU-ProX, we aim to advance the development of more inclusive AI systems and promote equitable access to technology across global contexts.
Search
Fix author
Co-authors
- Qingyu Chen 1
- Ruihai Dong 1
- Aosong Feng 1
- Fan Gao 1
- Randy Goebel 1
- show all...
- Yusuke Iwasawa 1
- Yuang Jiang 1
- Felix Juefei-Xu 1
- Foutse Khomh 1
- Huitao Li 1
- Xin Li 1
- Yuekang Li 1
- Irene Li 1
- Dairui Liu 1
- Nan Liu 1
- Jinghui Lu 1
- Shijian Lu 1
- Lei Ma 1
- Edison Marrese-Taylor 1
- Yutaka Matsuo 1
- Heli Qi 1
- Douglas Teodoro 1
- Junjue Wang 1
- Yunze Xiao 1
- Xiaofei Xie 1
- Yun Xing 1
- Weihao Xuan 1
- Rui Yang 1
- Osamu Yoshie 1
- Kunyu Yu 1
- Qingcheng Zeng 1