@inproceedings{son-etal-2025-kmmlu,
    title = "{KMMLU}: Measuring Massive Multitask Language Understanding in {K}orean",
    author = "Son, Guijin  and
      Lee, Hanwool  and
      Kim, Sungdong  and
      Kim, Seungone  and
      Muennighoff, Niklas  and
      Choi, Taekyoon  and
      Park, Cheonbok  and
      Yoo, Kang Min  and
      Biderman, Stella",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.206/",
    pages = "4076--4104",
    ISBN = "979-8-89176-189-6",
    abstract = "We propose KMMLU, a Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. While prior Korean evaluation tools heavily rely on translated versions of existing English benchmarks, KMMLU is collected from original Korean exams, thereby capturing linguistic and cultural aspects of the Korean language. Recent models struggle to show performance over 60{\%}, significantly below the pass mark of the source exams (80{\%}), highlighting the room for improvement. Notably, one-fifth of the questions in KMMLU require knowledge of Korean culture for accurate resolution. KMMLU thus provides a more accurate reflection of human preferences compared to translated versions of MMLU and offers deeper insights into LLMs' shortcomings in Korean knowledge. The dataset and codes are made publicly available for future research."
}