@inproceedings{zhang-etal-2026-greekmmlu,
    title = "{G}reek{MMLU}: A Native-Sourced Multitask Benchmark for Evaluating Language Models in {G}reek",
    author = "Zhang, Yang  and
      Konomi, Mersin  and
      Xypolopoulos, Christos  and
      Divriotis, Konstantinos  and
      Skianis, Konstantinos  and
      Nikolentzos, Giannis  and
      Stamou, Giorgos  and
      Shang, Guokan  and
      Vazirgiannis, Michalis",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.448/",
    pages = "9193--9217",
    ISBN = "979-8-89176-395-1",
    abstract = "Large Language Models (LLMs) are commonly trained on multilingual corpora that include Greek, yet reliable evaluation benchmarks for Greek{---}particularly those based on authentic, native-sourced content{---}remain limited. Existing datasets are often machine-translated from English, failing to capture Greek linguistic and cultural characteristics. We introduce GreekMMLU, a native-sourced benchmark for massive multitask language understanding in Greek, comprising 21,805 multiple-choice questions across 45 subject areas, organized under a newly defined subject taxonomy and annotated with educational difficulty levels spanning primary to professional examinations. All questions are sourced or authored in Greek from academic, professional, and governmental exams. We publicly release 16,857 samples and reserve 4,948 samples for a private leaderboard to enable robust and contamination-resistant evaluation. Evaluations of over 80 open- and closed-source LLMs reveal substantial performance gaps between frontier and open-weight models, as well as between Greek-adapted models and general multilingual ones. Finally, we provide a systematic analysis of factors influencing performance{---}including model scale, adaptation, and prompting{---}and derive insights for improving LLM capabilities in Greek."
}