@inproceedings{gema-etal-2025-done,
    title = "Are We Done with {MMLU}?",
    author = "Gema, Aryo Pradipta  and
      Leang, Joshua Ong Jun  and
      Hong, Giwon  and
      Devoto, Alessio  and
      Mancino, Alberto Carlo Maria  and
      Saxena, Rohit  and
      He, Xuanli  and
      Zhao, Yu  and
      Du, Xiaotang  and
      Ghasemi Madani, Mohammad Reza  and
      Barale, Claire  and
      McHardy, Robert  and
      Harris, Joshua  and
      Kaddour, Jean  and
      Van Krieken, Emile  and
      Minervini, Pasquale",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.262/",
    pages = "5069--5096",
    ISBN = "979-8-89176-189-6",
    abstract = "Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57{\%} of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU{'}s error-ridden questions to enhance its future utility and reliability as a benchmark. Therefore, we open up MMLU-Redux for additional annotation."
}