@inproceedings{gevers-etal-2025-benchmarks,
    title = "In Benchmarks We Trust ... Or Not?",
    author = "Gevers, Ine  and
      De Marez, Victor  and
      Van Nooten, Jens  and
      Lemmens, Jens  and
      Kosar, Andriy  and
      Lotfi, Ehsan  and
      Banar, Nikolay  and
      Fivez, Pieter  and
      De Bruyne, Luna  and
      Daelemans, Walter",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1208/",
    pages = "23673--23687",
    ISBN = "979-8-89176-332-6",
    abstract = "Standardized benchmarks are central to evaluating and comparing model performance in Natural Language Processing (NLP). However, Large Language Models (LLMs) have exposed shortcomings in existing benchmarks, and so far there is no clear solution. In this paper, we survey a wide scope of benchmarking issues, and provide an overview of solutions as they are suggested in the literature. We observe that these solutions often tackle a limited number of issues, neglecting other facets. Therefore, we propose concrete checklists to cover all aspects of benchmarking issues, both for benchmark creation and usage. We illustrate the use of our checklists by applying them to three popular NLP benchmarks (i.e., SuperGLUE, WinoGrande, and ARC-AGI). Additionally, we discuss the potential advantages of adding minimal-sized test-suites to benchmarking, which would ensure downstream applicability on real-world use cases."
}