@inproceedings{sokol-etal-2026-benchnavigator,
    title = "{B}ench{N}avigator: A Discovery Interface for Comparing {LLM} Benchmarks",
    author = "Sokol, Anna  and
      Vejsbjerg, Inge  and
      Daly, Elizabeth M.  and
      Piorkowski, David  and
      Hind, Michael  and
      Moniz, Nuno  and
      Chawla, Nitesh V.",
    editor = "Akhtar, Mubashara  and
      Batzner, Jan  and
      Choshen, Leshem  and
      Ghosh, Avijit  and
      Gohar, Usman  and
      Mickel, Jennifer  and
      Pant, Ichhya  and
      Talat, Zeerak  and
      Lin, Michelle",
    booktitle = "Proceedings of the Workshop on Evaluating Evaluations ({E}val{E}val)",
    month = jul,
    year = "2026",
    address = "San Diego, CA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.evaleval-1.29/",
    pages = "174--200",
    ISBN = "979-8-89176-429-3",
    abstract = "Evaluating large language models (LLMs) requires selecting benchmarks that fit the intended use case. However, the rapid growth of benchmarks has made discovery and comparison difficult, because practitioners must assemble information across papers, repositories, and dataset cards with heterogeneous metadata, inconsistent terminology, and uneven documentation. Prior work improves individual benchmark documentation and quality assessment, but does not provide a uniform way to compare benchmarks during discovery. We survey practitioners, analyze multi-source benchmark metadata, and identify the fields needed for effective benchmark discovery. We introduce BenchNavigator, a prototype that organizes heterogeneous metadata into a coherent, provenance-preserving interface aligned with practitioner priorities. Our results show that benchmark metadata can be presented in a comparable form without imposing new reporting burdens on benchmark producers. We frame this contribution as discovery infrastructure, not as a method for scoring benchmark quality or replacing contextual evaluation."
}