@inproceedings{kazemi-etal-2025-big,
    title = "{BIG}-Bench Extra Hard",
    author = "Kazemi, Mehran  and
      Fatemi, Bahare  and
      Bansal, Hritik  and
      Palowitch, John  and
      Anastasiou, Chrysovalantis  and
      Mehta, Sanket Vaibhav  and
      Jain, Lalit K  and
      Aglietti, Virginia  and
      Jindal, Disha  and
      Chen, Peter  and
      Dikkala, Nishanth  and
      Tyen, Gladys  and
      Liu, Xin  and
      Shalit, Uri  and
      Chiappa, Silvia  and
      Olszewska, Kate  and
      Tay, Yi  and
      Tran, Vinh Q.  and
      Le, Quoc V  and
      Firat, Orhan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1285/",
    pages = "26473--26501",
    ISBN = "979-8-89176-251-0",
    abstract = "Current benchmarks for large language model (LLM) reasoning predominantly focus on mathematical and coding abilities, leaving a gap in evaluating broader reasoning proficiencies. One particular exception is the BIG-Bench dataset, which has served as a crucial benchmark for evaluating the general reasoning capabilities of LLMs, thanks to its diverse set of challenging tasks that allowed for a comprehensive assessment of general reasoning across various skills within a unified framework. However, recent advances in LLMs have led to saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH). State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus diminishing its utility. To address this limitation, we introduce BIG-Bench Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM reasoning evaluation. BBEH replaces each task in BBH with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty. We evaluate various general-purpose and reasoning-specialized models on BBEH and observe an accuracy of 23.9{\%} for the best general-purpose model and 54.2{\%} for the best reasoning-specialized model, indicating substantial room for improvement and highlighting the ongoing challenge of achieving robust general reasoning in LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh."
}