@inproceedings{kim-etal-2025-biggen,
    title = "The {B}i{GG}en Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models",
    author = "Kim, Seungone  and
      Suk, Juyoung  and
      Cho, Ji Yong  and
      Longpre, Shayne  and
      Kim, Chaeeun  and
      Yoon, Dongkeun  and
      Son, Guijin  and
      Cho, Yejin  and
      Shafayat, Sheikh  and
      Baek, Jinheon  and
      Park, Sue Hyun  and
      Hwang, Hyeonbin  and
      Jo, Jinkyung  and
      Cho, Hyowon  and
      Shin, Haebin  and
      Lee, Seongyun  and
      Oh, Hanseok  and
      Lee, Noah  and
      Ho, Namgyu  and
      Joo, Se June  and
      Ko, Miyoung  and
      Lee, Yoonjoo  and
      Chae, Hyungjoo  and
      Shin, Jamin  and
      Jang, Joel  and
      Ye, Seonghyeon  and
      Lin, Bill Yuchen  and
      Welleck, Sean  and
      Neubig, Graham  and
      Lee, Moontae  and
      Lee, Kyungjae  and
      Seo, Minjoon",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-06/2025.naacl-long.303/",
    doi = "10.18653/v1/2025.naacl-long.303",
    pages = "5877--5919",
    ISBN = "979-8-89176-189-6",
    abstract = "As language models (LMs) become capable of handling a wide range of tasks, their evaluation is becoming as challenging as their development. Most generation benchmarks currently assess LMs using abstract evaluation criteria-like helpfulness and harmlessness-which often lack the flexibility and granularity of human assessment. Additionally, these benchmarks tend to focus disproportionately on specific capabilities such as instruction following, leading to coverage bias. To overcome these limitations, we introduce the BiGGen Bench, a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation. We apply this benchmark to assess 100 frontier LMs using five evaluator LMs. Our code, data, and evaluation results are all publicly available at https://github.com/prometheus-eval/prometheus-eval."
}