@inproceedings{huang-etal-2025-empirical,
    title = "An Empirical Study of {LLM}-as-a-Judge for {LLM} Evaluation: Fine-tuned Judge Model is not a General Substitute for {GPT}-4",
    author = "Huang, Hui  and
      Bu, Xingyuan  and
      Zhou, Hongli  and
      Qu, Yingqi  and
      Liu, Jing  and
      Yang, Muyun  and
      Xu, Bing  and
      Zhao, Tiejun",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.306/",
    pages = "5880--5895",
    ISBN = "979-8-89176-256-5",
    abstract = "Recently, there has been a growing trend of utilizing Large Language Model (LLM) to evaluate the quality of other LLMs. Many studies have fine-tuned judge models based on open-source LLMs for evaluation. While the fine-tuned judge models are claimed to achieve comparable evaluation capability with GPT-4, in this work, we conduct an empirical study of LLM-as-a-Judge. Our findings indicate that although the fine-tuned judge models achieve high performance on in-domain test sets, even surpassing GPT-4, they underperform GPT-4 across several dimensions, including generalizability, fairness and adaptability. We also reveal that the fine-tuned judge model inherently operates as a task-specific classifier, consequently imposing the limitations."
}