@inproceedings{li-etal-2025-generation,
    title = "From Generation to Judgment: Opportunities and Challenges of {LLM}-as-a-judge",
    author = "Li, Dawei  and
      Jiang, Bohan  and
      Huang, Liangjie  and
      Beigi, Alimohammad  and
      Zhao, Chengshuai  and
      Tan, Zhen  and
      Bhattacharjee, Amrita  and
      Jiang, Yuxuan  and
      Chen, Canyu  and
      Wu, Tianhao  and
      Shu, Kai  and
      Cheng, Lu  and
      Liu, Huan",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.138/",
    pages = "2757--2791",
    ISBN = "979-8-89176-332-6",
    abstract = "Assessment and evaluation have long been critical challenges in artificial intelligence (AI) and natural language processing (NLP). Traditional methods, usually matching-based or small model-based, often fall short in open-ended and dynamic scenarios. Recent advancements in Large Language Models (LLMs) inspire the ``LLM-as-a-judge'' paradigm, where LLMs are leveraged to perform scoring, ranking, or selection for various machine learning evaluation scenarios. This paper presents a comprehensive survey of LLM-based judgment and assessment, offering an in-depth overview to review this evolving field. We first provide the definition from both input and output perspectives. Then we introduce a systematic taxonomy to explore LLM-as-a-judge along three dimensions: \textit{what} to judge, \textit{how} to judge, and \textit{how} to benchmark. Finally, we also highlight key challenges and promising future directions for this emerging area."
}