@inproceedings{li-etal-2025-llm-judge,
    title = "{LLM}-as-a-Judge Failures at Automating the Identification of Poor Quality Outputs in Free-Form Texts",
    author = "Li, Zongxia  and
      Wu, Xiyang  and
      Mondal, Ishani  and
      Siu, Alexa  and
      Boyd-Graber, Jordan Lee  and
      Nenkova, Ani",
    editor = "Dong, Yue  and
      Xiao, Wen  and
      Zhang, Haopeng  and
      Zhang, Rui  and
      Ernst, Ori  and
      Wang, Lu  and
      Liu, Fei",
    booktitle = "Proceedings of The 5th New Frontiers in Summarization Workshop",
    month = nov,
    year = "2025",
    address = "Hybrid",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.newsum-main.1/",
    pages = "1--16",
    ISBN = "979-8-89176-337-1",
    abstract = "Large language models (LLMs) such as GPT-4, Claude and LLaMA are routinely used to evaluate long-form text generated by language models. We study the ability of these models to identify low quality texts, an increasingly rare subset of output which is of great interest to pinpoint during development. We present experiments with a panel of LLM judges, and crowd-sourced approximations of reference judgments. Pinpointing sub-par outputs is a difficult task for both models and crowdworkers, with models doing overall better. Moreover, unlike findings in prior work on factoid question answering, panels of cheaper models do not agree as well with high quality developer judgments of low quality as panels of frontier models. We present qualitative and quantitative analysis of the relative strengths of models in the panel, gleaning insights why they yield better results over a single model."
}