@inproceedings{ma-etal-2025-judging,
    title = "Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based {LLM}-as-Judge",
    author = "Ma, Chiyu  and
      Zhang, Enpei  and
      Zhao, Yilun  and
      Liu, Wenjun  and
      Jia, Yaning  and
      Qing, Peijun  and
      Shi, Lin  and
      Cohan, Arman  and
      Yan, Yujun  and
      Vosoughi, Soroush",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.941/",
    doi = "10.18653/v1/2025.findings-emnlp.941",
    pages = "17356--17392",
    ISBN = "979-8-89176-335-7",
    abstract = "LLM-as-Judge has emerged as a scalable alternative to human evaluation, enabling large language models (LLMs) to provide reward signals in trainings. While recent work has explored multi-agent extensions such as multi-agent debate and meta-judging to enhance evaluation quality, the question of how intrinsic biases manifest in these settings remains underexplored. In this study, we conduct a systematic analysis of four diverse bias types: position bias, verbosity bias, chain-of-thought bias, and bandwagon bias. We evaluate these biases across two widely adopted multi-agent LLM-as-Judge frameworks: \textit{Multi-Agent-Debate} and \textit{LLM-as-Meta-Judge}. \textbf{Our results show that debate framework amplifies biases sharply after the initial debate, and this increased bias is sustained in subsequent rounds, while meta-judge approaches exhibit greater resistance.} We further investigate the incorporation of PINE, a leading single-agent debiasing method, as a bias-free agent within these systems. \textbf{The results reveal that this bias-free agent effectively reduces biases in debate settings but provides less benefit in meta-judge scenarios.} Our work provides a comprehensive study of bias behavior in multi-agent LLM-as-Judge systems and highlights the need for targeted bias mitigation strategies in collaborative evaluation settings."
}