@inproceedings{li-etal-2025-rag,
    title = "{RAG}-Zeval: Enhancing {RAG} Responses Evaluator through End-to-End Reasoning and Ranking-Based Reinforcement Learning",
    author = "Li, Kun  and
      Li, Yunxiang  and
      Zhang, Tianhua  and
      Luo, Hongyin  and
      Wu, Xixin  and
      Glass, James R.  and
      Meng, Helen M.",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1267/",
    pages = "24936--24954",
    ISBN = "979-8-89176-332-6",
    abstract = "Robust evaluation is critical for deploying trustworthy retrieval-augmented generation (RAG) systems. However, current LLM-based evaluation frameworks predominantly rely on directly prompting resource-intensive models with complex multi-stage prompts, underutilizing models' reasoning capabilities and introducing significant computational cost. In this paper, we present RAG-Zeval (RAG-Zero Evaluator), a novel end-to-end framework that formulates faithfulness and correctness evaluation of RAG systems as a rule-guided reasoning task. Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments with detailed explanation in one-pass. We introduce a ranking-based outcome reward mechanism, using preference judgments rather than absolute scores, to address the challenge of obtaining precise pointwise reward signals. To this end, we synthesize the ranking references by generating quality-controlled responses with zero human annotation. Experiments demonstrate RAG-Zeval{'}s superior performance, achieving the strongest correlation with human judgments and outperforming baselines that rely on LLMs with $10-100\times$ more parameters. Our approach also exhibits superior interpretability in response evaluation."
}