@inproceedings{zhu-etal-2025-rageval,
    title = "{RAGE}val: Scenario Specific {RAG} Evaluation Dataset Generation Framework",
    author = "Zhu, Kunlun  and
      Luo, Yifan  and
      Xu, Dingling  and
      Yan, Yukun  and
      Liu, Zhenghao  and
      Yu, Shi  and
      Wang, Ruobing  and
      Wang, Shuo  and
      Li, Yishan  and
      Zhang, Nan  and
      Han, Xu  and
      Liu, Zhiyuan  and
      Sun, Maosong",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.418/",
    pages = "8520--8544",
    ISBN = "979-8-89176-251-0",
    abstract = "Retrieval-Augmented Generation (RAG) is a powerful approach that enables large language models (LLMs) to incorporate external knowledge. However, evaluating the effectiveness of RAG systems in specialized scenarios remains challenging due to the high costs of data construction and the lack of suitable evaluation metrics. This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios by generating high-quality documents, questions, answers, and references through a schema-based pipeline. With a focus on factual accuracy, we propose three novel metrics{---}Completeness, Hallucination, and Irrelevance{---}to evaluate LLM-generated responses rigorously. Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. Furthermore, the use of LLMs for scoring the proposed metrics demonstrates a high level of consistency with human evaluations. RAGEval establishes a new paradigm for evaluating RAG systems in real-world applications. The code and dataset are released at https://github.com/OpenBMB/RAGEval."
}