@article{katsis-etal-2025-mt,
    title = "mt {RAG}: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems",
    author = "Katsis, Yannis  and
      Rosenthal, Sara  and
      Fadnis, Kshitij  and
      Gunasekara, Chulaka  and
      Lee, Young-Suk  and
      Popa, Lucian  and
      Shah, Vraj  and
      Zhu, Huaiyu  and
      Contractor, Danish  and
      Danilevsky, Marina",
    journal = "Transactions of the Association for Computational Linguistics",
    volume = "13",
    year = "2025",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://preview.aclanthology.org/ingest-eacl/2025.tacl-1.36/",
    doi = "10.1162/tacl.a.19",
    pages = "784--808",
    abstract = "Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation, is an important and often overlooked task with several additional challenges. We present mtRAG, an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. mtRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on mtRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. mtRAG is available at https://github.com/ibm/mt-rag-benchmark."
}