@inproceedings{ruan-etal-2026-mtr,
    title = "{MTR}-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks",
    author = "Ruan, Junhao  and
      Abudula, Abudukeyumu  and
      Li, Bei  and
      Yin, Yongjing  and
      Liu, Xinyu  and
      Jiao, Kechen  and
      Chen, Xin  and
      Wang, Jingang  and
      Cai, Xunliang  and
      Xiao, Tong  and
      Zhu, JingBo",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1726/",
    pages = "37223--37250",
    ISBN = "979-8-89176-390-6",
    abstract = "Accurate evaluation of conversational retrieval is pivotal for advancing Retrieval-Augmented Generation (RAG) systems. However, existing conversational retrieval benchmarks suffer from costly, sparse human annotation or rigid, unnatural automated heuristics. To address these challenges, we introduce MTR-Suite, a unified framework for auditing, synthesizing, and benchmarking retrieval. It features: (1) MTR-Eval, an LLM-based auditor quantifying alignment gaps in previous benchmarks; (2) MTR-Pipeline, a multi-agent system using greedy traversal clustering to generate high-fidelity dialogues at 1/400th human cost; and (3) MTR-Bench, a rigorous general-domain benchmark. MTR-Bench mimics production-style challenges (hard topic switching, verbosity), offering superior discriminative power. We make our code and data publicly available to facilitate future research."
}