@inproceedings{you-etal-2026-agenticragtracer,
    title = "{A}gentic{RAGT}racer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic {RAG}",
    author = "You, Qijie  and
      Yu, Wenkai  and
      Liang, Hao  and
      Wong, Zhen Hao  and
      Zhang, Wentao",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.66/",
    pages = "1301--1335",
    ISBN = "979-8-89176-395-1",
    abstract = "With the rapid advancement of agent-based methods in recent years, Agentic RAG has undoubtedly become an important research direction. Multi-hop reasoning, which requires models to engage in deliberate thinking and multi-step interaction, serves as a critical testbed for assessing such capabilities. However, existing benchmarks typically provide only final questions and answers, while lacking the intermediate hop-level questions that gradually connect atomic questions to the final multi-hop query. This limitation prevents researchers from analyzing at which step an agent fails and restricts more fine-grained evaluation of model capabilities. Moreover, most current benchmarks are manually constructed, which is both time-consuming and labor-intensive, while also limiting scalability and generalization. To address these challenges, we introduce AgenticRAGTracer, the first Agentic RAG benchmark that is primarily constructed automatically by large language models and designed to support step-by-step validation. Our benchmark spans multiple domains, contains 1,305 data points, and has no overlap with existing mainstream benchmarks. Extensive experiments demonstrate that even the best large language models perform poorly on our dataset. For instance, GPT-5 attains merely 22.6{\%} EM accuracy on the hardest portion of our dataset. Hop-aware diagnosis reveals that failures are primarily driven by distorted reasoning chains{---}either collapsing prematurely or wandering into over-extension. This highlights a critical inability to allocate steps consistent with the task{'}s logical structure, providing a diagnostic dimension missing in traditional evaluations. Our code and data are available at https://github.com/YqjMartin/AgenticRAGTracer."
}