@inproceedings{zhu-etal-2026-raffles,
    title = "{RAFFLES}: Reasoning-based Attribution of Faults for {LLM} Systems",
    author = "Zhu, Chenyang  and
      Hong, Spencer  and
      Wu, Jingyu  and
      Chawla, Kushal  and
      Tang, Yuhui  and
      Yin, Youbing  and
      Wolfe, Nathan  and
      Babinsky, Erin  and
      Liu, Daben",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.359/",
    pages = "7659--7688",
    ISBN = "979-8-89176-380-7",
    abstract = "The advent of complex, interconnected long-horizon LLM systems has made it incredibly tricky to identify where and when these systems break down. Evaluation capabilities that currently exist today are limited in that they often focus on simple metrics, end-to-end outcomes, and are dependent on the perspectives of humans. In order to match the increasing complexity of these many component systems, evaluation frameworks must also be able to reason, probe, iterate, and understand the nuanced logic passing through these systems. In this paper, we present RAFFLES, an offline evaluation architecture that incorporates iterative reasoning. Specifically, RAFFLES operates as an iterative, multi-component pipeline, using a central Judge to systematically identify faults and a set of specialized Evaluators to assess the quality of the candidate faults as well as rationales of the Judge. We evaluated RAFFLES with several benchmarks - the Who{\&}When dataset to identify step-level faults in multi-agent systems and the ReasonEval datasets to diagnose step-level mathematical reasoning errors. RAFFLES outperforms strong baselines, achieving an accuracy of over 20{\%} and 50{\%} on the Who{\&}When Hand-Crafted and Algorithmically-Generated datasets, and over 80{\%} on the ReasonEval datasets. These results demonstrate a key step towards introducing automated fault detection for autonomous systems over labor-intensive manual review."
}