@inproceedings{cai-etal-2026-swe,
    title = "{SWE}-{QA}-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding",
    author = "Cai, Songcheng  and
      Lyu, Zhiheng  and
      Ni, Yuansheng  and
      Chen, Xiangchao  and
      Zhou, Baichuan  and
      Zhu, Shenzhe  and
      Lu, Yi  and
      Wang, Haozhe  and
      Ruan, Chi  and
      Schneider, Benjamin  and
      Zhang, Weixu  and
      Li, Xiang  and
      Zheng, Andy  and
      Zhang, Yuyu  and
      Nie, Ping  and
      Chen, Wenhu",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.837/",
    pages = "16958--16999",
    ISBN = "979-8-89176-395-1",
    abstract = "Agentic repository-level code understanding is essential for automating complex software engineering tasks, yet the field lacks reliable benchmarks. Existing evaluations often overlook the long tail topics and rely on popular repositories where Large Language Models (LLMs) can cheat via memorized knowledge. To address this, we introduce SWE-QA-Pro, a benchmark constructed from diverse, long-tail repositories with executable environments. We enforce topical balance via issue-driven clustering to cover under-represented task types and apply a rigorous difficulty calibration process: questions solvable by direct-answer baselines are filtered out. This results in a dataset where agentic workflows significantly outperform direct answering (e.g., a {\textasciitilde}13-point gap for Claude Sonnet 4.5), confirming the necessity of agentic codebase exploration. Furthermore, to tackle the scarcity of training data for such complex behaviors, we propose a scalable synthetic data pipeline that powers a two-stage training recipe: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from AI Feedback (RLAIF). This approach allows small open models to learn efficient tool usage and reasoning. Empirically, a Qwen3-8B model trained with our recipe surpasses GPT-4o by 2.3 points on SWE-QA-Pro and substantially narrows the gap to state-of-the-art proprietary models, demonstrating both the validity of our evaluation and the effectiveness of our agentic training workflow."
}