Weihan Peng
2026
SWE-QA: Can Language Models Answer Repository-level Code Questions?
Weihan Peng | Yuling Shi | Yuhang Wang | Xinyun Zhang | Beijun Shen | Xiaodong Gu
Findings of the Association for Computational Linguistics: ACL 2026
Weihan Peng | Yuling Shi | Yuhang Wang | Xinyun Zhang | Beijun Shen | Xiaodong Gu
Findings of the Association for Computational Linguistics: ACL 2026
Understanding and reasoning about entire soft-ware repositories is an essential capability for intelligent software engineering tools. While existing benchmarks such as CoSQA and CodeQA have advanced the field, they predominantly focus on small, self-contained code snippets. These setups fail to capture the complexity of real-world repositories, where effective understanding and reasoning often require navigating multiple files, understanding software architecture, and grounding answers in long-range code dependencies. In this paper, we present SWE-QA, a repository-level code question answering (QA) benchmark designed to facilitate research on automated QA systems in realistic code environments. SWE-QA involves 720 high-quality question-answer pairs spanning diverse categories, including intention understanding, cross-file reasoning, and multi-hop dependency analysis. To construct SWE-QA, we first crawled 77,100 GitHub issues from 12 popular repositories. Based on an analysis of naturally occurring developer questions extracted from these issues, we developed a two-level taxonomy of repository-level questions and constructed a set of seed questions for each category. For each category, we manually curated and validated questions and collected their corresponding answers. We evaluate six advanced LLMs on SWE-QA under various context augmentation strategies. Experimental results highlight the promise of LLMs.