Haojian Huang
2025
DependEval: Benchmarking LLMs for Repository Dependency Understanding
Junjia Du
|
Yadi Liu
|
Hongcheng Guo
|
Jiawei Wang
|
Haojian Huang
|
Yunyi Ni
|
Zhoujun Li
Findings of the Association for Computational Linguistics: ACL 2025
While large language models (LLMs) have shown considerable promise in code generation, real-world software development demands advanced repository-level reasoning. This includes understanding dependencies, project structures, and managing multi-file changes. However, the ability of LLMs to effectively comprehend and handle complex code repositories has yet to be fully explored. To address these challenges, we introduce a hierarchical benchmark designed to evaluate repository dependency understanding(DependEval) for LLMs. The benchmark is based on 2683 repositories collected from real-world websites. It evaluates models on three core tasks: Dependency Recognition, Repository Construction, and Multi-file Editing, across 8 programming languages from actual code repositories. Our evaluation of over 25 LLMs reveals substantial performance gaps and provides valuable insights into repository-level code understanding.
Search
Fix author
Co-authors
- Junjia Du 1
- Hongcheng Guo 1
- Zhoujun Li 1
- Yadi Liu 1
- Yunyi Ni 1
- show all...