@inproceedings{du-etal-2025-dependeval,
    title = "{D}epend{E}val: Benchmarking {LLM}s for Repository Dependency Understanding",
    author = "Du, Junjia  and
      Liu, Yadi  and
      Guo, Hongcheng  and
      Wang, Jiawei  and
      Huang, Haojian  and
      Ni, Yunyi  and
      Li, Zhoujun",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.findings-acl.373/",
    pages = "7150--7179",
    ISBN = "979-8-89176-256-5",
    abstract = "While large language models (LLMs) have shown considerable promise in code generation, real-world software development demands advanced repository-level reasoning. This includes understanding dependencies, project structures, and managing multi-file changes. However, the ability of LLMs to effectively comprehend and handle complex code repositories has yet to be fully explored. To address these challenges, we introduce a hierarchical benchmark designed to evaluate repository dependency understanding(DependEval) for LLMs. The benchmark is based on 2683 repositories collected from real-world websites. It evaluates models on three core tasks: Dependency Recognition, Repository Construction, and Multi-file Editing, across 8 programming languages from actual code repositories. Our evaluation of over 25 LLMs reveals substantial performance gaps and provides valuable insights into repository-level code understanding."
}