@inproceedings{zhang-etal-2025-di,
    title = "{DI}-{BENCH}: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale",
    author = "Zhang, Linghao  and
      Wang, Junhao  and
      He, Shilin  and
      Zhang, Chaoyun  and
      Kang, Yu  and
      Li, Bowen  and
      Wen, Jiaheng  and
      Xie, Chengxing  and
      Wang, Maoquan  and
      Huang, Yufan  and
      Nallipogu, Elsie  and
      Lin, Qingwei  and
      Dang, Yingnong  and
      Rajmohan, Saravan  and
      Zhang, Dongmei  and
      Zhang, Qi",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.528/",
    pages = "10134--10153",
    ISBN = "979-8-89176-256-5",
    abstract = "Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40{\%} of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs' capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C{\#}, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 48{\%} execution pass rate on Python, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis."
}