@inproceedings{pan-etal-2025-benchmarks,
    title = "When Benchmarks Talk: Re-Evaluating Code {LLM}s with Interactive Feedback",
    author = "Pan, Jane  and
      Shar, Ryan  and
      Pfau, Jacob  and
      Talwalkar, Ameet  and
      He, He  and
      Chen, Valerie",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.1267/",
    pages = "24672--24700",
    ISBN = "979-8-89176-256-5",
    abstract = "Programming with a coding assistant is a fundamentally interactive process, yet existing static benchmarks fail to capture key features of model-user collaboration. We introduce an interactive evaluation pipeline to examine how LLMs incorporate different types of feedback in a collaborative setting, in which we obfuscate the input of static coding benchmarks so that the code model must interact with a simulated user. Across 10 models and 3 datasets, the relative rankings of models often permute greatly between static and interactive settings, despite models being fairly robust to feedback that contains errors. We also observe that similarly effective feedback types differ in terms of how models respond to higher- vs. lower-quality feedback. Moreover, feedback type impacts the degree to which the models make aesthetic or behavioral edits to their output. Our work aims to ``re-evaluate'' model coding capabilities through an interactive lens toward bridging the gap between existing evaluations and real-world usage."
}