@inproceedings{ding-etal-2026-octobench,
    title = "{O}cto{B}ench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding",
    author = "Ding, Deming  and
      Liu, Shichun  and
      Yang, Enhui  and
      Lin, Jiahang  and
      Chen, Ziying  and
      Dou, Shihan  and
      Guo, Honglin  and
      Cheng, Weiyu  and
      Zhao, Pengyu  and
      Xiao, Chengjun  and
      Zeng, Qunhong  and
      Zhang, Qi  and
      Huang, Xuanjing  and
      Xu, Qidi  and
      Gui, Tao",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.269/",
    pages = "5958--5978",
    ISBN = "979-8-89176-390-6",
    abstract = "Modern coding scaffolds turn LLMs into capable software agents, but their ability to follow scaffold-specified instructions remains under-examined, especially when constraints are heterogeneous and persist across interactions. To fill this gap, we introduce OctoBench, which benchmarks scaffold-aware instruction following in repository-grounded agentic coding. OctoBench includes 34 environments and 217 tasks instantiated under three scaffold types, and is paired with 7,098 objective checklist items. To disentangle solving the task from following the rules, we provide an automated observation-and-scoring toolkit that captures full trajectories and performs fine-grained checks. Experiments on eight representative models reveal a systematic gap between task-solving and scaffold-aware compliance, underscoring the need for training and evaluation that explicitly targets heterogeneous instruction following. We will release the benchmark to support reproducible benchmarking and to accelerate the development of more scaffold-aware coding agents."
}