@inproceedings{edwards-etal-2026-rexbench,
    title = "{RE}x{B}ench: Can coding agents autonomously implement {AI} research extensions?",
    author = "Edwards, Nicholas  and
      Lee, Yukyung  and
      Mao, Yujun Audrey  and
      Qin, Yulu  and
      Schuster, Sebastian  and
      Kim, Najoung",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.745/",
    pages = "16380--16417",
    ISBN = "979-8-89176-390-6",
    abstract = "Agents based on Large Language Models (LLMs) have shown promise for performing sophisticated software engineering tasks autonomously. In addition, there has been progress towards developing agents that can perform parts of the research pipeline in machine learning and the natural sciences. We argue that research extension and its implementation is a critical capability for such systems, and introduce RExBench to support the evaluation of this capability. RExBench is a benchmark consisting of realistic extensions of 12 research papers that aim to investigate \textit{novel} research hypotheses. Each task is set up as an extension to an existing research paper and codebase, accompanied by domain expert-written instructions. RExBench is robust to data contamination, and supports an automatic evaluation infrastructure that executes agent outputs to determine whether the success criteria are met. We use this benchmark to evaluate 12 LLM agents implemented using two different frameworks: aider and OpenHands. We find that all agents fail to autonomously implement the majority of the extensions, with the best agent at around 33{\%} success rate. Although the success rate improves with additional human-written hints, the best performance under this setting remains below 44{\%}. This indicates that current agents are still short of being able to handle realistic research extension tasks without substantial human guidance."
}