@inproceedings{xi-etal-2026-agentgym2,
    title = "{A}gent{G}ym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments",
    author = "Xi, Zhiheng  and
      Yang, Dingwen  and
      Liu, Jiaqi  and
      Huang, Jixuan  and
      Guo, Honglin  and
      Huang, Baodai  and
      Chen, Tinggang  and
      Zhang, Qi  and
      Lu, Zhonghang  and
      Liu, Chenyu  and
      Sun, Jiajun  and
      Zhang, Jiazheng  and
      Zhu, Dingwei  and
      Guo, Xin  and
      Wang, Junzhe  and
      Zhang, Zhihao  and
      Yang, Yuming  and
      Ye, Junjie  and
      Gao, Minghe  and
      Liu, Dongrui  and
      Ji, Jiaming  and
      Li, Guohao  and
      Gui, Tao  and
      Zhang, Qi  and
      Huang, Xuanjing",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.2058/",
    pages = "44451--44479",
    ISBN = "979-8-89176-390-6",
    abstract = "Language agents, i.e., LLM agents, progress rapidly and are increasingly deployed in production environments. This trend underscores the urgent need for rigorous and realistic evaluations. However, most existing benchmarks evaluate agents in simplified, idealized settings. They typically rely on pre-packaged tool interfaces, overlook critical steps, and assume inputs are clean and fully specified. Consequently, they understate the difficulty of real deployments, where uncertainty and noise are ubiquitous and agents must proactively explore the environment to uncover new tools. To bridge this gap, we present AgentGym2, a new evaluation framework with task instances grounded in real-world end-to-end working demands. Beyond reasoning and planning, it measures agents' ability to execute end-to-end procedures, discover tools via exploration, compose tools for unseen tasks, and remain robust to noisy and underspecified information. Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2, revealing a substantial gap between the capability of current agents and the demands of real-world applications."
}