@inproceedings{wang-etal-2025-rethinking-stateful,
    title = "Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges",
    author = "Wang, Hongru  and
      Huang, Wenyu  and
      Wang, Yufei  and
      Xi, Yuanhao  and
      Lu, Jianqiao  and
      Zhang, Huan  and
      Hu, Nan  and
      Liu, Zeming  and
      Pan, Jeff Z.  and
      Wong, Kam-Fai",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.findings-acl.284/",
    doi = "10.18653/v1/2025.findings-acl.284",
    pages = "5433--5453",
    ISBN = "979-8-89176-256-5",
    abstract = "Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose DialogTool, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) \textit{tool creation}; 2) \textit{tool utilization}: tool awareness, tool selection, tool execution; and 3) \textit{role-consistent response}: response generation and role play. Furthermore, we build VirtualMobile {--} an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs. Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct open- and closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons ."
}