@inproceedings{lu-etal-2025-toolsandbox,
    title = "{T}ool{S}andbox: A Stateful, Conversational, Interactive Evaluation Benchmark for {LLM} Tool Use Capabilities",
    author = "Lu, Jiarui  and
      Holleis, Thomas  and
      Zhang, Yizhe  and
      Aumayer, Bernhard  and
      Nan, Feng  and
      Bai, Haoping  and
      Ma, Shuang  and
      Ma, Shen  and
      Li, Mengyu  and
      Yin, Guoli  and
      Wang, Zirui  and
      Pang, Ruoming",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.65/",
    pages = "1160--1183",
    ISBN = "979-8-89176-195-7",
    abstract = "Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over arbitrary trajectory. We show that open source and proprietary models has a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights to tool-use LLM capabilities. Datasets and evaluation scripts of ToolSandbox are released at {\ensuremath{<}}placeholder{\ensuremath{>}}."
}