@inproceedings{elder-etal-2026-live,
    title = "Live {API}-Bench: 2500+ Live {API}s for Testing Multi-Step Tool Calling",
    author = "Elder, Benjamin  and
      Murthi, Anupama  and
      Kang, Jungkoo  and
      Naik, Ankita  and
      Basu, Kinjal  and
      Kate, Kiran  and
      Contractor, Danish",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.143/",
    pages = "3092--3124",
    ISBN = "979-8-89176-380-7",
    abstract = "Large language models (LLMs) increasingly rely on external tools and APIs to execute complex tasks specified in natural language. Evaluating such \textit{}tool-calling capabilities in realistic enterprise settings is challenging: APIs are often proprietary, heterogeneous, and difficult to share, limiting reproducible benchmarks. To address this, we introduce \textbf{}Live API Bench, a comprehensive benchmark constructed by transforming NL2SQL datasets into interactive API environments. Our pipeline converts SQL queries from BIRD-SQL into executable API sequences across three formulations{---}\textit{}SLOT, \textit{}SEL, and \textit{}REST{---}covering minimal general-purpose operations, domain-specific multi-step tasks, and function-oriented RESTful interactions, respectively. The benchmark spans 11 databases with over 2,500 invocable tools, paired with human-authored queries, ground-truth API sequences, and verified final answers. Live API Bench enables systematic evaluation of core challenges in tool use, including error handling, sequential reasoning, parameter generation, response parsing, and robustness across diverse domains. We evaluate 10 LLMs and 4 ReACT agents, observing low task completion rates (7{--}47{\%}), which improve modestly to 50{\%} under interactive agent settings, highlighting substantial scope for improving LLM tool-calling performance. We release all code and data associated with this paper."
}