@inproceedings{alkhouli-etal-2025-confetti,
    title = "{CONFETTI}: Conversational Function-Calling Evaluation Through Turn-Level Interactions",
    author = "Alkhouli, Tamer  and
      Margatina, Katerina  and
      Gung, James  and
      Shu, Raphael  and
      Zaghi, Claudia  and
      Sunkara, Monica  and
      Zhang, Yi",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.394/",
    pages = "7993--8006",
    ISBN = "979-8-89176-251-0",
    abstract = "We introduce Conversational Function-Calling Evaluation Through Turn-Level Interactions (CONFETTI), a conversational benchmark designed to evaluate the function-calling capabilities and response quality of large language models (LLMs). Current benchmarks lack comprehensive assessment of LLMs in complex conversational scenarios. CONFETTI addresses this gap through 109 human-simulated conversations, comprising 313 user turns and covering 86 APIs. These conversations explicitly target various conversational complexities, such as follow-ups, goal correction and switching, ambiguous and implicit goals. We perform off-policy turn-level evaluation using this benchmark targeting function-calling. Our benchmark also incorporates dialog act annotations to assess agent responses. We evaluate a series of state-of-the-art LLMs and analyze their performance with respect to the number of available APIs, conversation lengths, and chained function calling. Our results reveal that while some models are able to handle long conversations, and leverage more than 20+ APIs successfully, other models struggle with longer context or when increasing the number of APIs. We also report that the performance on chained function-calls is severely limited across the models. Overall, the top performing models onCONFETTI are Nova Pro (40.01{\%}), Claude Sonnet v3.5 (35.46{\%}) and Llama 3.1 405B (33.19{\%}) followed by command-r-plus (31.18{\%}) and Mistral-Large-2407 (30.07{\%})."
}