@inproceedings{kulkarni-etal-2025-massive,
    title = "{MASSIVE}-Agents: A Benchmark for Multilingual Function-Calling in 52 Languages",
    author = "Kulkarni, Mayank  and
      Mazzia, Vittorio  and
      Gaspers, Judith  and
      Hench, Chris  and
      FitzGerald, Jack",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/name-variant-enfa-fane/2025.findings-emnlp.1099/",
    doi = "10.18653/v1/2025.findings-emnlp.1099",
    pages = "20193--20215",
    ISBN = "979-8-89176-335-7",
    abstract = "We present MASSIVE-Agents, a new benchmark for assessing multilingual function calling across 52 languages. We created MASSIVE-Agents by cleaning the original MASSIVE dataset and then reformatting it for evaluation within the Berkeley Function-Calling Leaderboard (BFCL) framework. The full benchmark comprises 47,020 samples with an average of 904 samples per language, covering 55 different functions and 286 arguments. We benchmarked 21 models using Amazon Bedrock and present the results along with associated analyses. MASSIVE-Agents is challenging, with the top model Nova Premier achieving an average Abstract Syntax Tree (AST) Accuracy of 34.05{\%} across all languages, with performance varying significantly from 57.37{\%} for English to as low as 6.81{\%} for Amharic. Some models, particularly smaller ones, yielded a score of zero for the more difficult languages. Additionally, we provide results from ablations using a custom 1-shot prompt, ablations with prompts translated into different languages, and comparisons based on model latency."
}