@inproceedings{zhu-etal-2025-multiagentbench,
    title = "{M}ulti{A}gent{B}ench : Evaluating the Collaboration and Competition of {LLM} agents",
    author = "Zhu, Kunlun  and
      Du, Hongyi  and
      Hong, Zhaochen  and
      Yang, Xiaocheng  and
      Guo, Shuyi  and
      Wang, Zhe  and
      Wang, Zhenhailong  and
      Qian, Cheng  and
      Tang, Robert  and
      Ji, Heng  and
      You, Jiaxuan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.421/",
    pages = "8580--8622",
    ISBN = "979-8-89176-251-0",
    abstract = "Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents; yet existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. In this paper, we introduce MultiAgentBench, a comprehensive benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. Our framework measures not only task completion but also the quality of collaboration and competition using novel, milestone-based key performance indicators. Moreover, we evaluate various coordination protocols (including star, chain, tree, and graph topologies) and innovative strategies such as group discussion and cognitive planning. Notably, cognitive planning improves milestone achievement rates by 3{\%}. Code and dataset will be made publicly available. Code and datasets are publicavailable at https://github.com/ulab-uiuc/MARBLE"
}