@inproceedings{dou-etal-2025-simulatorarena,
    title = "{S}imulator{A}rena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of {AI} Assistants?",
    author = "Dou, Yao  and
      Galley, Michel  and
      Peng, Baolin  and
      Kedzie, Chris  and
      Cai, Weixin  and
      Ritter, Alan  and
      Quirk, Chris  and
      Xu, Wei  and
      Gao, Jianfeng",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1786/",
    pages = "35200--35278",
    ISBN = "979-8-89176-332-6",
    abstract = "Large language models (LLMs) are increasingly used in interactive applications, and human evaluation remains the gold standard for assessing their performance in multi-turn conversations. Since human studies are costly, time-consuming, and hard to reproduce, recent work explores using LLMs to simulate users for automatic assistant evaluation. However, there is no benchmark or systematic study to evaluate whether these simulated users are reliable stand-ins for real users. To address this, we introduce SimulatorArena, a benchmark of 909 annotated human{--}LLM conversations on two interactive tasks{---}math tutoring and document creation. SimulatorArena evaluates simulators based on how closely their messages match human behavior and how well their assistant ratings align with human judgments. Experiments on various simulator methods show that simulators conditioned on user profiles, capturing traits like background and message styles, align closely with human judgments. They reach Spearman{'}s $\rho$ of 0.7 on both tasks, providing a practical, scalable alternative to human evaluation. Using the best simulator for each task, we benchmark 18 assistants, including the latest LLMs such as GPT-5, Claude 4.1 Opus, and Gemini 2.5 Pro."
}