@inproceedings{chen-etal-2025-towards-design,
    title = "Towards a Design Guideline for {RPA} Evaluation: A Survey of Large Language Model-Based Role-Playing Agents",
    author = "Chen, Chaoran  and
      Yao, Bingsheng  and
      Zou, Ruishi  and
      Hua, Wenyue  and
      Lyu, Weimin  and
      Li, Toby Jia-Jun  and
      Wang, Dakuo",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.938/",
    pages = "18229--18268",
    ISBN = "979-8-89176-256-5",
    abstract = "Role-Playing Agent (RPA) is an increasingly popular type of LLM Agent that simulates human-like behaviors in a variety of tasks. However, evaluating RPAs is challenging due to diverse task requirements and agent designs.This paper proposes an evidence-based, actionable, and generalizable evaluation design guideline for LLM-based RPA by systematically reviewing 1,676 papers published between Jan. 2021 and Dec. 2024.Our analysis identifies six agent attributes, seven task attributes, and seven evaluation metrics from existing literature.Based on these findings, we present an RPA evaluation design guideline to help researchers develop more systematic and consistent evaluation methods."
}