@inproceedings{zhang-etal-2025-revealing,
    title = "Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in {LLM} Role-Playing",
    author = "Zhang, Wenyuan  and
      Nie, Shuaiyi  and
      Sheng, Jiawei  and
      Zhang, Zefeng  and
      Zhang, Xinghua  and
      He, Yongquan  and
      Liu, Tingwen",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1689/",
    pages = "33267--33290",
    ISBN = "979-8-89176-332-6",
    abstract = "Large language model (LLM) role-playing has gained widespread attention. Authentic character knowledge is crucial for constructing realistic LLM role-playing agents. However, existing works usually overlook the exploration of LLMs' ability to detect characters' known knowledge errors (KKE) and unknown knowledge errors (UKE) while playing roles, which would lead to low-quality automatic construction of character trainable corpus. In this paper, we propose RoleKE-Bench to evaluate LLMs' ability to detect errors in KKE and UKE. The results indicate that even the latest LLMs struggle to detect these two types of errors effectively, especially when it comes to familiar knowledge. We experimented with various reasoning strategies and propose an agent-based reasoning method, Self-Recollection and Self-Doubt (S$^2$RD), to explore further the potential for improving error detection capabilities."
}