@inproceedings{zhang-etal-2025-omnicharacter,
    title = "{O}mni{C}haracter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction",
    author = "Zhang, Haonan  and
      Luo, Run  and
      Liu, Xiong  and
      Wu, Yuchuan  and
      Lin, Ting-En  and
      Zeng, Pengpeng  and
      Qu, Qiang  and
      Fang, Feiteng  and
      Yang, Min  and
      Gao, Lianli  and
      Song, Jingkuan  and
      Huang, Fei  and
      Li, Yongbin",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1276/",
    pages = "26318--26331",
    ISBN = "979-8-89176-251-0",
    abstract = "Role-Playing Agents (RPAs), benefiting from large language models, is an emerging interactive AI system that simulates roles or characters with diverse personalities. However, existing methods primarily focus on mimicking dialogues among roles in textual form, neglecting the role{'}s voice traits (e.g., voice style and emotions) as playing a crucial effect in interaction, which tends to be more immersive experiences in realistic scenarios. Towards this goal, we propose OmniCharacter, a first seamless speech-language personality interaction model to achieve immersive RPAs with low latency. Specifically, OmniCharacter enables agents to consistently exhibit role-specific personality traits and vocal traits throughout the interaction, enabling a mixture of speech and language responses. To align the model with speech-language scenarios, we construct a dataset named OmniCharacter-10K, which involves more distinctive characters (20), richly contextualized multi-round dialogue (10K), and dynamic speech response (135K). Experimental results showcase that our method yields better responses in terms of both content and style compared to existing RPAs and mainstream speech-language models, with a response latency as low as 289ms."
}