@inproceedings{ji-etal-2025-controlspeech,
    title = "{C}ontrol{S}peech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control",
    author = "Ji, Shengpeng  and
      Chen, Qian  and
      Wang, Wen  and
      Zuo, Jialong  and
      Fang, Minghui  and
      Jiang, Ziyue  and
      Huang, Hai  and
      Wang, Zehan  and
      Cheng, Xize  and
      Zheng, Siqi  and
      Zhao, Zhou",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.346/",
    pages = "6966--6981",
    ISBN = "979-8-89176-251-0",
    abstract = "In this paper, we present ControlSpeech, a text-to-speech (TTS) system capable of fully cloning the speaker{'}s voice and enabling arbitrary control and adjustment of speaking style. Prior zero-shot TTS models only mimic the speaker{'}s voice without further control and adjustment capabilities while prior controllable TTS models cannot perform speaker-specific voice generation. Therefore, ControlSpeech focuses on a more challenging task{---}a TTS system with controllable timbre, content, and style at the same time. ControlSpeech takes speech prompts, content prompts, and style prompts as inputs and utilizes bidirectional attention and mask-based parallel decoding to capture codec representations corresponding to timbre, content, and style in a discrete decoupling codec space. Moreover, we analyze the many-to-many issue in textual style control and propose the Style Mixture Semantic Density (SMSD) module, which is based on Gaussian mixture density networks, to resolve this problem. To facilitate empirical validations, we make available a new style controllable dataset called VccmDataset. Our experimental results demonstrate that ControlSpeech exhibits comparable or state-of-the-art (SOTA) performance in terms of controllability, timbre similarity, audio quality, robustness, and generalizability. Codes are available at https://github.com/jishengpeng/ControlSpeech."
}