@inproceedings{zuo-etal-2025-rhythm,
    title = "Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching",
    author = "Zuo, Jialong  and
      Ji, Shengpeng  and
      Fang, Minghui  and
      Li, Mingze  and
      Jiang, Ziyue  and
      Cheng, Xize  and
      Yang, Xiaoda  and
      Feiyang, Chen  and
      Duan, Xinyu  and
      Zhao, Zhou",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.790/",
    pages = "16203--16217",
    ISBN = "979-8-89176-251-0",
    abstract = "Zero-Shot Voice Conversion (VC) aims to transform the source speaker{'}s timbre into an arbitrary unseen one while retaining speech content. Most prior work focuses on preserving the source{'}s prosody, while fine-grained timbre information may leak through prosody, and transferring target prosody to synthesized speech is rarely studied. In light of this, we propose R-VC, a rhythm-controllable and efficient zero-shot voice conversion model. R-VC employs data perturbation techniques and discretize source speech into Hubert content tokens, eliminating much content-irrelevant information. By leveraging a Mask Generative Transformer for in-context duration modeling, our model adapts the linguistic content duration to the desired target speaking style, facilitating the transfer of the target speaker{'}s rhythm. Furthermore, R-VC introduces a powerful Diffusion Transformer (DiT) with shortcut flow matching during training, conditioning the network not only on the current noise level but also on the desired step size, enabling high timbre similarity and quality speech generation in fewer sampling steps, even in just two, thus minimizing latency. Experimental results show that R-VC achieves comparable speaker similarity to state-of-the-art VC methods with a smaller dataset, and surpasses them in terms of speech naturalness, intelligibility and style transfer performance."
}