@inproceedings{liu-etal-2025-hfsd,
    title = "{HFSD}-{V}2{C}: Zero-Shot Visual Voice Cloning Via Hierarchical Face-Styled Diffusion Model",
    author = "Liu, Yaping  and
      Wang, Linqin  and
      Gao, Shengxiang  and
      Yu, Zhengtao  and
      Dong, Ling",
    editor = "Sun, Maosong  and
      Duan, Peiyong  and
      Liu, Zhiyuan  and
      Xu, Ruifeng  and
      Sun, Weiwei",
    booktitle = "Proceedings of the 24th {C}hina National Conference on Computational Linguistics ({CCL} 2025)",
    month = aug,
    year = "2025",
    address = "Jinan, China",
    publisher = "Chinese Information Processing Society of China",
    url = "https://preview.aclanthology.org/ingest-ccl/2025.ccl-1.77/",
    pages = "1020--1030",
    abstract = "``The goal of this work is zero-shot visual voice cloning (ZS-V2C), which aims to generate speech samples with unseen speaker identity and prosody derived from a video clip and an acoustic reference. ZS-V2C presents greater challenges as: 1) unseen speaker modeling; and 2) unseen prosody modeling. Unlike previous works, we propose a novel ZS-V2C framework that incorporates a hierarchical face-styled diffusion model (HFSD-V2C). Specifically, first, we leverage cross-modal biometrics to predict unseen speaker embeddings based on facial features. Then, we jointly model the unseen prosodic features at the text, speech and video levels. Finally, a diffusion model is constructed based on the embeddings of the unseen speaker and prosodic features,enabling the generation of expressive and diverse speech. Extensive experiments on the LRS2and GRID benchmark dataset demonstrate the superior performance of our proposed method.''"
}