@inproceedings{li-etal-2024-pause,
    title = "Pause-Aware Automatic Dubbing using {LLM} and Voice Cloning",
    author = "Li, Yuang  and
      Guo, Jiaxin  and
      Zhang, Min  and
      Miaomiao, Ma  and
      Rao, Zhiqiang  and
      Zhang, Weidong  and
      He, Xianghui  and
      Wei, Daimeng  and
      Yang, Hao",
    editor = "Salesky, Elizabeth  and
      Federico, Marcello  and
      Carpuat, Marine",
    booktitle = "Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand (in-person and online)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.iwslt-1.2",
    doi = "10.18653/v1/2024.iwslt-1.2",
    pages = "12--16",
    abstract = "Automatic dubbing aims to translate the speech of a video into another language, ensuring the new speech naturally fits the original video. This paper details Huawei Translation Services Center{'}s (HW-TSC) submission for IWSLT 2024{'}s automatic dubbing task, under an unconstrained setting. Our system{'}s machine translation (MT) component utilizes a Transformer-based MT model and an LLM-based post-editor to produce translations of varying lengths. The text-to-speech (TTS) component employs a VITS-based TTS model and a voice cloning module to emulate the original speaker{'}s vocal timbre. For enhanced dubbing synchrony, we introduce a parsing-informed pause selector. Finally, we rerank multiple results based on lip-sync error distance (LSE-D) and character error rate (CER). Our system achieves LSE-D of 10.75 and 12.19 on subset1 and subset2 of DE-EN test sets respectively, superior to last year{'}s best system.",
}