Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer
Yongqi Wang, Bai Jionghao, Rongjie Huang, Ruiqi Li, Zhiqing Hong, Zhou Zhao
Abstract
Direct speech-to-speech translation (S2ST) with discrete self-supervised representations has achieved remarkable accuracy, but is unable to preserve the speaker timbre of the source speech. Meanwhile, the scarcity of high-quality speaker-parallel data poses a challenge for learning style transfer during translation. We design an S2ST pipeline with style-transfer capability on the basis of discrete self-supervised speech representations and codec units. The acoustic language model we introduce for style transfer leverages self-supervised in-context learning, acquiring style transfer ability without relying on any speaker-parallel data, thereby overcoming data scarcity. By using extensive training data, our model achieves zero-shot cross-lingual style transfer on previously unseen source languages. Experiments show that our model generates translated speeches with high fidelity and speaker similarity. Audio samples are available at http://stylelm.github.io/ .- Anthology ID:
- 2024.acl-srw.5
- Volume:
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Xiyan Fu, Eve Fleisig
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 34–41
- Language:
- URL:
- https://aclanthology.org/2024.acl-srw.5
- DOI:
- 10.18653/v1/2024.acl-srw.5
- Cite (ACL):
- Yongqi Wang, Bai Jionghao, Rongjie Huang, Ruiqi Li, Zhiqing Hong, and Zhou Zhao. 2024. Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 34–41, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer (Wang et al., ACL 2024)
- PDF:
- https://preview.aclanthology.org/autopr/2024.acl-srw.5.pdf