Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech

Rongjie Huang, Chunlei Zhang, Yi Ren, Zhou Zhao, Dong Yu

Abstract
Expressive text-to-speech aims to generate high-quality samples with rich and diverse prosody, which is hampered by dual challenges: 1) prosodic attributes in highly dynamic voices are difficult to capture and model without intonation; and 2) highly multimodal prosodic representations cannot be well learned by simple regression (e.g., MSE) objectives, which causes blurry and over-smoothing predictions. This paper proposes Prosody-TTS, a two-stage pipeline that enhances prosody modeling and sampling by introducing several components: 1) a self-supervised masked autoencoder to model the prosodic representation without relying on text transcriptions or local prosody attributes, which ensures to cover diverse speaking voices with superior generalization; and 2) a diffusion model to sample diverse prosodic patterns within the latent space, which prevents TTS models from generating samples with dull prosodic performance. Experimental results show that Prosody-TTS achieves new state-of-the-art in text-to-speech with natural and expressive synthesis. Both subjective and objective evaluation demonstrate that it exhibits superior audio quality and prosody naturalness with rich and diverse prosodic attributes. Audio samples are available at https://improved_prosody.github.io
Anthology ID:
2023.findings-acl.508
Volume:
Findings of the Association for Computational Linguistics: ACL 2023
Month:
July
Year:
2023
Address:
Toronto, Canada
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8018–8034
Language:
URL:
https://aclanthology.org/2023.findings-acl.508
DOI:
10.18653/v1/2023.findings-acl.508
Bibkey:
Cite (ACL):
Rongjie Huang, Chunlei Zhang, Yi Ren, Zhou Zhao, and Dong Yu. 2023. Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8018–8034, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech (Huang et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/remove-xml-comments/2023.findings-acl.508.pdf
PDF Search