MultiVerse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech

Taejun Bak, Youngsik Eom, SeungJae Choi, Young-Sun Joo

Abstract
Text-to-speech (TTS) systems that scale up the amount of training data have achieved significant improvements in zero-shot speech synthesis. However, these systems have certain limitations: they require a large amount of training data, which increases costs, and often overlook prosody similarity. To address these issues, we propose MultiVerse, a zero-shot multi-task TTS system that is able to perform TTS or speech style transfer in zero-shot and cross-lingual conditions. MultiVerse requires much less training data than traditional data-driven approaches. To ensure zero-shot performance even with limited data, we leverage source-filter theory-based disentanglement, utilizing the prompt for modeling filter-related and source-related representations. Additionally, to further enhance prosody similarity, we adopt a prosody modeling approach combining prompt-based autoregressive and non-autoregressive methods. Evaluations demonstrate the remarkable zero-shot multi-task TTS performance of MultiVerse and show that MultiVerse not only achieves zero-shot TTS performance comparable to data-driven TTS systems with much less data, but also significantly outperforms other zero-shot TTS systems trained with the same small amount of data. In particular, our novel prosody modeling technique significantly contributes to MultiVerse’s ability to generate speech with high prosody similarity to the given prompts.
Anthology ID:
2024.findings-emnlp.533
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2024
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9130–9147
Language:
URL:
https://preview.aclanthology.org/icon-24-ingestion/2024.findings-emnlp.533/
DOI:
10.18653/v1/2024.findings-emnlp.533
Bibkey:
Cite (ACL):
Taejun Bak, Youngsik Eom, SeungJae Choi, and Young-Sun Joo. 2024. MultiVerse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9130–9147, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
MultiVerse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech (Bak et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/icon-24-ingestion/2024.findings-emnlp.533.pdf
PDF Search Fix data