@inproceedings{qiang-etal-2026-unisonate,
    title = "{U}ni{S}onate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions",
    author = "Qiang, Chunyu  and
      Wang, Xiaopeng  and
      Yin, Kang  and
      Liang, Yuzhe  and
      Guo, Yuxin  and
      Ma, Teng  and
      Zhang, Ziyu  and
      Wang, Tianrui  and
      Gong, Cheng  and
      Chen, Yushen  and
      Fu, Ruibo  and
      Wang, Longbiao  and
      Dang, Jianwu",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1293/",
    pages = "28043--28054",
    ISBN = "979-8-89176-390-6",
    abstract = "Generative audio modeling has largely been fragmented into specialized tasks, text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA), each operating under heterogeneous control paradigms. Unifying these modalities remains a fundamental challenge due to the intrinsic dissonance between structured semantic representations (speech/music) and unstructured acoustic textures (sound effects). In this paper, we introduce \textbf{UniSonate}, a unified flow-matching framework capable of synthesizing speech, music, and sound effects through a standardized, reference-free natural language instruction interface. To reconcile structural disparities, we propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space, enabling precise duration control within a phoneme-driven Multimodal Diffusion Transformer (MM-DiT). Coupled with a multi-stage curriculum learning strategy, this approach effectively mitigates cross-modal optimization conflicts. Extensive experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS (WER 1.47{\%}) and TTM (SongEval Coherence 3.18), while maintaining competitive fidelity in TTA. Crucially, we observe \textit{positive transfer}, where joint training on diverse audio data significantly enhances structural coherence and prosodic expressiveness compared to single-task baselines."
}