@inproceedings{hussain-etal-2025-koel,
    title = "Koel-{TTS}: Enhancing {LLM} based Speech Generation with Preference Alignment and Classifier Free Guidance",
    author = "Hussain, Shehzeen Samarah  and
      Neekhara, Paarth  and
      Yang, Xuesong  and
      Casanova, Edresson  and
      Ghosh, Subhankar  and
      Fejgin, Roy  and
      Desta, Mikyas T.  and
      Valle, Rafael  and
      Li, Jason",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1076/",
    pages = "21230--21245",
    ISBN = "979-8-89176-332-6",
    abstract = "Autoregressive speech token generation models produce speech with remarkable variety and naturalness but often suffer from hallucinations and undesired vocalizations that do not conform to conditioning inputs. To address these challenges, we introduce Koel-TTS, an encoder-decoder transformer model for multilingual TTS that improves contextual adherence of speech generation LLMs through preference alignment and classifier-free guidance (CFG). For preference alignment, we design a reward system that ranks model outputs using automatic metrics derived from speech recognition and speaker verification models, encouraging generations that better match the input text and speaker identity. CFG further allows fine-grained control over the influence of conditioning inputs during inference by interpolating conditional and unconditional logits. Notably, applying CFG to a preference-aligned model yields additional gains in transcription accuracy and speaker similarity, demonstrating the complementary benefits of both techniques. Koel-TTS achieves state-of-the-art results in zero-shot TTS, outperforming prior LLM-based models on intelligibility, speaker similarity, and naturalness, despite being trained on significantly less data."
}