@inproceedings{wang-etal-2025-vocalnet,
    title = "{V}ocal{N}et: Speech {LLM}s with Multi-Token Prediction for Faster and High-Quality Generation",
    author = "Wang, Yuhao  and
      Liu, Heyang  and
      Cheng, Ziyang  and
      Wu, Ronghua  and
      Gu, Qunshan  and
      Wang, Yanfeng  and
      Wang, Yu",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.989/",
    pages = "19595--19612",
    ISBN = "979-8-89176-332-6",
    abstract = "Speech large language models (LLMs) have emerged as a prominent research focus in speech processing. In this work, we introduce VocalNet, a series of high-performance speech LLMs featuring a scalable and model-agnostic training framework as well as a novel multi-token prediction (MTP) paradigm for speech generation. We first propose an efficient two-stage training framework that enables LLMs to acquire real-time speech interaction capabilities. Through extensive experiments on various training configurations, we ensure both simplicity and effectiveness in the training strategy. Furthermore, inspired by advances in language modeling, we introduce MTP into the domain of speech LLMs{---}an alternative to traditional next-token prediction (NTP){---}which enables the model to predict multiple future tokens at each step. Through systematic analysis and improved implementation, we show that MTP not only accelerates inference speed but also significantly enhances speech quality. Experimental results demonstrate that VocalNet achieves performance comparable to state-of-the-art Omni LLMs while outperforming existing open-source speech LLMs, despite using limited training data."
}