@inproceedings{chung-etal-2026-llm,
    title = "{LLM}-Codec: Neural Audio Codec Meets Language Model Objectives",
    author = "Chung, Ho-Lam  and
      Chen, Yiming  and
      Lee, Hung-yi",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1308/",
    pages = "26251--26265",
    ISBN = "979-8-89176-395-1",
    abstract = "Neural audio codecs are widely used as tokenizers for spoken language models, but they are optimized for waveform reconstruction rather than autoregressive prediction.This mismatch injects acoustically driven uncertainty into the discrete token space and increases language-model perplexity.We propose , which augments codec training with language-model-facing objectives while keeping both codec and LLM architectures unchanged.introduces (i) future token prediction with Medusa-style multi-step heads to encourage multi-step predictability, and (ii) semantic alignment that matches audio and text representations via a memory-bank contrastive loss.A differentiable Gumbel bridge enables end-to-end gradients from these objectives to the codec encoder.On SALMon speech coherence, token LMs trained on reach 61.6{\%} accuracy (+12.1 points over AUV) while reducing perplexity 35$\times$.On Codec-SUPERB-tiny, improves speech Mel distance by 5.0{\%} over AUV while simultaneously achieving the learnability gains, demonstrating that reconstruction fidelity and token predictability can be improved together."
}