@inproceedings{liu-etal-2026-logitspec,
    title = "{L}ogit{S}pec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation",
    author = "Liu, Tianyu  and
      Lv, Qitan  and
      Li, Hao  and
      Gao, Xing  and
      Sun, Xiao  and
      Sun, Xiaoyan",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-nlpsi/2026.findings-acl.1655/",
    doi = "10.18653/v1/2026.findings-acl.1655",
    pages = "33070--33092",
    ISBN = "979-8-89176-395-1",
    abstract = "Speculative decoding (SD), where a small draft model is employed to propose \textit{draft} tokens in advance and then the target model validates them in parallel, has emerged as a promising technique for LLM inference acceleration. Many endeavors to improve SD are to eliminate the need for a draft model and generate draft tokens in a retrieval-based manner in order to further alleviate the drafting overhead and significantly reduce the difficulty in deployment and applications. However, retrieval-based SD relies on a matching paradigm to retrieve the most relevant reference as the draft tokens, where these methods often fail to find matched and accurate draft tokens. To address this challenge, we propose \textit{LogitSpec} to effectively expand the retrieval range and find the most relevant reference as drafts. \textit{LogitSpec} is motivated by the observation that the logit of the last token can not only predict \textbf{the next token}, but also speculate \textbf{the next next token}. Specifically, \textit{LogitSpec} generates draft tokens in two steps: (1) utilizing the last logit to speculate the next next token; (2) retrieving relevant reference for both the next token and the next next token. \textit{LogitSpec} is training-free and plug-and-play, which can be easily integrated into existing LLM inference frameworks. Extensive experiments on a wide range of text generation benchmarks demonstrate that \textit{LogitSpec} can achieve up to 2.61{\texttimes} speedup and 3.28 mean accepted tokens per decoding step."
}