@inproceedings{zhouxuwen-etal-2026-calibrated,
    title = "Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference",
    author = "Zhouxuwen  and
      Liu, Fangxin  and
      Wang, Chao  and
      Zheng, Xiao  and
      Zheng, Hao  and
      He, Min  and
      Jiang, Li  and
      Guan, Haibing",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1369/",
    pages = "29677--29688",
    ISBN = "979-8-89176-390-6",
    abstract = "Speculative decoding accelerates autoregressive generation by letting draft tokens bypass full verification, but conventional frameworks suffer from frequent false rejections, particularly when draft models produce semantically correct but lexically divergent outputs. In this paper, we present Calibrated Speculative Decoding (CSD), a training-free framework that recovers valid tokens discarded by standard verification. Guided by the principle of ``Frequency-Guided Candidate Selection and Probability-Guarded Acceptance,'' CSD incorporates two lightweight modules: Online Correction Memory, which aggregates historical rejections to propose recurring divergence patterns as rescue candidates, and Semantic Consistency Gating, which verifies candidate admissibility using probability ratios instead of exact token matching. Our evaluation across diverse large language models demonstrates that CSD outperforms existing methods, achieving a peak throughput speedup of 2.33x. CSD preserves model accuracy across all tasks while further boosting performance on complex reasoning datasets. These results establish CSD as a highly effective, lightweight solution for practical LLM deployments."
}