@inproceedings{zhao-etal-2025-fr,
    title = "{FR}-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling",
    author = "Zhao, Weilin  and
      Pan, Tengyu  and
      Han, Xu  and
      Zhang, Yudi  and
      Ao, Sun  and
      Huang, Yuxiang  and
      Zhang, Kaihuo  and
      Zhao, Weilun  and
      Li, Yuxuan  and
      Zhou, Jie  and
      Zhou, Hao  and
      Wang, Jianyong  and
      Sun, Maosong  and
      Liu, Zhiyuan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.198/",
    pages = "3909--3921",
    ISBN = "979-8-89176-251-0",
    abstract = "Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models (LLMs) by utilizing a draft-then-verify mechanism to produce multiple tokens per forward pass. While state-of-the-art speculative sampling methods use only a single layer and a language modeling (LM) head as the draft model to achieve impressive layer compression, their efficiency gains are substantially reduced for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens. To address this, we present FR-Spec, a frequency-ranked speculative sampling framework that optimizes draft candidate selection through vocabulary space compression. By constraining the draft search to a frequency-prioritized token subset, our method reduces LM Head computation overhead by 75{\%} while ensuring the equivalence of the final output distribution. Experiments across multiple datasets demonstrate an average of 1.12$\times$ speedup over the state-of-the-art speculative sampling method EAGLE-2. Code is availableat https://github.com/thunlp/FR-Spec."
}