@inproceedings{wen-etal-2026-speccache,
    title = "{S}pec{C}ache: Speculative {KV} Cache Reuse for Efficient {RAG} Serving",
    author = "Wen, Zijian  and
      Zhang, Tao  and
      Chen, Shuangwu  and
      Ye, Shenghao  and
      Guo, Yu  and
      Chen, Qirui  and
      Shuai, Jingxian  and
      Hou, Yunpeng  and
      He, Huasen  and
      Jianyang",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.859/",
    pages = "18861--18871",
    ISBN = "979-8-89176-390-6",
    abstract = "Retrieval-Augmented Generation (RAG) significantly enhances LLMs but faces high prefill latency during long-context processing. While KV cache reuse can mitigate this, current methods relying on shallow features or static heuristics often fail to identify critical tokens for recomputation, resulting in generation quality degradation.We have an insight that KV deviations are more pronounced in deep layers.However, directly extracting deep-layer features from the target model is computationally prohibitive. Crucially, we find that the deep-layer features of a lightweight speculative model exhibit strong consistency with the target model in the selection of critical tokens for recomputation.In light of these insights, we propose SpecCache, which employs deep-layer hidden-state norms from a speculative model as a proxy to guide the critical token selection for target large model.Experiments demonstrate that SpecCache outperforms state-of-the-art (SOTA) baselines. Compared to full KV recomputation, it reduces time-to-first-token (TTFT) by $2.17-3.95\times$ and increases inference throughput by $2.7-5.2\times$, with negligible degradation in generation quality relative to full recomputation."
}