@inproceedings{shi-etal-2026-heterocache,
    title = "{H}etero{C}ache: A Dynamic Retrieval Approach to Heterogeneous {KV} Cache Compression for Long-Context {LLM} Inference",
    author = "Shi, Zhiyuan  and
      Qiu, Qibo  and
      Xuefeng  and
      Jiang, Zhonglin  and
      Yu, Li  and
      Jiang, Jian  and
      He, Xiaofei  and
      Wang, Wenxiao",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1999/",
    pages = "43172--43187",
    ISBN = "979-8-89176-390-6",
    abstract = "The linear memory growth of the KV cache poses a significant bottleneck for LLM inference in long-context tasks. Existing static compression methods often fail to preserve globally important information. Although recent dynamic retrieval approaches attempt to address this issue, they typically suffer from coarse-grained caching strategies and incur high I/O overhead. To overcome these limitations, we propose HeteroCache, a training-free dynamic compression framework. Our method is built on two key insights: attention heads exhibit diverse temporal heterogeneity, and there is significant spatial redundancy among heads within the same layer.Guided by these insights, HeteroCache categorizes heads based on stability and similarity, applying a fine-grained weighting strategy that allocates larger cache budgets to heads with rapidly shifting attention to capture context changes.Furthermore, it features a hierarchical storage mechanism where representative heads monitor attention drift to trigger asynchronous, on-demand context retrieval, thereby hiding I/O latency.Experiments demonstrate that HeteroCache achieves state-of-the-art performance on long-context benchmarks and accelerates decoding by up to 3{\texttimes} compared to the original model with a 224K context. Our code is available at https://github.com/ponytaill/HeteroCache."
}