@inproceedings{zhang-etal-2025-clusterattn,
    title = "{C}luster{A}ttn: {KV} Cache Compression under Intrinsic Attention Clustering",
    author = "Zhang, Minwei  and
      Sun, Haifeng  and
      Wang, Jingyu  and
      Li, Shaolong  and
      Ning, Wanyi  and
      Qi, Qi  and
      Zhuang, Zirui  and
      Liao, Jianxin",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.703/",
    pages = "14451--14473",
    ISBN = "979-8-89176-251-0",
    abstract = "Sparse attention can effectively alleviate the significant demands on memory when large language models (LLMs) process long contexts. Existing methods typically apply the same sparse pattern across different attention heads and inputs. However, this uniform approach fails to capture the inherent diversity of attention patterns within LLMs {---} the intrinsic attention clustering. To address this, we propose ClusterAttn, a training-free sparse attention method that provides an efficient prompt cache compression scheme under intrinsic attention clustering for efficient LLM inference.Our findings show that attention heads consistently focus on specific clusters of the prompt during decoding, a pattern detectable from an observation window at the prompt{'}s end. ClusterAttn adaptively fits these clusters utilizing a density-based attention clustering algorithm, thus compressing the KV cache of the prompt. Evaluations on different models across various benchmarks demonstrate ClusterAttn{'}s superior compression rates and efficiency. By utilizing only 1024 tokens, it can reduce memory usage by 10{\%}{--}65{\%}, resulting in a latency reduction of 12{\%}{--}23{\%} and a throughput increase of 2.6{--}4.8 times, all with nearly no accuracy loss. Additionally, ClusterAttn can handle up to 128k context on a single A100-80GB GPU, outperforming existing methods."
}