@inproceedings{chitty-venkata-etal-2026-pagedeviction,
    title = "{P}aged{E}viction: Structured Block-wise {KV} Cache Pruning for Efficient Large Language Model Inference",
    author = "Chitty-Venkata, Krishna Teja  and
      Ye, Jie  and
      Raskar, Siddhisanket  and
      Kougkas, Anthony  and
      Sun, Xian  and
      Emani, Murali  and
      Vishwanath, Venkatram  and
      Nicolae, Bogdan",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.findings-eacl.168/",
    pages = "3207--3218",
    ISBN = "979-8-89176-386-9",
    abstract = "KV caching significantly improves the efficiency of Large Language Model (LLM) inference by storing attention states from previously processed tokens, enabling faster generation of subsequent tokens. However, as sequence length increases, the KV cache quickly becomes a major memory bottleneck. To address this, we propose PagedEviction, a novel fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM{'}s PagedAttention. Unlike existing approaches that rely on attention-based token importance or evict tokens across different vLLM pages, PagedEviction introduces an efficient block-wise eviction algorithm tailored for paged memory layouts. Our method integrates seamlessly with PagedAttention without requiring any modifications to its CUDA attention kernels. We evaluate PagedEviction across Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and Llama-3.2-3B-Instruct models on the LongBench benchmark suite, demonstrating improved memory usage with better accuracy than baselines on long context tasks."
}