@inproceedings{yang-etal-2026-zoomr,
    title = "{Z}oom{R}: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval",
    author = "Yang, David H.  and
      Zhu, Yuxuan  and
      Amiri, Mohammad Mohammadi  and
      Murugesan, Keerthiram  and
      Pedapati, Tejaswini  and
      Chaudhury, Subhajit  and
      Chen, Pin-Yu",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.76/",
    pages = "1656--1669",
    ISBN = "979-8-89176-390-6",
    abstract = "Large language models (LLMs) have shown great performance on complex reasoning tasks but often require generating long intermediate thoughts before reaching a final answer. During generation, LLMs rely on a key-value (KV) cache for autoregressive decoding. However, the memory footprint of the KV cache grows with output length. Prior work on KV cache optimization mostly focus on compressing the long input context, while retaining the full KV cache for decoding. For tasks requiring long output generation, this leads to increased computational and memory costs. In this paper, we introduce ZoomR, a novel approach that enables LLMs to adaptively compress verbose reasoning thoughts into summaries and uses a dynamic KV cache selection policy that leverages these summaries while also strategically ``zooming in'' on fine-grained details. By using summary keys as a coarse-grained index during decoding, ZoomR uses the query to retrieve details for only the most important thoughts. This hierarchical strategy significantly reduces memory usage by avoiding full-cache attention at each step. Experiments across math and reasoning tasks show that our approach achieves competitive performance compared to baselines, while reducing inference memory requirements by more than $4 \times$. These results demonstrate that a multi-granularity KV selection enables more memory efficient decoding, especially for long output generation."
}