@inproceedings{liu-etal-2026-hiprune,
    title = "{H}i{P}rune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Models",
    author = "Liu, Jizhihui  and
      Zhu, Guangdao  and
      Du, Feiyi  and
      Lian, Niu  and
      Li, Jun  and
      Chen, Bin  and
      Guan, Weili  and
      Wang, Yaowei",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.162/",
    pages = "3274--3291",
    ISBN = "979-8-89176-395-1",
    abstract = "Vision-Language Models (VLMs) encode images and videos into abundant tokens, which contain substantial redundancy and computation cost. While visual token pruning mitigates the issue, most existing methods lack insight into the intrinsic property of the vision encoder itself. In this work, we dive into the vision encoder and prove that the middle layers pay more attention to the main objects of the image qualitatively and quantitatively, while the deep layers to tokens with rich global information. Utilizing this Hierarchical attention pattern, we propose HiPrune, a training-free and model-agnostic token Pruning method. HiPrune identifies three types of visual tokens according to their attention in different phases of the vision encoder, which preserves different levels of information. By coupling with the similarity of text tokens, we propose a prompt-aware variance, HiPrune++, which further improves instruction following performance under a very low token budget. Extensive experiments across four representative VLMs show that HiPrune achieves up to 99.3{\%} of task accuracy with only 1/3 of the tokens, while reducing inference FLOPs by 58.7{\%}. HiPrune++ maintains up to 99.9{\%} accuracy with 2/9 tokens, highlighting robustness under high-resolution."
}