@inproceedings{zhang-etal-2026-trimtokenator,
    title = "{T}rim{T}okenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models",
    author = "Zhang, Hao  and
      Mengsi, Lyu  and
      He, Chenrui  and
      Ao, Yulong  and
      Lin, Yonghua",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1633/",
    pages = "32633--32650",
    ISBN = "979-8-89176-395-1",
    abstract = "Large Multimodal Models (LMMs) have achieved significant success across various tasks. These models usually encode visual inputs into dense token sequences, which are then concatenated with textual tokens and jointly processed by a language model. However, the increased token count substantially raises computational and memory costs during inference. Token pruning has emerged as a promising approach to address this issue. Existing token pruning methods often rely on costly calibration or suboptimal importance metrics, leading to redundant retained tokens. In this paper, we analyze the redundancy differences between visual and textual tokens and propose pruning exclusively on visual tokens. Based on this, we propose a visual token pruning strategy that explicitly preserves both cross-modal alignment and intra-modal informational diversity. We introduce a mutual information-based token pruning strategy that removes visual tokens semantically misaligned with textual tokens, effectively preserving the alignment between the visual and textual modalities. We further refine the retained tokens by maximizing their expected pairwise distances in the latent space to enhance representational quality and reduce redundancy. which is solved efficiently with a greedy algorithm. Extensive experiments demonstrate that our method maintains strong performance while reducing tokens by 88.9{\%} on models such as LLaVA-1.5-7B and LLaVA-NEXT-7B, resulting in a 56.7{\%} improvement in inference speed."
}