@inproceedings{wang-etal-2025-activating,
    title = "Activating Distributed Visual Region within {LLM}s for Efficient and Effective Vision-Language Training and Inference",
    author = "Wang, Siyuan  and
      Wang, Dianyi  and
      Zhou, Chengxing  and
      Li, Zejun  and
      Fan, Zhihao  and
      Huang, Xuanjing  and
      Wei, Zhongyu",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1484/",
    pages = "30715--30727",
    ISBN = "979-8-89176-251-0",
    abstract = "Large Vision-Language Models (LVLMs) typically learn visual capacity through visual instruction tuning, involving updates to both a projector and their LLM backbones. Inspired by the concept of a visual region in the human brain, we investigate the existence of an analogous \textit{visual region} within LLMs that functions as a cognitive core, and explore the potential of efficient training of LVLMs via selective layers tuning. Using Bunny-Llama-3-8B-V for detailed analysis and other three LVLMs for validation across diverse visual and textual tasks, we find that selectively updating 25{\%} of LLMs layers, when sparsely and uniformly distributed, can preserve nearly 99{\%} of visual performance and maintain or improve textual task results, while effectively reducing training time. Based on this targeted training approach, we further propose a novel visual region-based pruning paradigm, removing non-critical layers outside the visual region, which can achieve minimal performance loss. This study offers an effective and efficient strategy for LVLM training and inference by activating a layer-wise visual region within LLMs, which proves consistently effective across different models."
}