@inproceedings{liang-etal-2026-expanding,
    title = "Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Models",
    author = "Liang, Qiao  and
      Liu, Yanjiang  and
      Zhou, Weixiang  and
      He, Ben  and
      Lu, Yaojie  and
      Lin, Hongyu  and
      Zheng, Jia  and
      Han, Xianpei  and
      Sun, Le  and
      Sun, Yingfei",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.146/",
    pages = "3170--3184",
    ISBN = "979-8-89176-380-7",
    abstract = "Does the prior knowledge of the vision encoder constrain the capability boundary of Multi-modal Large Language Models (MLLMs)? While most existing research treats MLLMs as unified systems optimized through end-to-end training, the impact of vision encoder{'}s prior knowledge is seldom investigated. In this work, we introduce a novel metric $Rank_e$ to quantify the effect of prior knowledge of the vision encoder on MLLM performance. Our analysis reveals a positive correlation between prior knowledge and MLLM performance. Moreover, we find that domain-specific fine-tuning using solely end-to-end visual question answering (VQA) data is insufficient, particularly for entities with low inherent visual prior knowledge. To address this issue, we propose VisPRE (Vision Prior Remediation), a two-stage training framework that explicitly incorporates prior knowledge at the vision encoder level. Experimental results demonstrate that augmenting vision encoder{'}s prior knowledge substantially boosts the visual understanding capabilities of MLLMs, offering a novel and effective strategy for improving performance, especially in scenarios involving uncommon visual entities."
}