@inproceedings{yan-etal-2026-beyond-cross,
    title = "Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models",
    author = "Yan, Hanqi  and
      Cui, Xiangxiang  and
      Yin, Lu  and
      Gu, Jindong  and
      Liang, Paul Pu  and
      He, Yulan  and
      Wang, Yifei",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.588/",
    pages = "12123--12138",
    ISBN = "979-8-89176-395-1",
    abstract = "The success of vision-language models is primarily attributed to effective cross-modal alignment between vision and language. However, modality gaps persist even in well-aligned models and may be necessary for human perception, as evidenced by modality-specific phenomena such as visual texture and linguistic tone. These observations motivate us to computationally measure and leverage modality gaps to explore their utility in downstream applications. In this paper, we introduce the \textbf{M}odality \textbf{D}ominance \textbf{S}core (\textbf{MDS}), which attributes multimodal features to specific modalities by categorizing them as vision-dominant, language-dominant, or cross-modal. We then propose automatic interpretability metrics to evaluate these modality-specific features in a scalable manner. Finally, we demonstrate how the identified modality-specific features enable training-free probing and editing methods for understanding model perception across genders, generating adversarial examples, and controlling text-to-image generation. Combined with task-agnostic interpretability tools, our work provides a systematic framework for analyzing and efficiently controlling multimodal models."
}