@inproceedings{yin-etal-2025-sea,
    title = "{SEA}: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in {MLLM}s",
    author = "Yin, Yuanyang  and
      Zhao, Yaqi  and
      Zhang, Yajie  and
      Zhang, Yuanxing  and
      Lin, Ke  and
      Wang, Jiahao  and
      Tao, Xin  and
      Wan, Pengfei  and
      Zhang, Wentao  and
      Zhao, Feng",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.55/",
    pages = "1058--1070",
    ISBN = "979-8-89176-332-6",
    abstract = "Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual inputs, yet modality alignment remains one of the most challenging aspects. Current MLLMs typically rely on simple adapter architectures and pretraining approaches to bridge vision encoders with large language models (LLM), guided by image-level supervision. We identify this paradigm often leads to suboptimal alignment between modalities, significantly constraining the LLM{'}s ability to properly interpret and reason with visual features particularly for smaller language models. To address this fundamental limitation, we propose Supervised Embedding Alignment (SEA), a token-level supervision alignment method that enables more precise visual-text alignment during pretraining. SEA introduces minimal computational overhead while preserving language capabilities and substantially improving cross-modal understanding. Our comprehensive analyses reveal critical insights into the adapter{'}s role in multimodal integration, and extensive experiments demonstrate that SEA consistently improves performance across various model sizes, with smaller models benefiting the most (average performance gain of 7.61{\%} for Gemma-2B). This work establishes a foundation for developing more effective alignment strategies for future multimodal systems."
}