@inproceedings{tu-etal-2025-mlan,
    title = "{MLAN}: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models",
    author = "Tu, Jianhong  and
      Ni, Zhuohao  and
      Crispino, Nicholas  and
      Yu, Zihao  and
      Bendersky, Michael  and
      Gunel, Beliz  and
      Jia, Ruoxi  and
      Liu, Xin  and
      Lyu, Lingjuan  and
      Song, Dawn  and
      Wang, Chenguang",
    editor = "Zhang, Yuji  and
      Chen, Canyu  and
      Li, Sha  and
      Geva, Mor  and
      Han, Chi  and
      Wang, Xiaozhi  and
      Feng, Shangbin  and
      Gao, Silin  and
      Augenstein, Isabelle  and
      Bansal, Mohit  and
      Li, Manling  and
      Ji, Heng",
    booktitle = "Proceedings of the 3rd Workshop on Towards Knowledgeable Foundation Models (KnowFM)",
    month = aug,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.knowllm-1.6/",
    pages = "59--74",
    ISBN = "979-8-89176-283-1",
    abstract = "We present a novel visual instruction tuning strategy to improve the zero-shot task generalization of multimodal large language models by building a firm text-only knowledge base. Existing work lacks sufficient experimentation on the importance of each modality in the instruction tuning stage, often using a majority of vision-language data while keeping text-only data limited and fixing mixtures of modalities. By incorporating diverse text-only data in the visual instruction tuning stage, we vary vision-language data in various controlled experiments to investigate the importance of modality in visual instruction tuning. Our comprehensive evaluation shows that the text-heavy instruction tuning approach is able to perform on par with traditional vision-heavy mixtures on both modalities across 12 general datasets while using as low as half the total training tokens. We find that simply increasing sufficiently diverse text-only data enables transfer of instruction following ability and domain knowledge across modalities while being more efficient than the vision-language approach."
}