@inproceedings{wu-etal-2025-mitigating-visual,
    title = "Mitigating Visual Knowledge Forgetting in {MLLM} Instruction-tuning via Modality-decoupled Gradient Descent",
    author = "Wu, Junda  and
      Xiong, Yuxin  and
      Li, Xintong  and
      Xia, Yu  and
      Wang, Ruoyu  and
      Wang, Yu  and
      Yu, Tong  and
      Kim, Sungchul  and
      Rossi, Ryan A.  and
      Yao, Lina  and
      Shang, Jingbo  and
      McAuley, Julian",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-luhme/2025.findings-emnlp.123/",
    doi = "10.18653/v1/2025.findings-emnlp.123",
    pages = "2282--2295",
    ISBN = "979-8-89176-335-7",
    abstract = "Recent MLLMs have demonstrated strong visual understanding and reasoning after large-scale multimodal pre-training. However, instruction-tuning is typically text-driven with limited visual supervision, leading to significant visual forgetting and degradation of pre-trained visual knowledge. Existing fine-tuning and continual learning methods compress visual representations and emphasize task alignment over visual retention, failing to address this challenge. We present a novel perspective using effective rank to quantify the loss of visual representation richness, framing visual forgetting as excessive compression under the information bottleneck principle. To address this, we propose modality-decoupled gradient descent (MDGD), which regulates gradient updates to preserve the effective rank of visual features and explicitly disentangles visual learning from task-specific alignment. We further introduce a memory-efficient fine-tuning variant using gradient masking for parameter-efficient adaptation. Extensive experiments show that MDGD effectively mitigates visual forgetting across downstream tasks and models, maintaining pre-trained visual knowledge while supporting strong task adaptation."
}