@inproceedings{zeng-etal-2025-modalprompt,
    title = "{M}odal{P}rompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt",
    author = "Zeng, Fanhu  and
      Zhu, Fei  and
      Guo, Haiyang  and
      Zhang, Xu-Yao  and
      Liu, Cheng-Lin",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.609/",
    pages = "12137--12152",
    ISBN = "979-8-89176-332-6",
    abstract = "Large Multimodal Models (LMMs) exhibit remarkable multi-tasking ability by learning mixed instruction datasets. However, novel tasks would be encountered sequentially in dynamic world, which urges for equipping LMMs with multimodal continual instruction learning (MCIT) ability especially for diverse and challenging generative tasks. Existing MCIT methods do not fully exploit the unique attribute of LMMs and often gain performance at the expense of efficiency. In this paper, we propose a novel prompt learning framework for MCIT to effectively alleviate forgetting of previous knowledge while managing computational complexity with natural image-text supervision. Concretely, we learn prompts for each task and exploit efficient prompt fusion for knowledge transfer and prompt selection for complexity management with dual-modality guidance. Extensive experiments demonstrate that our approach achieves substantial +14.26{\%} performance gain on MCIT benchmarks with remarkable x1.42 inference speed free from growing computation."
}