@inproceedings{qu-etal-2025-uq,
    title = "$\texttt{UQ-Merge}$: Uncertainty Guided Multimodal Large Language Model Merging",
    author = "Qu, Huaizhi  and
      Zhao, Xinyu  and
      Peng, Jie  and
      Lee, Kwonjoon  and
      Dariush, Behzad  and
      Chen, Tianlong",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.73/",
    pages = "1401--1417",
    ISBN = "979-8-89176-256-5",
    abstract = "Multimodal Large Language Models (MLLMs) have gained increasing popularity as a promising framework for leveraging the strong language reasoning capabilities in the vision-language domain. Given a wide range of MLLMs, model merging potentially offers a cheap way to aggregate their diverse knowledge into a single MLLM. However, directly plug-in existing model merging approaches often leads to suboptimal performance due to (1) inclusion of harmful models that have over-confident predictions in the target task; (2) the lack of specialized designs for vision-language inputs. To tackle these pain points, we conduct pioneering investigations to dissect the merging procedures and propose an uncertainty-guided MLLM merging algorithm, $\textit{i.e.}$, $\texttt{UQ-Merge}$, which $i$) identifies beneficial candidates for merging, $ii$) determines the merging order and the number of helpful candidates, and $iii$) performs appropriate merging. Within our framework, we consider uncertainty quantification on both text and vision inputs to examine the MLLM prediction confidence, and then decide whether and when a MLLM needs to be included. It is worth mentioning that our vision-language uncertainty quantification does not require access to sample labels, making it more practical in various scenarios. Extensive experiments consistently demonstrate the superior MLLM merging performance of $\texttt{UQ-Merge}$ in both held-in and held-out vision-language benchmarks. For example, compared to existing state-of-the-art merging methods, $\texttt{UQ-Merge}$ brings substantial performance improvements of up to 44.3{\%} on average accuracy in 12 datasets. Codes are available at https://anonymous.4open.science/r/UQ-Merge-7CD7."
}