@inproceedings{perin-etal-2024-rankmean,
    title = "{R}ank{M}ean: Module-Level Importance Score for Merging Fine-tuned {LLM} Models",
    author = "Perin, Gabriel  and
      Chen, Xuxi  and
      Liu, Shusen  and
      Kailkhura, Bhavya  and
      Wang, Zhangyang  and
      Gallagher, Brian",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.findings-acl.104/",
    doi = "10.18653/v1/2024.findings-acl.104",
    pages = "1776--1782",
    abstract = "Traditionally, developing new language models (LMs) capable of addressing multiple tasks involves fine-tuning pre-trained LMs using a wide collection of datasets, a process that often incurs significant computational expenses. Model merging emerges as a cost-effective alternative, allowing the integration of existing models fine-tuned on different tasks into a single model that performs well across all tasks, eliminating the need for additional training. In this paper, we propose RankMean, an algorithm for merging fine-tuned LMs without requiring any downstream data. RankMean determines merging coefficients based on the relative rankings of weight change magnitudes and applies these coefficients for module-wise integration of various fine-tuned models. Our experimental results demonstrate that RankMean outperforms existing baseline methods on multiple benchmarks. The code is available at https://github.com/VITA-Group/RankMean."
}