@inproceedings{lee-etal-2026-prime,
    title = "{PRIME}: Ultra-Low-Rank Principal{--}Residual Model Merging",
    author = "Lee, Seung-Ho  and
      Lee, Kyungsu  and
      Zuchi, Bazarvaani  and
      Ahn, Jeongmin  and
      Seo, Insuk  and
      Jeon, Donghyeon  and
      Kang, Inho  and
      Na, Seung-Hoon",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.168/",
    pages = "3415--3436",
    ISBN = "979-8-89176-395-1",
    abstract = "Model merging has emerged as an effective approach for integrating multiple task-specific fine-tuned models into a single unified model without requiring additional data-intensive training. A central challenge in model merging is to reduce task interference while preserving the task-specific capabilities of the original models. In this work, we propose PRIME, an ultra-low-rank principal-residual model merging framework that decomposes task vector merging into two complementary stages. First, ultra-low-rank principal task vector merging retains only a small fraction of singular vectors, effectively reducing task interference while preserving most of the task-specific performance. Second, orthogonal residual task vector merging incorporates the remaining components by projecting them onto the null space of the principal subspace, thereby avoiding interference while recovering additional task-relevant information. Extensive experiments on eight natural language processing tasks demonstrate that PRIME consistently outperforms existing model merging methods, achieving improvements of up to 1.18{\%} on T5 and 1.9{\%} on LLaMA-3.2-3B."
}