@inproceedings{li-etal-2026-two,
    title = "Two-Stage Parameter Alignment for Multi-{L}o{RA} Merging in Large Language Models",
    author = "Li, Zijian  and
      Feng, Xiachong  and
      Ma, Weitao  and
      Huang, Yichong  and
      Feng, Xiaocheng  and
      Qin, Bing",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1504/",
    pages = "30079--30093",
    ISBN = "979-8-89176-395-1",
    abstract = "Merging a large number of low-rank adaptations (LoRAs) is a key technology for enhancing the integration and deployment efficiency of large language models (LLMs). However, current general model merging methods are prone to ``parameter interference'' problem, and this issue is especially pronounced when merging high-rank LoRAs, where parameter conflicts tend to be more severe. While the classical rotation alignment approach can enhance robustness, it is difficult to apply due to incompatibility with the LoRA structure and its high computational complexity. To address these challenges, we propose a novel two-stage parameter alignment (TSPA) framework. TSPA is designed from the perspective of the LoRA architecture, overcoming the limitations of existing methods and reducing the computational complexity from quadratic to linear. We conduct experiments on Natural Language Processing (NLP) tasks using models such as Llama-3-8B. The results show that the two-stage design of TSPA achieves a balance between task capabilities and general knowledge. It exhibits greater robustness than other methods in high-rank and high-interference scenarios, while effectively preserving fine-grained functions."
}