Renyi Cai
2026
On the Representation Geometry of LoRA Model Merging
Chenyang Lu | Jiaru Li | Jinman Zhao | Xinran Chen | Yining Wang | Renyi Cai | Yuchen Li | Chao He
Findings of the Association for Computational Linguistics: ACL 2026
Chenyang Lu | Jiaru Li | Jinman Zhao | Xinran Chen | Yining Wang | Renyi Cai | Yuchen Li | Chao He
Findings of the Association for Computational Linguistics: ACL 2026
Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning, yet merging multiple task-specific LoRA updates without additional training remains challenging. Most existing LoRA merging methods rely on SVD-based alignment, which emphasizes globally shared structure across tasks. In this work, we show that LoRA merging performance can be further improved by combining SVD with CUR decomposition. Through a representation-level analysis, we find that SVD-based decompositions primarily model shared components across tasks, while CUR-based decompositions better preserve task-specific and localized updates. These two perspectives are geometrically misaligned and exhibit complementary advantages, revealing an inherent trade-off between capturing shared structure and preserving task-specific information in LoRA model merging. Guided by this analysis, we propose a training-free merging procedure that explicitly combines the shared structure captured by SVD with the task-specific components preserved by CUR. Experiments on both vision and language benchmarks demonstrate consistent improvements over existing gradient-free LoRA merging methods.