@inproceedings{wi-park-2026-spectral,
    title = "Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?",
    author = "Wi, Hyowon  and
      Park, Noseong",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1179/",
    pages = "25708--25734",
    ISBN = "979-8-89176-390-6",
    abstract = "In recent years, low-rank adaptation (LoRA) has emerged as a significant paradigm that freezes pre-trained weights and introduces small, learnable adapters instead of fine-tuning the full set of parameters. In this work, we uncover several key insights regarding the $\textit{singular}$ components of network parameters based on Singular Value Decomposition (SVD).Firstly, the $\textit{principal}$ singular components with large singular values in pre-trained network parameters can be effectively reused during fine-tuning, whereas the $\textit{minor}$ components with smaller singular values are more task-specific and require substantial adaptation. Secondly, we first establish the theoretical connection that the uncontrolled growth of singular values in LoRA adapters leads to the forgetting of pre-trained knowledge {---} a well-known issue referred to as $\textit{catastrophic forgetting}$.Building on these observations, we propose $\textbf{SCLoRA}$, which injects parameterized singular components with spectral clipping into the pre-trained model in a way that is aware of the spectral distribution of the pre-trained model. $\textbf{SCLoRA}$ effectively adapts to new tasks by focusing updates on components that require adaptation, while simultaneously alleviating catastrophic forgetting. We conduct extensive experiments and demonstrate that $\textbf{SCLoRA}$ not only improves downstream performance but also effectively retains pre-trained knowledge."
}