@inproceedings{zhelnin-etal-2025-gift,
    title = "{GIFT}-{SW}: {G}aussian noise Injected Fine-Tuning of Salient Weights for {LLM}s",
    author = "Zhelnin, Maxim  and
      Moskvoretskii, Viktor  and
      Shvetsov, Egor  and
      Krylova, Maria  and
      Egor, Venediktov  and
      Aleksandr, Zuev  and
      Burnaev, Evgeny",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.324/",
    pages = "6463--6480",
    ISBN = "979-8-89176-251-0",
    abstract = "Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and democratized the usage of Large Language Models (LLMs). Recent studies have shown that a small subset of weights significantly impacts performance. Based on this observation, we introduce a novel PEFT method, called Gaussian noise Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only salient columns, while injecting Gaussian noise into non-salient ones. To identify these columns, we developed a generalized sensitivity metric that extends and unifies metrics from previous studies. Experiments with LLaMA models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT methods under the same computational budget. Moreover, GIFT-SW offers practical advantages to recover performance of models subjected to mixed-precision quantization with keeping salient weights in full precision."
}