@inproceedings{cho-etal-2025-assigning,
    title = "Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition",
    author = "Cho, Yoonjun  and
      Kim, Soeun  and
      Jeon, Dongjae  and
      Lee, Kyelim  and
      Lee, Beomsoo  and
      No, Albert",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/watermark/2025.findings-acl.746/",
    doi = "10.18653/v1/2025.findings-acl.746",
    pages = "14453--14470",
    ISBN = "979-8-89176-256-5",
    abstract = "Decomposing weight matrices into quantization and low-rank components ($\bf W\approx Q+LR$) is a widely used technique for compressing large language models (LLMs). Existing joint optimization methods iteratively alternate between quantization and low-rank approximation. However, these methods tend to prioritize one component at the expense of the other, resulting in suboptimal decompositions that fail to leverage each component{'}s unique strengths. In this work, we introduce Outlier-Driven Low-Rank Initialization (ODLRI), which assigns low-rank components the specific role of capturing activation-sensitive weights. This structured decomposition mitigates outliers' negative impact on quantization, enabling more effective balance between quantization and low-rank approximation. Experiments on Llama2 (7B, 13B, 70B), Llama3-8B, and Mistral-7B demonstrate that incorporating ODLRI into the joint optimization framework consistently reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy in low-bit settings."
}