@inproceedings{solgi-etal-2025-saten,
    title = "Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models",
    author = "Solgi, Ryan  and
      Zhen, Kai  and
      Swaminathan, Rupak Vignesh  and
      Susanj, Nathan  and
      Mouchtaris, Athanasios  and
      Kunzmann, Siegfried  and
      Zhang, Zheng",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.1287/",
    doi = "10.18653/v1/2025.findings-emnlp.1287",
    pages = "23674--23683",
    ISBN = "979-8-89176-335-7",
    abstract = "The efficient implementation of large language models (LLMs) is crucial for deployment on resource-constrained devices. Low-rank tensor compression techniques, such as tensor-train (TT) networks, have been widely studied for over-parameterized neural networks. However, their applications to compress pre-trained LLMs for downstream tasks (post-training) remains challenging due to the high-rank nature of pre-trained LLMs and the lack of access to pretraining data. In this study, we investigate low-rank tensorized LLMs during fine-tuning and propose sparse augmented tensor networks (Saten) to enhance their performance. The proposed Saten framework enables full model compression. Experimental results demonstrate that Saten enhances both accuracy and compression efficiency in tensorized language models, achieving state-of-the-art performance."
}