@inproceedings{smith-etal-2025-flexigpt,
    title = "{F}lexi{GPT}: Pruning and Extending Large Language Models with Low-Rank Weight Sharing",
    author = "Smith, James Seale  and
      Lin, Chi-Heng  and
      Tuli, Shikhar  and
      Jeelani, Haris  and
      Gao, Shangqian  and
      Shen, Yilin  and
      Jin, Hongxia  and
      Hsu, Yen-Chang",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.31/",
    pages = "718--730",
    ISBN = "979-8-89176-189-6",
    abstract = "The rapid proliferation of large language models (LLMs) in natural language processing (NLP) has created a critical need for techniques that enable efficient deployment on memory-constrained devices without compromising performance. We present a method to prune LLMs that selectively prunes model blocks based on an importance score and replaces them with a low-parameter replacement strategy. Specifically, we propose a principled metric to replace each pruned block using a weight-sharing mechanism that leverages unpruned counterparts from the model and block-specific low-rank adapters. Furthermore, we facilitate the learning of these replacement blocks with output feature normalization and an adapter initialization scheme built on low-rank SVD reconstructions. Empirical evaluations demonstrate substantial performance gains over existing methods, achieving state-of-the-art performance on 5/6 benchmarks for a compression rate of 30{\%} and 6/6 benchmarks for a compression rate of 40{\%}. We also demonstrate that our approach can extend smaller models, boosting performance on 6/6 benchmarks using only {\textasciitilde}0.3{\%} tokens of extended training with minimal additional parameter costs."
}