@inproceedings{chen-etal-2025-p2,
    title = "P$^2$ Law: Scaling Law for Post-Training After Model Pruning",
    author = "Chen, Xiaodong  and
      Hu, Yuxuan  and
      Zhang, Xiaokang  and
      Wang, Yanling  and
      Li, Cuiping  and
      Chen, Hong  and
      Zhang, Jing",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.283/",
    pages = "5668--5686",
    ISBN = "979-8-89176-251-0",
    abstract = "Pruning has become a widely adopted technique for reducing the hardware requirements of large language models (LLMs). To recover model performance after pruning, post-training is commonly employed to mitigate the resulting performance degradation. While post-training benefits from larger datasets, once the dataset size is already substantial, increasing the training data provides only limited performance gains. To balance post-training cost and model performance, it is necessary to explore the optimal amount of post-training data. Through extensive experiments on the Llama-3 and Qwen-2.5 series models, pruned using various common pruning methods, we uncover the scaling \textbf{Law} for \textbf{P}ost-training after model \textbf{P}runing, referred to as the P$^2$ Law. This law identifies four key factors for predicting the pruned model{'}s post-training loss: the model size before pruning, the number of post-training tokens, the pruning rate, and the model{'}s loss before pruning. Moreover, P$^2$ Law can generalize to larger dataset sizes, larger model sizes, and higher pruning rates, offering valuable insights for the post-training of pruned LLMs."
}