Xiaodong Chen
2025
P2 Law: Scaling Law for Post-Training After Model Pruning
Xiaodong Chen
|
Yuxuan Hu
|
Xiaokang Zhang
|
Yanling Wang
|
Cuiping Li
|
Hong Chen
|
Jing Zhang
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Pruning has become a widely adopted technique for reducing the hardware requirements of large language models (LLMs). To recover model performance after pruning, post-training is commonly employed to mitigate the resulting performance degradation. While post-training benefits from larger datasets, once the dataset size is already substantial, increasing the training data provides only limited performance gains. To balance post-training cost and model performance, it is necessary to explore the optimal amount of post-training data. Through extensive experiments on the Llama-3 and Qwen-2.5 series models, pruned using various common pruning methods, we uncover the scaling Law for Post-training after model Pruning, referred to as the P2 Law. This law identifies four key factors for predicting the pruned model’s post-training loss: the model size before pruning, the number of post-training tokens, the pruning rate, and the model’s loss before pruning. Moreover, P2 Law can generalize to larger dataset sizes, larger model sizes, and higher pruning rates, offering valuable insights for the post-training of pruned LLMs.
2024
SP3: Enhancing Structured Pruning via PCA Projection
Yuxuan Hu
|
Jing Zhang
|
Zhe Zhao
|
Chen Zhao
|
Xiaodong Chen
|
Cuiping Li
|
Hong Chen
Findings of the Association for Computational Linguistics: ACL 2024