Task-specific Compression for Multi-task Language Models using Attribution-based Pruning
Nakyeong Yang, Yunah Jang, Hwanhee Lee, Seohyeong Jeong, Kyomin Jung
Abstract
Multi-task language models show outstanding performance for various natural language understanding tasks with only a single model.However, these language models inevitably utilize an unnecessarily large number of model parameters, even when used only for a specific task.In this paper, we propose a novel training-free compression method for multi-task language models using pruning method.Specifically, we use an attribution method to determine which neurons are essential for performing a specific task.We task-specifically prune unimportant neurons and leave only task-specific parameters.Furthermore, we extend our method to be applicable in both low-resource and unsupervised settings. Since our compression method is training-free, it uses little computing resources and does not update the pre-trained parameters of language models, reducing storage space usage.Experimental results on the six widely-used datasets show that our proposed pruning method significantly outperforms baseline pruning methods.In addition, we demonstrate that our method preserves performance even in an unseen domain setting.- Anthology ID:
- 2023.findings-eacl.43
- Volume:
- Findings of the Association for Computational Linguistics: EACL 2023
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 582–592
- Language:
- URL:
- https://aclanthology.org/2023.findings-eacl.43
- DOI:
- Cite (ACL):
- Nakyeong Yang, Yunah Jang, Hwanhee Lee, Seohyeong Jeong, and Kyomin Jung. 2023. Task-specific Compression for Multi-task Language Models using Attribution-based Pruning. In Findings of the Association for Computational Linguistics: EACL 2023, pages 582–592, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Task-specific Compression for Multi-task Language Models using Attribution-based Pruning (Yang et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/author-url/2023.findings-eacl.43.pdf