Parameter-Efficient Tuning Makes a Good Classification Head

Zhuoyi Yang, Ming Ding, Yanhui Guo, Qingsong Lv, Jie Tang


Abstract
In recent years, pretrained models revolutionized the paradigm of natural language understanding (NLU), where we append a randomly initialized classification head after the pretrained backbone, e.g. BERT, and finetune the whole model. As the pretrained backbone makes a major contribution to the improvement, we naturally expect a good pretrained classification head can also benefit the training. However, the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning, making the usual head-only pretraining ineffective. In this paper, we find that parameter-efficient tuning makes a good classification head, with which we can simply replace the randomly initialized heads for a stable performance gain. Our experiments demonstrate that the classification head jointly pretrained with parameter-efficient tuning consistently improves the performance on 9 tasks in GLUE and SuperGLUE.
Anthology ID:
2022.emnlp-main.514
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7576–7586
Language:
URL:
https://aclanthology.org/2022.emnlp-main.514
DOI:
Bibkey:
Cite (ACL):
Zhuoyi Yang, Ming Ding, Yanhui Guo, Qingsong Lv, and Jie Tang. 2022. Parameter-Efficient Tuning Makes a Good Classification Head. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7576–7586, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Parameter-Efficient Tuning Makes a Good Classification Head (Yang et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.emnlp-main.514.pdf