Early Exit with Disentangled Representation and Equiangular Tight Frame

Yixin Ji, Jikai Wang, Juntao Li, Qiang Chen, Wenliang Chen, Min Zhang

Abstract
Dynamic early exit has demonstrated great potential in coping with the sharply increasing number of pre-trained language model parameters, which can achieve a good trade-off between performance and efficiency. The existing early exit paradigm relies on training parametrical internal classifiers at each intermediate layer to complete specific tasks. Based on the predictions of these internal classifiers, different methods are designed to decide when to exit. Under this circumstance, each intermediate layer takes on both generic language representation learning and task-specific feature extraction, which makes each intermediate layer struggle to balance two types of backward loss signals during training. To break this dilemma, we propose an adapter method to decouple the two distinct types of representation and further introduce a non-parametric simplex equiangular tight frame classifier (ETF) for improvement. Extensive experiments on monolingual and multilingual tasks demonstrate that our method gains significant improvements over strong PLM backbones and early exit methods.
Anthology ID:
2023.findings-acl.889
Volume:
Findings of the Association for Computational Linguistics: ACL 2023
Month:
July
Year:
2023
Address:
Toronto, Canada
Editors:
Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
14128–14142
Language:
URL:
https://aclanthology.org/2023.findings-acl.889
DOI:
10.18653/v1/2023.findings-acl.889
Bibkey:
Cite (ACL):
Yixin Ji, Jikai Wang, Juntao Li, Qiang Chen, Wenliang Chen, and Min Zhang. 2023. Early Exit with Disentangled Representation and Equiangular Tight Frame. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14128–14142, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
Early Exit with Disentangled Representation and Equiangular Tight Frame (Ji et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2023.findings-acl.889.pdf
PDF Search