Abstract
In this work, we study the language model backbone replacement problem for personalized downstream tasks in a non-stationary on-device scenario. In real world, company may periodically update the knowledge and architectures of backbones to keep the competitive in the market, meanwhile, to accommodate the users’ own preference, models are personalized to fit users’ own distribution locally. Traditional full model tuning or transfer learning for such replacements often incur considerable local device training costs and necessitate extensive backpropagation within deep transformer layers. Addressing this issue, we propose a novel, lightweight tuning method for personalized NLP classification tasks post-backbone replacement. Our approach leverages a personalized matrix calculated from documents corresponding to users’ old and new backbones. This matrix facilitates top-layer parameter tuning, drastically reducing backpropagation computation. To further mitigate training costs associated with matrix linear optimization, we employ correlation clustering to curate a few examples from personalized cluster sets for individuals. Our method achieves over 1000 times computation reduction in Flops for backpropagation and brings the user-specific initialization for personal matrix yielding significant performance boost compared with popular transfer learning methods.- Anthology ID:
- 2023.findings-emnlp.323
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4874–4885
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.323
- DOI:
- 10.18653/v1/2023.findings-emnlp.323
- Cite (ACL):
- Shuo Huang, Lizhen Qu, Xingliang Yuan, and Chunyang Chen. 2023. FaLA: Fast Linear Adaptation for Replacing Backbone Models on Edge Devices. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4874–4885, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- FaLA: Fast Linear Adaptation for Replacing Backbone Models on Edge Devices (Huang et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.findings-emnlp.323.pdf