Efficiently Distilling LLMs for Edge Applications
Achintya Kundu, Yu Chin Fabian Lim, Aaron Chew, Laura Wynter, Penny Chong, Rhui Lee
Abstract
Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time.- Anthology ID:
- 2024.naacl-industry.5
- Volume:
- Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Yi Yang, Aida Davani, Avi Sil, Anoop Kumar
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 52–62
- Language:
- URL:
- https://aclanthology.org/2024.naacl-industry.5
- DOI:
- Cite (ACL):
- Achintya Kundu, Yu Chin Fabian Lim, Aaron Chew, Laura Wynter, Penny Chong, and Rhui Lee. 2024. Efficiently Distilling LLMs for Edge Applications. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages 52–62, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- Efficiently Distilling LLMs for Edge Applications (Kundu et al., NAACL 2024)
- PDF:
- https://preview.aclanthology.org/bionlp-24-ingestion/2024.naacl-industry.5.pdf