Abstract
Recent advancements in large language models (LLMs) have raised concerns about inference costs, increasing the need for research into model compression. While knowledge distillation (KD) is a prominent method for this, research on KD for generative language models like LLMs is relatively sparse, and the approach of distilling student-friendly knowledge, which has shown promising performance in KD for classification models, remains unexplored in generative language models. To explore this approach, we propose PromptKD, a simple yet effective method that utilizes prompt tuning - for the first time in KD - to enable generative language models to transfer student-friendly knowledge. Unlike previous works in classification that require fine-tuning the entire teacher model for extracting student-friendly knowledge, PromptKD achieves similar effects by adding a small number of prompt tokens and tuning only the prompt with student guidance. Extensive experiments on instruction-following datasets show that PromptKD achieves state-of-the-art performance while adding only 0.0007% of the teacher’s parameters as prompts. Further analysis suggests that distilling student-friendly knowledge alleviates exposure bias effectively throughout the entire training process, leading to performance enhancements.- Anthology ID:
- 2024.findings-emnlp.364
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2024
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6266–6282
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2024.findings-emnlp.364/
- DOI:
- 10.18653/v1/2024.findings-emnlp.364
- Cite (ACL):
- Gyeongman Kim, Doohyuk Jang, and Eunho Yang. 2024. PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6266–6282, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning (Kim et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2024.findings-emnlp.364.pdf