Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training
Haode Zhang, Haowen Liang, Liming Zhan, Albert Y.S. Lam, Xiao-Ming Wu
Abstract
We consider the task of few-shot intent detection, which involves training a deep learning model to classify utterances based on their underlying intents using only a small amount of labeled data. The current approach to address this problem is through continual pre-training, i.e., fine-tuning pre-trained language models (PLMs) on external resources (e.g., conversational corpora, public intent detection datasets, or natural language understanding datasets) before using them as utterance encoders for training an intent classifier. In this paper, we show that continual pre-training may not be essential, since the overfitting problem of PLMs on this task may not be as serious as expected. Specifically, we find that directly fine-tuning PLMs on only a handful of labeled examples already yields decent results compared to methods that employ continual pre-training, and the performance gap diminishes rapidly as the number of labeled data increases. To maximize the utilization of the limited available data, we propose a context augmentation method and leverage sequential self-distillation to boost performance. Comprehensive experiments on real-world benchmarks show that given only two or more labeled samples per class, direct fine-tuning outperforms many strong baselines that utilize external data sources for continual pre-training. The code can be found at https://github.com/hdzhang-code/DFTPlus.- Anthology ID:
- 2023.findings-acl.706
- Original:
- 2023.findings-acl.706v1
- Version 2:
- 2023.findings-acl.706v2
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2023
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 11105–11121
- Language:
- URL:
- https://aclanthology.org/2023.findings-acl.706
- DOI:
- 10.18653/v1/2023.findings-acl.706
- Cite (ACL):
- Haode Zhang, Haowen Liang, Liming Zhan, Albert Y.S. Lam, and Xiao-Ming Wu. 2023. Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11105–11121, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training (Zhang et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2023.findings-acl.706.pdf