AdaPrompt: Adaptive Model Training for Prompt-based NLP
Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, Yue Zhang
Abstract
Prompt-based learning, with its capability to tackle zero-shot and few-shot NLP tasks, has gained much attention in the community.The main idea is to bridge the gap between NLP downstream tasks and language modeling (LM), by mapping these tasks into natural language prompts, which are then filled by pre-trained language models (PLMs).However, for prompt learning, there are still two salient gaps between NLP tasks and pretraining.First, prompt information is not necessarily sufficiently present during LM pre-training. Second, task-specific data are not necessarily well represented during pre-training. We address these two issues by proposing AdaPrompt, adaptively retrieving external data for continual pretraining of PLMs by making use of both task and prompt characteristics. In addition, we make use of knowledge in Natural Language Inference models for deriving adaptive verbalizers.Experimental results on five NLP benchmarks show that AdaPrompt can improve over standard PLMs in few-shot settings. In addition, in zero-shot settings, our method outperforms standard prompt-based methods by up to 26.35% relative error reduction.- Anthology ID:
- 2022.findings-emnlp.448
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6057–6068
- Language:
- URL:
- https://aclanthology.org/2022.findings-emnlp.448
- DOI:
- 10.18653/v1/2022.findings-emnlp.448
- Cite (ACL):
- Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, and Yue Zhang. 2022. AdaPrompt: Adaptive Model Training for Prompt-based NLP. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6057–6068, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- AdaPrompt: Adaptive Model Training for Prompt-based NLP (Chen et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2022.findings-emnlp.448.pdf