Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding
Rishabh Bhardwaj, Amrita Saha, Steven C.H. Hoi, Soujanya Poria
Abstract
Prompt Tuning has been largely successful as a parameter-efficient method of conditioning large-scale pre-trained language models to perform downstream tasks. Thus far, soft prompt tuning learns a fixed set of task-specific continuous vectors, i.e., soft tokens that remain static across the task samples. A fixed prompt, however, may not generalize well to the diverse kinds of inputs the task comprises. In order to address this, we propose Vector-quantized Input-contextualized Prompts (VIP) as an extension to the soft prompt tuning framework. VIP particularly focuses on two aspects—contextual prompts that learns input-specific contextualization of the soft prompt tokens through a small-scale sentence encoder and quantized prompts that maps the contextualized prompts to a set of learnable codebook vectors through a Vector quantization network. On various language understanding tasks like SuperGLUE, QA, Relation classification, NER and NLI, VIP outperforms the soft prompt tuning (PT) baseline by an average margin of 1.19%. Further, our generalization studies show that VIP learns more robust prompt representations, surpassing PT by a margin of 0.6% - 5.3% on Out-of-domain QA and NLI tasks respectively, and by 0.75% on Multi-Task setup over 4 tasks spanning across 12 domains.- Anthology ID:
- 2022.emnlp-main.455
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6776–6791
- Language:
- URL:
- https://aclanthology.org/2022.emnlp-main.455
- DOI:
- 10.18653/v1/2022.emnlp-main.455
- Cite (ACL):
- Rishabh Bhardwaj, Amrita Saha, Steven C.H. Hoi, and Soujanya Poria. 2022. Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6776–6791, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding (Bhardwaj et al., EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2022.emnlp-main.455.pdf