Is Prompt Transfer Always Effective? An Empirical Study of Prompt Transfer for Question Answering

Minji Jung, Soyeon Park, Jeewoo Sul, Yong Suk Choi

Abstract
Prompt tuning, which freezes all parameters of a pre-trained model and only trains a soft prompt, has emerged as a parameter-efficient approach. For the reason that the prompt initialization becomes sensitive when the model size is small, the prompt transfer that uses the trained prompt as an initialization for the target task has recently been introduced. Since previous works have compared tasks in large categories (e.g., summarization, sentiment analysis), the factors that influence prompt transfer have not been sufficiently explored. In this paper, we characterize the question answering task based on features such as answer format and empirically investigate the transferability of soft prompts for the first time. We analyze the impact of initialization during prompt transfer and find that the train dataset size of source and target tasks have the influence significantly. Furthermore, we propose a novel approach for measuring catastrophic forgetting and investigate how it occurs in terms of the amount of evidence. Our findings can help deeply understand transfer learning in prompt tuning.
Anthology ID:
2024.naacl-short.44
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
528–539
Language:
URL:
https://aclanthology.org/2024.naacl-short.44
DOI:
Bibkey:
Cite (ACL):
Minji Jung, Soyeon Park, Jeewoo Sul, and Yong Suk Choi. 2024. Is Prompt Transfer Always Effective? An Empirical Study of Prompt Transfer for Question Answering. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 528–539, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Is Prompt Transfer Always Effective? An Empirical Study of Prompt Transfer for Question Answering (Jung et al., NAACL 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/retraction/2024.naacl-short.44.pdf
PDF Search