Abstract
A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer by updating only a small set of additional parameters while keeping the parameters of the original model frozen. While proven to be an effective approach, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach calls for complementary measures after pre-training and before fine-tuning. In this work, we show that taking the ultimate choice of fine-tuning into consideration boosts the performance of parameter-efficient fine-tuning. By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pre-trained model specifically for parameter-efficient fine-tuning, resulting in gains of up to 4.96 points on cross-lingual NER fine-tuning. Our ablation settings and analyses further reveal that the specific approach we take to meta-learning is crucial for the attained gains.- Anthology ID:
- 2023.findings-acl.737
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2023
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 11602–11612
- Language:
- URL:
- https://aclanthology.org/2023.findings-acl.737
- DOI:
- 10.18653/v1/2023.findings-acl.737
- Cite (ACL):
- Mozhdeh Gheini, Xuezhe Ma, and Jonathan May. 2023. Know Where You’re Going: Meta-Learning for Parameter-Efficient Fine-Tuning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11602–11612, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Know Where You’re Going: Meta-Learning for Parameter-Efficient Fine-Tuning (Gheini et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2023.findings-acl.737.pdf