Gradual Fine-Tuning for Low-Resource Domain Adaptation
Haoran Xu, Seth Ebner, Mahsa Yarmohammadi, Aaron Steven White, Benjamin Van Durme, Kenton Murray
Abstract
Fine-tuning is known to improve NLP models by adapting an initial model trained on more plentiful but less domain-salient examples to data in a target domain. Such domain adaptation is typically done using one stage of fine-tuning. We demonstrate that gradually fine-tuning in a multi-step process can yield substantial further gains and can be applied without modifying the model or learning objective.- Anthology ID:
- 2021.adaptnlp-1.22
- Volume:
- Proceedings of the Second Workshop on Domain Adaptation for NLP
- Month:
- April
- Year:
- 2021
- Address:
- Kyiv, Ukraine
- Venue:
- AdaptNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 214–221
- Language:
- URL:
- https://aclanthology.org/2021.adaptnlp-1.22
- DOI:
- Cite (ACL):
- Haoran Xu, Seth Ebner, Mahsa Yarmohammadi, Aaron Steven White, Benjamin Van Durme, and Kenton Murray. 2021. Gradual Fine-Tuning for Low-Resource Domain Adaptation. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 214–221, Kyiv, Ukraine. Association for Computational Linguistics.
- Cite (Informal):
- Gradual Fine-Tuning for Low-Resource Domain Adaptation (Xu et al., AdaptNLP 2021)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2021.adaptnlp-1.22.pdf
- Code
- fe1ixxu/Gradual-Finetune