Predictive Model Selection for Transfer Learning in Sequence Labeling Tasks
Parul Awasthy, Bishwaranjan Bhattacharjee, John Kender, Radu Florian
Abstract
Transfer learning is a popular technique to learn a task using less training data and fewer compute resources. However, selecting the correct source model for transfer learning is a challenging task. We demonstrate a novel predictive method that determines which existing source model would minimize error for transfer learning to a given target. This technique does not require learning for prediction, and avoids computational costs of trail-and-error. We have evaluated this technique on nine datasets across diverse domains, including newswire, user forums, air flight booking, cybersecurity news, etc. We show that it per-forms better than existing techniques such as fine-tuning over vanilla BERT, or curriculum learning over the largest dataset on top of BERT, resulting in average F1 score gains in excess of 3%. Moreover, our technique consistently selects the best model using fewer tries.- Anthology ID:
- 2020.sustainlp-1.15
- Volume:
- Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Nafise Sadat Moosavi, Angela Fan, Vered Shwartz, Goran Glavaš, Shafiq Joty, Alex Wang, Thomas Wolf
- Venue:
- sustainlp
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 113–118
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2020.sustainlp-1.15/
- DOI:
- 10.18653/v1/2020.sustainlp-1.15
- Cite (ACL):
- Parul Awasthy, Bishwaranjan Bhattacharjee, John Kender, and Radu Florian. 2020. Predictive Model Selection for Transfer Learning in Sequence Labeling Tasks. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 113–118, Online. Association for Computational Linguistics.
- Cite (Informal):
- Predictive Model Selection for Transfer Learning in Sequence Labeling Tasks (Awasthy et al., sustainlp 2020)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2020.sustainlp-1.15.pdf