Abstract
Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100 times more data. We open-source our pretrained models and code.- Anthology ID:
- P18-1031
- Volume:
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- July
- Year:
- 2018
- Address:
- Melbourne, Australia
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 328–339
- Language:
- URL:
- https://aclanthology.org/P18-1031
- DOI:
- 10.18653/v1/P18-1031
- Cite (ACL):
- Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
- Cite (Informal):
- Universal Language Model Fine-tuning for Text Classification (Howard & Ruder, ACL 2018)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/P18-1031.pdf
- Code
- fastai/fastai + additional community code
- Data
- AG News, DBpedia, IMDb Movie Reviews, WikiText-103, WikiText-2