Extreme Fine-tuning: A Novel and Fast Fine-tuning Approach for Text Classification
Boonnithi Jiaramaneepinit, Thodsaporn Chay-intr, Kotaro Funakoshi, Manabu Okumura
Abstract
Although fine-tuning a pre-trained model with a conventional approach has shown to be effective in various downstream tasks, previous work has used only backpropagation to fine-tune the model, which causes a massive amount of computational resources and time. We propose Extreme Fine-Tuning (EFT), a novel approach for fine-tuning a pre-trained model effectively and efficiently. EFT uses backpropagation for a brief fine-tuning and an iterative extreme learning machine for training a classifier. We applied EFT to four text classification datasets, MELD, IEMOCAP, IMDb, and AG News, and compared its performance with state-of-the-art (SOTA) approaches. The results indicate that EFT noticeably outperformed the other approaches in training-time measurement with comparable model performance. We will release our code at https://github.com/up-33/extreme-fine-tuning.- Anthology ID:
- 2024.eacl-short.32
- Volume:
- Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- March
- Year:
- 2024
- Address:
- St. Julian’s, Malta
- Editors:
- Yvette Graham, Matthew Purver
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 368–379
- Language:
- URL:
- https://aclanthology.org/2024.eacl-short.32
- DOI:
- Cite (ACL):
- Boonnithi Jiaramaneepinit, Thodsaporn Chay-intr, Kotaro Funakoshi, and Manabu Okumura. 2024. Extreme Fine-tuning: A Novel and Fast Fine-tuning Approach for Text Classification. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 368–379, St. Julian’s, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Extreme Fine-tuning: A Novel and Fast Fine-tuning Approach for Text Classification (Jiaramaneepinit et al., EACL 2024)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2024.eacl-short.32.pdf