Abstract
Recently, language models have demonstrated exceptional performance compared to their predecessors. In this context, attention mechanisms and pre-training significantly contribute to the enhanced performance of modern language models. Additionally, a continuously increasing number of parameters plays a crucial role in these advancements . However, an increase in the number of parameters significantly increases the GPU memory and training time required during fine-tuning of language models, this makes fine-tuning infeasible in environments with limited computing resources. Furthermore, after fine-tuning, the storage space required for deployment increases proportionally with the number of tasks, making it challenging to deploy devices with limited storage capacities. In this study, we propose IT-Tuning, a Parameter Efficient Fine-Tuning method that introduces a new concept called information tokens to address these issues.- Anthology ID:
- 2024.repl4nlp-1.6
- Volume:
- Proceedings of the 9th Workshop on Representation Learning for NLP (RepL4NLP-2024)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Chen Zhao, Marius Mosbach, Pepa Atanasova, Seraphina Goldfarb-Tarrent, Peter Hase, Arian Hosseini, Maha Elbayad, Sandro Pezzelle, Maximilian Mozes
- Venues:
- RepL4NLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 58–68
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2024.repl4nlp-1.6/
- DOI:
- Cite (ACL):
- Jungu Kim and Hyeoncheol Kim. 2024. IT-Tuning : Parameter Efficient Information Token Tuning for Language Model. In Proceedings of the 9th Workshop on Representation Learning for NLP (RepL4NLP-2024), pages 58–68, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- IT-Tuning : Parameter Efficient Information Token Tuning for Language Model (Kim & Kim, RepL4NLP 2024)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2024.repl4nlp-1.6.pdf