It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

Timo Schick, Hinrich Schütze


Abstract
When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much “greener” in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.
Anthology ID:
2021.naacl-main.185
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2339–2352
Language:
URL:
https://aclanthology.org/2021.naacl-main.185
DOI:
10.18653/v1/2021.naacl-main.185
Award:
 Outstanding Long Paper
Bibkey:
Cite (ACL):
Timo Schick and Hinrich Schütze. 2021. It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Association for Computational Linguistics.
Cite (Informal):
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (Schick & Schütze, NAACL 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2021.naacl-main.185.pdf
Video:
 https://preview.aclanthology.org/emnlp-22-attachments/2021.naacl-main.185.mp4
Code
 timoschick/pet +  additional community code
Data
FewGlueBoolQCOPAMultiNLIMultiRCReCoRDSuperGLUEWSCWiC