Distilling the Knowledge of Romanian BERTs Using Multiple Teachers

Andrei-Marius Avram, Darius Catrina, Dumitru-Clementin Cercel, Mihai Dascalu, Traian Rebedea, Vasile Pais, Dan Tufis


Abstract
Running large-scale pre-trained language models in computationally constrained environments remains a challenging problem yet to be addressed, while transfer learning from these models has become prevalent in Natural Language Processing tasks. Several solutions, including knowledge distillation, network quantization, or network pruning have been previously proposed; however, these approaches focus mostly on the English language, thus widening the gap when considering low-resource languages. In this work, we introduce three light and fast versions of distilled BERT models for the Romanian language: Distil-BERT-base-ro, Distil-RoBERT-base, and DistilMulti-BERT-base-ro. The first two models resulted from the individual distillation of knowledge from two base versions of Romanian BERTs available in literature, while the last one was obtained by distilling their ensemble. To our knowledge, this is the first attempt to create publicly available Romanian distilled BERT models, which were thoroughly evaluated on five tasks: part-of-speech tagging, named entity recognition, sentiment analysis, semantic textual similarity, and dialect identification. Our experimental results argue that the three distilled models offer performance comparable to their teachers, while being twice as fast on a GPU and ~35% smaller. In addition, we further test the similarity between the predictions of our students versus their teachers by measuring their label and probability loyalty, together with regression loyalty - a new metric introduced in this work.
Anthology ID:
2022.lrec-1.39
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
374–384
Language:
URL:
https://aclanthology.org/2022.lrec-1.39
DOI:
Bibkey:
Cite (ACL):
Andrei-Marius Avram, Darius Catrina, Dumitru-Clementin Cercel, Mihai Dascalu, Traian Rebedea, Vasile Pais, and Dan Tufis. 2022. Distilling the Knowledge of Romanian BERTs Using Multiple Teachers. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 374–384, Marseille, France. European Language Resources Association.
Cite (Informal):
Distilling the Knowledge of Romanian BERTs Using Multiple Teachers (Avram et al., LREC 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/author-url/2022.lrec-1.39.pdf
Code
 racai-ai/romanian-distilbert
Data
GLUEMOROCORONEC