mALBERT: Is a Compact Multilingual BERT Model Still Worth It?

Christophe Servan, Sahar Ghannay, Sophie Rosset

Abstract
Within the current trend of Pretained Language Models (PLM), emerge more and more criticisms about the ethical and ecological impact of such models. In this article, considering these critical remarks, we propose to focus on smaller models, such as compact models like ALBERT, which are more ecologically virtuous than these PLM. However, PLMs enable huge breakthroughs in Natural Language Processing tasks, such as Spoken and Natural Language Understanding, classification, Question–Answering tasks. PLMs also have the advantage of being multilingual, and, as far as we know, a multilingual version of compact ALBERT models does not exist. Considering these facts, we propose the free release of the first version of a multilingual compact ALBERT model, pre-trained using Wikipedia data, which complies with the ethical aspect of such a language model. We also evaluate the model against classical multilingual PLMs in classical NLP tasks. Finally, this paper proposes a rare study on the subword tokenization impact on language performances.
Anthology ID:
2024.lrec-main.960
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
11023–11029
Language:
URL:
https://preview.aclanthology.org/add_missing_videos/2024.lrec-main.960/
DOI:
Bibkey:
Cite (ACL):
Christophe Servan, Sahar Ghannay, and Sophie Rosset. 2024. mALBERT: Is a Compact Multilingual BERT Model Still Worth It?. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 11023–11029, Torino, Italia. ELRA and ICCL.
Cite (Informal):
mALBERT: Is a Compact Multilingual BERT Model Still Worth It? (Servan et al., LREC-COLING 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/add_missing_videos/2024.lrec-main.960.pdf
PDF Search Fix data