Efficient Multilingual Language Model Compression through Vocabulary Trimming

Asahi Ushio, Yi Zhou, Jose Camacho-Collados

Abstract
Multilingual language models (LMs) have become a powerful tool in NLP, especially for non-English languages. Nevertheless, model parameters of multilingual LMs remain large due to the larger embedding matrix of the vocabulary covering tokens in different languages. Instead, monolingual LMs can be trained in a target language with the language-specific vocabulary only. In this paper, we propose vocabulary-trimming (VT), a method to reduce a multilingual LM vocabulary to a target language by deleting potentially irrelevant tokens from its vocabulary. In theory, VT can compress any existing multilingual LM to any language covered by the original model. In our experiments, we show that VT can retain the original performance of the multilingual LM, while being considerably smaller in size than the original multilingual LM. The evaluation is performed over four NLP tasks (two generative and two classification tasks) among four widely used multilingual LMs in seven languages. The results show that this methodology can keep the best of both monolingual and multilingual worlds by keeping a small size as monolingual models without the need for specifically retraining them, and can even help limit potentially harmful social biases.
Anthology ID:
2023.findings-emnlp.981
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
14725–14739
Language:
URL:
https://preview.aclanthology.org/remove-affiliations/2023.findings-emnlp.981/
DOI:
10.18653/v1/2023.findings-emnlp.981
Bibkey:
Cite (ACL):
Asahi Ushio, Yi Zhou, and Jose Camacho-Collados. 2023. Efficient Multilingual Language Model Compression through Vocabulary Trimming. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14725–14739, Singapore. Association for Computational Linguistics.
Cite (Informal):
Efficient Multilingual Language Model Compression through Vocabulary Trimming (Ushio et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/remove-affiliations/2023.findings-emnlp.981.pdf
PDF Search Fix data