An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference

Atsuki Yamaguchi, Aline Villavicencio, Nikolaos Aletras

Abstract
The development of state-of-the-art generative large language models (LLMs) disproportionately relies on English-centric tokenizers, vocabulary and pre-training data. Despite the fact that some LLMs have multilingual capabilities, recent studies have shown that their inference efficiency deteriorates when generating text in languages other than English. This results in increased inference time and costs. Cross-lingual vocabulary adaptation (CVA) methods have been proposed for adapting models to a target language aiming to improve downstream performance. However, the effectiveness of these methods on increasing inference efficiency of generative LLMs has yet to be explored. In this paper, we perform an empirical study of five CVA methods on four generative LLMs (including monolingual and multilingual models) across four typologically-diverse languages and four natural language understanding tasks. We find that CVA substantially contributes to LLM inference speedups of up to 271.5%. We also show that adapting LLMs that have been pre-trained on more balanced multilingual data results in downstream performance comparable to the original models.
Anthology ID:
2024.findings-emnlp.396
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2024
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6760–6785
Language:
URL:
https://preview.aclanthology.org/icon-24-ingestion/2024.findings-emnlp.396/
DOI:
10.18653/v1/2024.findings-emnlp.396
Bibkey:
Cite (ACL):
Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2024. An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6760–6785, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference (Yamaguchi et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/icon-24-ingestion/2024.findings-emnlp.396.pdf
PDF Search Fix data