@inproceedings{zhu-etal-2025-second,
    title = "Second Language ({A}rabic) Acquisition of {LLM}s via Progressive Vocabulary Expansion",
    author = "Zhu, Jianqing  and
      Huang, Huang  and
      Lin, Zhihang  and
      Liang, Juhao  and
      Tang, Zhengyang  and
      Almubarak, Khalid  and
      Alharthi, Mosen  and
      An, Bang  and
      He, Juncai  and
      Wu, Xiangbo  and
      Yu, Fei  and
      Chen, Junying  and
      Zhuoheng, Ma  and
      Du, Yuhao  and
      Zhang, He  and
      Alshahrani, Saied  and
      Alghamdi, Emad A.  and
      Zhang, Lian  and
      Sun, Ruoyu  and
      Li, Haizhou  and
      Wang, Benyou  and
      Xu, Jinchao",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.100/",
    pages = "2025--2042",
    ISBN = "979-8-89176-251-0",
    abstract = "This paper addresses the critical need for democratizing large language models (LLM) in the Arab world, a region that has seen slower progress in developing models comparable to state-of-the-art offerings like GPT-4 or GPT-3.5, due to a predominant focus on mainstream languages (e.g., English and Chinese). One practical objective for Arabic LLMs is to utilize Arabic-specific vocabulary in the tokenizer to accelerate decoding. However, using a different vocabulary often leads to degradation of the model{'}s learned knowledge, since many words become out-of-vocabulary (OOV) at the beginning of training. Inspired by the vocabulary learning during Second Language (Arabic) Acquisition for humans, the released AraLLaMA employs progressive vocabulary expansion, which is implemented by a modified BPE algorithm that progressively extends the Arabic subwords in its dynamic vocabulary during training, thereby balancing the OOV ratio at every stage. The ablation study demonstrated the effectiveness of Progressive Vocabulary Expansion.Moreover, AraLLaMA achieves decent performance comparable to the best Arabic LLMs across a variety of Arabic benchmarks. Our model weights are available at: \url{https://github.com/FreedomIntelligence/AraLLaMa}."
}