@inproceedings{yang-etal-2026-curated,
    title = "From Curated Data to Scalable Models: Continual Pre-training of Dense and {M}o{E} Large Language Models for {T}ibetan",
    author = "Yang, Lei  and
      Pan, Leiyu  and
      Xiong, Bojian  and
      Jin, Renren  and
      Zhang, Shaowei  and
      Chen, Yue  and
      Shi, Ling  and
      Zhou, Jiang  and
      Wu, Junru  and
      Wang, Zhen  and
      Peng, Jianxiang  and
      Xiao, Juesi  and
      Dong, Tianyu  and
      Han, Zhuowen  and
      Chen, Zhuo  and
      Ren, Yuqi  and
      Xiong, Deyi",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1866/",
    pages = "40167--40182",
    ISBN = "979-8-89176-390-6",
    abstract = "Large language models (LLMs) have achieved remarkable success across a wide range of natural language processing tasks, yet their performance remains heavily biased toward high-resource languages. Tibetan, despite its cultural significance and large speaker population, is still substantially underrepresented. In this work, we present a comprehensive pipeline for advancing Tibetan language modeling through large-scale data curation and continual pre-training. We construct a 72 GB high-quality Tibetan corpus, the largest to date, and adapt Qwen2.5-7B through balanced multilingual continual pre-training with Tibetan, Chinese, and English, followed by multilingual instruction tuning. To further scale capacity efficiently, we extend the dense model to a 50B-A10B Mixture-of-Experts architecture. Due to the absence of standardized Tibetan benchmarks, we build multiple evaluation datasets via high-quality translation and human verification. Experimental results show that both dense and MoE models consistently outperform existing open-source and Tibetan-focused models of similar scale across diverse tasks. Our work advances Tibetan-centric LLM research and provides transferable insights for extending LLMs to other low-resource languages. We will release the model weights, evaluation benchmarks, and detailed data processing documentation in the follow-up."
}