@inproceedings{yiwen-etal-2025-yulan,
    title = "{Y}u{L}an-Mini: Pushing the Limits of Open Data-efficient Language Model",
    author = "Yiwen, Hu  and
      Song, Huatong  and
      Chen, Jie  and
      Deng, Jia  and
      Wang, Jiapeng  and
      Zhou, Kun  and
      Zhu, Yutao  and
      Jiang, Jinhao  and
      Dong, Zican  and
      Lu, Yang  and
      Miao, Xu  and
      Zhao, Xin  and
      Wen, Ji-Rong",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.268/",
    pages = "5374--5400",
    ISBN = "979-8-89176-251-0",
    abstract = "Due to the immense resource demands and the involved complex techniques, it is still challenging for successfully pre-training a large language models (LLMs) with state-of-the-art performance. In this paper, we explore the key bottlenecks and designs during pre-training, and make the following contributions: (1) a comprehensive investigation into the factors contributing to training instability; (2) a robust optimization approach designed to mitigate training instability effectively; (3) an elaborate data pipeline that integrates data synthesis, data curriculum, and data selection. By integrating the above techniques, we create a rather low-cost training recipe and use it to pre-train YuLan-Mini, a fully-open base model with 2.4B parameters on 1.08T tokens. Remarkably, YuLan-Mini achieves top-tier performance among models of similar parameter scale, with comparable performance to industry-leading models that require significantly more data. To facilitate reproduction, we release the full details of training recipe and data composition. Project details can be accessed at the following link: https://anonymous.4open.science/r/YuLan-Mini/README.md."
}