@inproceedings{chen-etal-2025-towards-effective,
    title = "Towards Effective and Efficient Continual Pre-training of Large Language Models",
    author = "Chen, Jie  and
      Chen, Zhipeng  and
      Wang, Jiapeng  and
      Zhou, Kun  and
      Zhu, Yutao  and
      Jiang, Jinhao  and
      Min, Yingqian  and
      Zhao, Xin  and
      Dou, Zhicheng  and
      Mao, Jiaxin  and
      Lin, Yankai  and
      Song, Ruihua  and
      Xu, Jun  and
      Chen, Xu  and
      Yan, Rui  and
      Wei, Zhewei  and
      Hu, Di  and
      Huang, Wenbing  and
      Wen, Ji-Rong",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.289/",
    pages = "5779--5795",
    ISBN = "979-8-89176-251-0",
    abstract = "Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks. In this paper, we comprehensively study its key designs to balance the new abilities while retaining the original abilities, and present an effective CPT method that can greatly improve the Chinese language ability and scientific reasoning ability of LLMs. To achieve it, we design specific data mixture and curriculum strategies based on existing datasets and synthetic high-quality data. Concretely, we synthesize multidisciplinary scientific QA pairs based on related web pages to guarantee the data quality, and also devise the performance tracking and data mixture adjustment strategy to ensure the training stability. For the detailed designs, we conduct preliminary studies on a relatively small model, and summarize the findings to help optimize our CPT method. Extensive experiments on a number of evaluation benchmarks show that our approach can largely improve the performance of Llama-3 (8B), including both the general abilities (+8.81 on C-Eval and +6.31 on CMMLU) and the scientific reasoning abilities (+12.00 on MATH and +4.13 on SciEval). Our model, data, and codes are available at https://github.com/RUC-GSAI/Llama-3-SynE."
}