@inproceedings{zhang-etal-2026-beyond,
    title = "Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning",
    author = "Zhang, Yang  and
      Mohamed, Amr  and
      Abdine, Hadi  and
      Shang, Guokan  and
      Vazirgiannis, Michalis",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.271/",
    pages = "5776--5794",
    ISBN = "979-8-89176-380-7",
    abstract = "Curriculum learning{---}organizing training data from easy to hard{---}has improved efficiency across machine learning domains, yet remains underexplored for language model pretraining. We present the first systematic investigation of curriculum learning in LLM pretraining, with over 200 models trained on up to 100B tokens across three strategies: vanilla curriculum learning, pacing-based sampling, and interleaved curricula, guided by six difficulty metrics spanning linguistic and information-theoretic properties. We evaluate performance on eight benchmarks under three realistic scenarios: limited data, unlimited data, and continual training. Our experiments show that curriculum learning consistently accelerates convergence in early and mid-training phases, reducing training steps by 18-45{\%} to reach baseline performance. When applied as a warmup strategy before standard random sampling, curriculum learning yields sustained improvements up to 3.5{\%}. We identify compression ratio, lexical diversity (MTLD), and readability (Flesch Reading Ease) as the most effective difficulty signals. Our findings demonstrate that data ordering{---}orthogonal to existing data selection methods{---}provides a practical mechanism for more efficient LLM pretraining."
}