@inproceedings{prakriya-etal-2025-accelerating,
    title = "Accelerating Large Language Model Pretraining via {LFR} Pedagogy: Learn, Focus, and Review",
    author = "Prakriya, Neha  and
      Yen, Jui-Nan  and
      Hsieh, Cho-Jui  and
      Cong, Jason",
    editor = "Boleda, Gemma  and
      Roth, Michael",
    booktitle = "Proceedings of the 29th Conference on Computational Natural Language Learning",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.conll-1.18/",
    pages = "268--290",
    ISBN = "979-8-89176-271-8",
    abstract = "We introduce an effective and scalable data selection technique to accelerate the pretraining of large language models (LLMs). Given the variation in quality and informativeness of web-scale corpora, we present the Learn-Focus-Review (LFR) paradigm-a dynamic training approach that adapts to the model{'}s learning progress. Inspired by human learning techniques like spaced repetition, LFR tracks the model{'}s learning performance across data instances and prioritizes revisiting challenging and diverse regions of the dataset that are more prone to being forgotten, enabling better retention and more efficient learning. Through experiments spanning over 2200 GPU hours, we show that LFR significantly enhances data efficiency in pretraining while improving downstream performance across commonsense reasoning, question answering, problem-solving, language modeling, and translation tasks. LFR consistently achieves lower perplexity and higher accuracy using just 5{\%}{--}19{\%} of the training tokens as models trained on the full dataset. Notably, LFR matches the performance of industry-standard Pythia models with up to 2$\times$ the parameter count while requiring only 3.2{\%} of the training tokens. Unlike prior work on data selection, LFR models are Chinchilla-optimal demonstrating the effectiveness of our training methodology."
}