@inproceedings{chang-etal-2024-scaling,
    title = "Scaling Parameter-Constrained Language Models with Quality Data",
    author = "Chang, Ernie  and
      Paltenghi, Matteo  and
      Li, Yang  and
      Lin, Pin-Jie  and
      Zhao, Changsheng  and
      Huber, Patrick  and
      Liu, Zechun  and
      Rabatin, Rastislav  and
      Shi, Yangyang  and
      Chandra, Vikas",
    editor = "Dernoncourt, Franck  and
      Preo{\c{t}}iuc-Pietro, Daniel  and
      Shimorina, Anastasia",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2024",
    address = "Miami, Florida, US",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Add-Cong-Liu-Florida-Atlantic-University-author-id/2024.emnlp-industry.8/",
    doi = "10.18653/v1/2024.emnlp-industry.8",
    pages = "80--97",
    abstract = "Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization.In this paper, we extend the conventional understanding of scaling law by offering a microscopic view of data quality within the original formulation {--} \textit{effective training tokens} {--} which we posit to be a critical determinant of performance for parameter-constrained language models.Specifically, we formulate the proposed term of effective training tokens to be a combination of two readily-computed indicators of text:(i) text diversity and (ii) syntheticity as measured by a teacher model.We pretrained over 200 models of 25M to 1.5B parameters on a diverse set of sampled, synthetic data, and estimated the constants that relate text quality, model size, training tokens, and eight reasoning task accuracy scores.We demonstrated the estimated constants yield +0.83 Pearson correlation with true accuracies, and analyze it in scenarios involving widely-used data techniques such as data sampling and synthesis which aim to improve data quality."
}