@inproceedings{li-etal-2025-tic,
    title = "{T}i{C}-{LM}: A Web-Scale Benchmark for Time-Continual {LLM} Pretraining",
    author = "Li, Jeffrey  and
      Armandpour, Mohammadreza  and
      Mirzadeh, Seyed Iman  and
      Mehta, Sachin  and
      Shankar, Vaishaal  and
      Vemulapalli, Raviteja  and
      Bengio, Samy  and
      Tuzel, Oncel  and
      Farajtabar, Mehrdad  and
      Pouransari, Hadi  and
      Faghri, Fartash",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1551/",
    pages = "32231--32273",
    ISBN = "979-8-89176-251-0",
    abstract = "Large Language Models (LLMs) trained on historical web data inevitably become outdated. We investigate evaluation strategies and update methods for LLMs as new data becomes available. We introduce a web-scale dataset for time-continual pretraining of LLMs derived from 114 dumps of Common Crawl (CC) {--} orders of magnitude larger than previous continual language modeling benchmarks. We also design time-stratified evaluations across both general CC data and specific domains (Wikipedia, StackExchange, and code documentation) to assess how well various continual learning methods adapt to new data while retaining past knowledge. Our findings demonstrate that, on general CC data, autoregressive meta-schedules combined with a fixed-ratio replay of older data can achieve comparable held-out loss to re-training from scratch, while requiring significantly less computation (2.6x). However, the optimal balance between incorporating new data and replaying old data differs as replay is crucial to avoid forgetting on generic web data but less so on specific domains."
}