@inproceedings{nakamura-etal-2026-demystifying,
    title = "Demystifying Mixed Outcomes of Self-Training: Pre-training Analyses on Non-Toy {LLM}s",
    author = "Nakamura, Yusuke  and
      Kiyomaru, Hirokazu  and
      Liu, Chaoran  and
      Kurita, Shuhei  and
      Kawahara, Daisuke",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.findings-eacl.213/",
    pages = "4107--4113",
    ISBN = "979-8-89176-386-9",
    abstract = "We investigate whether large language models (LLMs) can improve through recursive training on self-generated text, a topic where prior studies report conflicting outcomes: some find evidence of performance gains (i.e., \textit{self-improvement}), while others observe performance degradation (i.e., \textit{model collapse}). To clarify this discrepancy, we use the OLMo-2 models as non-toy LLMs and perform multiple rounds of continual pre-training using self-generated text with different prompting strategies and data filtering. Our experiments show that naive recursive self-training does not improve either perplexity or downstream task performance, regardless of model size. These results suggest that model collapse observed in naive recursive training is inherent to the training procedure itself, while self-improvement likely owes its success not to the model{'}s autonomous refinement but to human-designed, strategic synthetic pipelines that inject external intelligence."
}