@inproceedings{africa-etal-2025-learning,
    title = "Learning Dynamics of Meta-Learning in Small Model Pretraining",
    author = "Africa, David Demitri  and
      Weiss, Yuval  and
      Buttery, Paula  and
      Diehl Martinez, Richard",
    editor = "T.y.s.s, Santosh  and
      Shimizu, Shuichiro  and
      Gong, Yifan",
    booktitle = "The 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.ijcnlp-srw.2/",
    pages = "10--23",
    ISBN = "979-8-89176-304-3",
    abstract = "Large language models are powerful but costly. We ask whether meta-learning can make the pretraining of small language models not only faster but also more interpretable. We integrate first{--}order MAML with subset-masked LM pretraining, producing four LLama-style decoder-only models (11M{--}570M params), and evaluate on multilingual Universal NER. Compared with vanilla training, our hybrid setup (i) reaches the same loss up to 1.6{\texttimes} sooner, (ii) yields modest but consistent average gains on Universal NER at medium/large scales under equal compute (+2{--}3 percentage points), and (iii) and (iii) reveals phase-like learning dynamics: models first diversify their representations, then compress them in a pattern that aligns with improved episodic accuracy. These observations are correlational, not causal, and we do not claim generality beyond NER or across seeds. We also document a trade-off: perplexity on Paloma (a diverse language modeling benchmark spanning 18 domains) is worse at most scales. Code, checkpoints and analysis logs are released."
}