@article{slack-al-moubayed-2025-early,
    title = "Early Detection and Reduction of Memorization for Domain Adaptation and Instruction Tuning",
    author = "Slack, Dean L.  and
      Al Moubayed, Noura",
    journal = "Transactions of the Association for Computational Linguistics",
    volume = "13",
    year = "2025",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://preview.aclanthology.org/fix-opsupmap-display/2025.tacl-1.66/",
    doi = "10.1162/tacl.a.49",
    pages = "1459--1473",
    abstract = "Although large language models excel across many tasks, they can memorize training data and thereby expose private or copyrighted text. Most defenses target the pre-training stage, leaving memorization during fine-tuning{--}especially for domain adaptation and instruction tuning{--}poorly understood. We fine-tune Pythia, Llama3, and Mistral models spanning 1.4B{--}70B parameters on common evaluation datasets and track verbatim memorization throughout training. We find that memorization increases dramatically in the first few epochs, often significantly before either validation perplexity or evaluation performance is optimized. We use a simple but effective n-gram memorization score which reliably precedes verbatim memorization; using it as an early-stopping criterion mitigates memorization with minimal performance loss. Further, we introduce an n-gram{--}aware loss regularizer and show that it reduces memorization across all model families tested by up to 40{\%} while minimizing evaluation performance trade-offs when compared to an existing memorization mitigation strategy. These results yield practical, scalable insights into memorization dynamics during language model fine-tuning."
}