@inproceedings{ma-etal-2025-memorization,
    title = "Memorization: A Close Look at Books",
    author = "Ma, Iris  and
      Domingo, Ian  and
      Krone-Martins, Alberto  and
      Baldi, Pierre  and
      Lopes, Cristina",
    editor = "Jia, Robin  and
      Wallace, Eric  and
      Huang, Yangsibo  and
      Pimentel, Tiago  and
      Maini, Pratyush  and
      Dankers, Verna  and
      Wei, Johnny  and
      Lesci, Pietro",
    booktitle = "Proceedings of the First Workshop on Large Language Model Memorization (L2M2)",
    month = aug,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.l2m2-1.13/",
    pages = "169--182",
    ISBN = "979-8-89176-278-7",
    abstract = "To what extent can entire books be extracted from LLMs? Using the Llama 3 70B family of models, and the ``prefix-prompting'' extractiontechnique, we were able to auto-regressively reconstruct, with a very high level of similarity, one entire book (Alice{'}s Adventures in Wonderland) from just the first 500 tokens. We were also able to obtain high extraction rates on several other books, piece-wise. However, these successes do not extend uniformly to all books. We show that extraction rates of books correlate with book popularity and thus, likely duplication in the training data. We also confirm the undoing of mitigations in the instruction-tuned Llama 3.1, following recent work (Nasr et al., 2025). We further find that this undoing comes from changes to only a tiny fraction of weights concentrated primarily in the lower transformer blocks. Our results provide evidence of the limits of current regurgitation mitigation strategies and introduce a framework for studying how fine-tuning affects the retrieval of verbatim memorization in aligned LLMs."
}