@inproceedings{lee-etal-2026-scale,
    title = "{SCALE}: Upscaled Continual Learning of Large Language Models",
    author = "Lee, Jin-woo  and
      Choi, Junhwa  and
      Hwang, Bongkyu  and
      Choo, Jinho  and
      Kim, Bogun  and
      Yi, Jeongseon  and
      Lee, Joonseok  and
      Jung, DongYoung  and
      Park, Jaeseon  and
      Park, Kyoungwon  and
      Jung, Suk-hoon",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.2037/",
    pages = "41006--41020",
    ISBN = "979-8-89176-395-1",
    abstract = "We revisit continual pre-training for large language models and argue that progress now depends less on scaling parameters than on scaling the right structure. We introduce SCALE, a width upscaling architecture that inserts lightweight expansions into linear modules while freezing all pre-trained parameters, preserving residual and attention topologies and increasing capacity without perturbing the base model{'}s original functionality. SCALE follows two principles: Persistent Preservation, which maintains the base model{'}s behavior via preservation-oriented initialization and freezing of the pre-trained weights, and Collaborative Adaptation, which trains only selected expansion components to acquire new knowledge with minimal interference. We instantiate these ideas as SCALE-Preserve (preservation-first), SCALE-Adapt (adaptation-first), and SCALE-Route, an optional routing extension that performs token-level routing between preservation and adaptation heads. On a controlled synthetic biography benchmark, SCALE reduces the severe forgetting seen in depth expansion while still learning new knowledge. In continual pre-training on a Korean corpus, SCALE variants forget less on English evaluations and achieve competitive gains on Korean benchmarks, yielding the best overall stability-plasticity trade-off. We further analyze when preservation holds provably and why combining preservation and adaptation stabilizes optimization relative to standard continual learning."
}