@inproceedings{feng-etal-2025-aimmerging,
    title = "{AIMM}erging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learning",
    author = "Feng, Yujie  and
      Li, Jian  and
      Dong, Xiaoyu  and
      Xu, Pengfei  and
      Zhou, Xiaohui  and
      Zhang, Yujia  and
      Lu, Zexin  and
      Wang, Yasha  and
      Zhao, Alan  and
      Chu, Xu  and
      Wu, Xiao-Ming",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.678/",
    pages = "13431--13448",
    ISBN = "979-8-89176-332-6",
    abstract = "Continual learning (CL) is essential for deploying large language models (LLMs) in dynamic real-world environments without the need for costly retraining. Recent model merging-based methods have attracted significant attention, but they still struggle to effectively manage the trade-off between learning new knowledge and preventing forgetting, a challenge largely stemming from suboptimal number of merges and merging frequency. In this paper, we introduce Adaptive Iterative Model Merging (AimMerging), a novel CL framework that utilizes learning and forgetting signals from the training trajectory to dynamically monitor the model{'}s training status. Guided by dynamic monitoring, the training trajectory-guided merge controller adaptively determines the timing and frequency of iterative fusion, while the rehearsal-based knowledge fusion module computes the merging weights and executes the fusion. Comprehensive experiments on three CL benchmarks with various model sizes (from 770M to 13B) demonstrate that AimMerging achieves significant performance improvements over existing state-of-the-art methods, with an average relative improvement of 80{\%} and 59{\%} on FWT and BWT, respectively. The source code is provided for reproducibility."
}