@inproceedings{dong-etal-2025-longred,
    title = "{L}ong{R}e{D}: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation",
    author = "Dong, Zican  and
      Li, Junyi  and
      Jiang, Jinhao  and
      Xu, Mingyu  and
      Zhao, Xin  and
      Wang, Bingning  and
      Chen, Weipeng",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.524/",
    pages = "10687--10707",
    ISBN = "979-8-89176-251-0",
    abstract = "Large language models (LLMs) have gained extended context windows through scaling positional encodings and lightweight continual pre-training. However, this often leads to degraded performance on short-text tasks, while the reasons for this degradation remain insufficiently explored. In this work, we identify two primary factors contributing to this issue: distribution drift in hidden states and attention scores, and catastrophic forgetting during continual pre-training. To address these challenges, we propose Long Context Pre-training with Restoration Distillation (LongReD), a novel approach designed to mitigate short-text performance degradation through minimizing the distribution discrepancy between the extended and original models. Besides training on long texts, LongReD distills the hidden state of selected layers from the original model on short texts. Additionally, LongReD also introduces a short-to-long distillation, aligning the output distribution on short texts with that on long texts by leveraging skipped positional indices. Experiments on common benchmarks demonstrate that LongReD effectively preserves the model{'}s short-text performance while maintaining or even enhancing its long-context abilities."
}