@inproceedings{hu-etal-2025-longrecipe,
    title = "{L}ong{R}ecipe: Recipe for Efficient Long Context Generalization in Large Language Models",
    author = "Hu, Zhiyuan  and
      Liu, Yuliang  and
      Zhao, Jinman  and
      Wang, Suyuchen  and
      WangYan, WangYan  and
      Shen, Wei  and
      Gu, Qing  and
      Luu, Anh Tuan  and
      Ng, See-Kiong  and
      Jiang, Zhiwei  and
      Hooi, Bryan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.581/",
    pages = "11857--11870",
    ISBN = "979-8-89176-251-0",
    abstract = "Large language models (LLMs) face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. Meanwhile, extending the context window in LLMs through post-pretraining is highly resource-intensive.To address this, we introduce LongRecipe, an efficient training strategy for extending the context window of LLMs, including impactful token analysis, position index transformation, and training optimization strategies. It simulates long-sequence inputs while maintaining training efficiency and significantly improves the model{'}s understanding of long-range dependencies. Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30{\%} of the target context window size, and reduces computational training resource over 85{\%} compared to full sequence training. Furthermore, LongRecipe also preserves the original LLM{'}s capabilities in general tasks. Ultimately, \textit{we can extend effective context window of open-source LLMs from 8k to 128k, achieving performance close to GPT-4 with just one day of dedicated training using a single GPU with 80G memory.}Our code is released at https://github.com/zhiyuanhubj/LongRecipe."
}