@inproceedings{hu-etal-2025-pepe,
    title = "{PEPE}: Long-context Extension for Large Language Models via Periodic Extrapolation Positional Encodings",
    author = "Hu, Jikun  and
      Guo, Dongsheng  and
      Liu, Yuli  and
      Ai, Qingyao  and
      Wang, Lixuan  and
      Sun, Xuebing  and
      Zhang, Qilei  and
      Zhou, Quan  and
      Luo, Cheng",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.1149/",
    doi = "10.18653/v1/2025.findings-emnlp.1149",
    pages = "21075--21085",
    ISBN = "979-8-89176-335-7",
    abstract = "Long-context extension seeks to expand the contextual window in pre-trained large language models (LLMs), allowing them to handle several multiples of their original training context lengths. The primary method for extending the window length involves expanding the initial positional encodings, such as interpolating and extrapolation new positions based on Rotary Position Embedding (RoPE). This expansion inevitably disrupts the positional encodings learned during pre-training, thereby affecting the attention allotment and introducing unseen positional encoding distributions. To address this issue, we propose a new extension strategy based on RoPE, namely Periodic Extrapolation Positional Encodings (PEPE). This strategy expands pre-trained high dimensional components of positional encodings by replicating them in a periodic manner, thereby neither altering the learned positional encoding spaces nor introducing new positional encoding distributions. Experiments demonstrate that PEPE-based approaches can significantly improve long-context extension capabilities using just one-fourth the fine-tuning steps required by state-of-the-art methods. In addition, we analyze the characteristics of PEPE based methods and the key parameters that contribute to their effectiveness. The code is publicly available."
}