@inproceedings{puvvada-etal-2025-swan,
    title = "{SWAN}: An Efficient and Scalable Approach for Long-Context Language Modeling",
    author = "Puvvada, Krishna C  and
      Ladhak, Faisal  and
      Akle Serano, Santiago  and
      Hsieh, Cheng-Ping  and
      Acharya, Shantanu  and
      Majumdar, Somshubra  and
      Jia, Fei  and
      Kriman, Samuel  and
      Sun, Simeng  and
      Rekesh, Dima  and
      Ginsburg, Boris",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.123/",
    pages = "2424--2438",
    ISBN = "979-8-89176-332-6",
    abstract = "We present SWAN, a causal Transformer architecture in the decoder-only style that generalizes robustly to sequence lengths substantially longer than those seen during training. SWAN interleaves layers without positional encodings (NoPE) and sliding-window attention layers equipped with rotary positional encodings (SWA-RoPE), and applies a dynamic scaling mechanism for attention scores during inference. Experiments demonstrate that SWAN achieves strong length extrapolation without requiring additional long-context training. In addition, SWAN is more computationally efficient than the standard Transformer architecture, resulting in lower training cost and higher inference throughput. We further demonstrate that existing pre-trained decoder-only models can be adapted to the SWAN architecture with minimal continued training, enabling extended contexts. Overall, our work presents an effective approach for scaling language models to longer contexts in a robust and efficient manner."
}