@inproceedings{kinder-etal-2025-positional,
    title = "Positional Overload: Positional Debiasing and Context Window Extension for Large Language Models using Set Encoding",
    author = {Kinder, Lukas  and
      Edman, Lukas  and
      Fraser, Alexander  and
      K{\"a}fer, Tobias},
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.197/",
    pages = "3896--3908",
    ISBN = "979-8-89176-251-0",
    abstract = "Large Language Models (LLMs) typically track the order of tokens using positional encoding, which causes the following problems: positional bias, where the model is influenced by an ordering within the prompt, and a fixed context window, as models struggle to generalize to positions beyond those encountered during training. To address these limitations, we developed a novel method called $\textit{set encoding}$. This method allows multiple pieces of text to be encoded in the same position, thereby eliminating positional bias entirely. Another promising use case for set encoding is to increase the size of the input an LLM can handle. Our experiments demonstrate that set encoding allows an LLM to solve tasks with far more tokens than without set encoding. To our knowledge, set encoding is the first technique to effectively extend an LLM{'}s context window without requiring any additional training."
}