@inproceedings{yu-etal-2025-mitigate,
    title = "Mitigate Position Bias in {LLM}s via Scaling a Single Hidden States Channel",
    author = "Yu, Yijiong  and
      Jiang, Huiqiang  and
      Luo, Xufang  and
      Wu, Qianhui  and
      Lin, Chin-Yew  and
      Li, Dongsheng  and
      Yang, Yuqing  and
      Huang, Yongfeng  and
      Qiu, Lili",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.316/",
    pages = "6092--6111",
    ISBN = "979-8-89176-256-5",
    abstract = "Long-context language models (LCLMs) can process long context, but still exhibit position bias, also known as ``lost in the middle'', which indicates placing key information in the middle of the context will significantly affect performance. To mitigating this, we first explore the micro-level manifestations of position bias, concluding that attention weights are a micro-level expression of position bias. Then we identify that, in addition to position embeddings, positional information in hidden states also contributes to position bias, and it manifests itself in specific channels of hidden states, called positional hidden states. Based on these, we propose a method to mitigate position bias by scaling positional hidden states. Experiments on NaturalQuestions Multi-document QA, KV retrieval and LongBench, using various models including RoPE models, context window-extended models, and Alibi models, demonstrate the effectiveness and generalizability of our approach. Our method can improve performance by up to 15.2{\%} in ``lost in the middle'' benchmark by modifying just one channel of hidden states. Our code is available at https://aka.ms/PositionalHidden."
}