@inproceedings{wang-etal-2026-mitigating-position,
    title = "Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling",
    author = "Wang, Zhenghua  and
      Ding, Yiran  and
      Lv, Changze  and
      Wu, Yixin  and
      Li, Tianlong  and
      Xu, Zhibo  and
      Wu, Muling  and
      Shi, Tianyuan  and
      Li, Shizheng  and
      Qian, Qi  and
      Huang, Xuanjing  and
      Zheng, Xiaoqing",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1059/",
    pages = "21084--21098",
    ISBN = "979-8-89176-395-1",
    abstract = "Large Language Models (LLMs) still struggle with the ``lost-in-the-middle'' problem, where critical information located in the middle of long-context inputs is often underrepresented or lost. While existing methods attempt to address this by combining multi-scale rotary position embeddings (RoPE), they typically suffer from high latency or rely on suboptimal hand-crafted scaling strategies. To overcome these limitations, we introduce a layer-specific positional embedding scaling (LPES) method that assigns distinct scaling factors to each layer. LPES achieves a more balanced attention distribution without fine-tuning model parameters or increasing inference delay. A specially designed genetic algorithm is employed to efficiently select the optimal scaling factors for each layer by incorporating B'{e}zier curves to significantly reduce the search space. Extensive experiments demonstrate that LPES effectively mitigates positional attention bias and delivers consistent improvements across multiple long-context benchmarks, yielding up to an 11.2{\%} accuracy gain on the key-value retrieval dataset."
}