@inproceedings{zhao-etal-2025-position,
    title = "Position {ID}s Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models",
    author = "Zhao, Runsong  and
      Liu, Xin  and
      Liu, Xinyu  and
      Huang, Pengcheng  and
      Xiao, Chunyang  and
      Xiao, Tong  and
      Zhu, JingBo",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.962/",
    doi = "10.18653/v1/2025.findings-emnlp.962",
    pages = "17715--17734",
    ISBN = "979-8-89176-335-7",
    abstract = "Using special tokens (e.g., gist, memory, or compressed tokens) to compress context information is a common practice for large language models (LLMs). However, existing approaches often neglect that position encodings inherently induce local inductive biases in models, causing the compression process to ignore holistic contextual dependencies. We propose **Enhanced Position Layout (EPL)**, a simple yet effective method that improves the context compression capability of LLMs by only adjusting position IDs, the numerical identifiers that specify token positions. EPL minimizes the distance between context tokens and their corresponding special tokens and at the same time maintains the sequence order in position IDs between context tokens, special tokens, and the subsequent tokens. Integrating EPL into our best performing context compression model results in 1.9 ROUGE-1 F1 improvement on out-of-domain question answering datasets in average. When extended to multimodal scenarios, EPL brings an average accuracy gain of 2.6 to vision compression LLMs."
}