@inproceedings{sun-etal-2026-sepseq,
    title = "{S}ep{S}eq: A Training-Free Framework for Long Numerical Sequence Processing in {LLM}s",
    author = "Sun, Jie  and
      Liu, Yu  and
      Han, Lu  and
      Deng, Qiwen  and
      Shu, Xiang  and
      Xiao, Yang  and
      Ma, Lintao  and
      Lu, Xingyu  and
      Zhou, Jun  and
      Liu, Pengfei  and
      Wu, Jiancan  and
      Wang, Xiang",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.272/",
    pages = "5522--5537",
    ISBN = "979-8-89176-395-1",
    abstract = "While transformer-based Large Language Models (LLMs) theoretically support massive context windows, they suffer from severe performance degradation when processing long numerical sequences. We attribute this failure to the attention dispersion in the Softmax mechanism, which prevents the model from concentrating attention. To overcome this, we propose \textbf{Sep}arate \textbf{Seq}uence (\textbf{SepSeq}), a training-free, plug-and-play framework to mitigate dispersion by strategically inserting separator tokens. Mechanistically, we demonstrate that separator tokens act as an attention anchor, recalibrating attention to focus on local segments while preserving global context. Extensive evaluations on 9 widely-adopted LLMs confirm the effectiveness of our approach: SepSeq yields an average relative accuracy improvement of 35.6{\%} across diverse domains while reducing 16.4{\%} inference token consumption."
}