@inproceedings{ma-etal-2026-match,
    title = "{MATCH}: Modulating Attention via In-Context Retrieval for Long-Context Transformers",
    author = "Ma, Linrui  and
      Lo, Chun Hei  and
      Wang, Xinyu  and
      Lu, Peng  and
      Yuan, Xihao  and
      Chen, Hanting  and
      Han, Kai  and
      Chen, Xinghao  and
      Zhan, Chengjun  and
      xu, Hanlin  and
      Yin, Yichun  and
      Shang, Lifeng  and
      Wen, Feng  and
      Chen, Boxing  and
      Cui, Yufei",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.692/",
    pages = "15165--15179",
    ISBN = "979-8-89176-390-6",
    abstract = "The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particularly in long-context scenarios. To improve efficiency, existing approaches often enforce rigid structural constraints such as local attention windows. However, these strategies typically lead to substantial performance degradation on tasks requiring precise long-range recall. In this work, we propose MATCH, a scalable and efficient framework that augments sparsified attention mechanisms with dynamically integrated in-context information through an efficient retrieval system. Empirical results show that MATCH significantly improves the performance of sparse-attention models on both synthetic and real-world natural-language tasks. These findings highlight the versatility of MATCH as a general approach for enhancing in-context retrieval capabilities while maintaining the efficiency benefits of sparse attention architectures."
}