@inproceedings{xiao-etal-2025-efficient,
    title = "Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention",
    author = "Xiao, Emily  and
      Li, Chin-Jou  and
      Zhang, Yilin  and
      Neubig, Graham  and
      Bertsch, Amanda",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1542/",
    pages = "31946--31958",
    ISBN = "979-8-89176-251-0",
    abstract = "Many-shot in-context learning has recently shown promise as an alternative to finetuning, with the major advantage that the same model can be served for multiple tasks. However, this shifts the computational burden from training-time to inference-time, making deployment of many-shot ICL challenging to justify in-practice. This cost is further increased if a custom demonstration set is retrieved for each inference example. We present Dynamic Block-Sparse Attention, an optimized method for retrieval-based many-shot in-context learning. By combining carefully designed block-sparse attention and retrieval of cached groups of demonstrations, we achieve comparable per-example latency to finetuning while maintaining on average {\ensuremath{>}}95{\%} of the best method{'}s accuracy across strong ICL and finetuning baselines. We hope that this will further enable the deployment of many-shot ICL at scale."
}