@inproceedings{yuan-etal-2024-kv,
    title = "{KV} Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches",
    author = "Yuan, Jiayi  and
      Liu, Hongyi  and
      Zhong, Shaochen  and
      Chuang, Yu-Neng  and
      Li, Songchen  and
      Wang, Guanchu  and
      Le, Duy  and
      Jin, Hongye  and
      Chaudhary, Vipin  and
      Xu, Zhaozhuo  and
      Liu, Zirui  and
      Hu, Xia",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.findings-emnlp.266/",
    doi = "10.18653/v1/2024.findings-emnlp.266",
    pages = "4623--4648",
    abstract = "Long context capability is a crucial competency for large language models (LLMs) as it mitigates the human struggle to digest long-form texts. This capability enables complex task-solving scenarios such as book summarization, code assistance, and many more tasks that are traditionally manpower-intensive. However, transformer-based LLMs face significant challenges with long context input due to the growing size of the KV cache and the intrinsic complexity of attending to extended inputs; where multiple schools of efficiency-driven approaches {---} such as KV cache quantization, token dropping, prompt compression, linear-time sequence models, and hybrid architectures {---} have been proposed to produce efficient yet long context-capable models. Despite these advancements, no existing work has comprehensively benchmarked these methods in a reasonably aligned environment. In this work, we fill this gap by providing a taxonomy of current methods and evaluating 10+ state-of-the-art approaches across seven categories of long context tasks. Our work reveals numerous previously unknown phenomena and offers insights {---} as well as a friendly workbench {---} for the future development of long context-capable LLMs. The source code is available at https://github.com/henryzhongsc/longctx{\_}bench."
}