@inproceedings{chen-etal-2025-edgeinfinite,
    title = "{E}dge{I}nfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices",
    author = "Chen, Jiyu  and
      Peng, Shuang  and
      Luo, Daxiong  and
      Yang, Fan  and
      Wu, Renshou  and
      Li, Fangyuan  and
      Chen, Xiaoxin",
    editor = "Rehm, Georg  and
      Li, Yunyao",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.acl-industry.40/",
    doi = "10.18653/v1/2025.acl-industry.40",
    pages = "568--575",
    ISBN = "979-8-89176-288-6",
    abstract = "Transformer-based large language models (LLMs) encounter challenges in processing long sequences on edge devices due to the quadratic complexity of attention mechanisms and growing memory demands from Key-Value (KV) cache. Existing KV cache optimizations struggle with irreversible token eviction in long-output tasks, while alternative sequence modeling architectures prove costly to adopt within established Transformer infrastructure. We present EdgeInfinite, a memory-efficient solution for infinite contexts that integrates compressed memory into Transformer-based LLMs through a trainable memory-gating module. This approach maintains full compatibility with standard Transformer architectures, requiring fine-tuning only a small part of parameters, and enables selective activation of the memory-gating module for long and short context task routing. The experimental result shows that EdgeInfinite achieves comparable performance to baseline Transformer-based LLM on long context benchmarks while optimizing memory consumption and time to first token."
}