@inproceedings{potamitis-etal-2025-cache,
    title = "Cache Saver: A Modular Framework for Efficient, Affordable, and Reproducible {LLM} Inference",
    author = "Potamitis, Nearchos  and
      Klein, Lars Henning  and
      Mohammadi, Bardia  and
      Xu, Chongyang  and
      Mukherjee, Attreyee  and
      Tandon, Niket  and
      Bindschaedler, Laurent  and
      Arora, Akhil",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/name-variant-enfa-fane/2025.findings-emnlp.1402/",
    doi = "10.18653/v1/2025.findings-emnlp.1402",
    pages = "25703--25724",
    ISBN = "979-8-89176-335-7",
    abstract = "Inference constitutes the majority of costs throughout the lifecycle of a large language model (LLM). While numerous LLM inference engines focusing primarily on low-level optimizations have been developed, there is a scarcity of non-intrusive client-side frameworks that perform high-level optimizations. In this paper, we introduce Cache Saver, a modular, plug-and-play, and asynchronous framework that facilitates high-level inference optimizations, thereby integrating cleanly into existing systems without requiring changes to the end-user application logic or the underlying LLM. The key novelty is a *namespace-aware list-valued cache* that ensures *statistical integrity* of LLM responses by generating *i.i.d.* responses within a namespace as well as ensuring *reproducibility*. Moreover, as a direct consequence of operating at a high level, Cache Saver supports both local and online models. We conduct extensive experiments with five representative state-of-the-art reasoning strategies, five diverse benchmark tasks, and three different LLMs. On average across all methods, tasks, and LLMs, Cache Saver reduces cost by $\simeq 25\%$ and CO$_2$ by $\simeq 35\%$. Notably, Cache Saver excels in practical machine learning scenarios such as benchmarking across multiple methods or conducting ablation analysis of a specific method, obtaining substantial cost and carbon footprint reduction of $\simeq 60\%$. Cache Saver is publicly available at [https://github.com/au-clan/cachesaver](https://github.com/au-clan/cachesaver)."
}