@inproceedings{chen-etal-2025-cost,
    title = "Cost-Optimal Grouped-Query Attention for Long-Context Modeling",
    author = "Chen, Yingfa  and
      Wu, Yutong  and
      Song, Chenyang  and
      Thai, Zhen Leng  and
      Shen, Xingyu  and
      Han, Xu  and
      Liu, Zhiyuan  and
      Sun, Maosong",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.272/",
    pages = "5360--5376",
    ISBN = "979-8-89176-332-6",
    abstract = "Grouped-Query Attention (GQA) is a widely adopted strategy for reducing the computational cost of attention layers in large language models (LLMs). However, current GQA configurations are often suboptimal because they overlook how context length influences inference cost. Since inference cost grows with context length, the most cost-efficient GQA configuration should vary accordingly. In this work, we analyze the relationship among context length, model size, GQA configuration, and model loss, and introduce two innovations: (1) we decouple the total head size from the hidden size, enabling more flexible control over attention FLOPs; and (2) we jointly optimize the model size and the GQA configuration to arrive at a better allocation of inference resources between attention layers and other components. Our analysis reveals that commonly used GQA configurations are highly suboptimal for long-context scenarios. Moreover, we propose a recipe for deriving cost-optimal GQA configurations. Our results show that for long-context scenarios, one should use fewer attention heads while scaling up the model size. Configurations selected by our recipe can reduce both memory usage and FLOPs by more than 50{\%} compared to Llama-3{'}s GQA, with *no degradation in model capabilities*. Our findings offer valuable insights for designing efficient long-context LLMs."
}