@inproceedings{zhou-etal-2025-qimeng,
    title = "{Q}i{M}eng-Attention: {SOTA} Attention Operator is generated by {SOTA} Attention Algorithm",
    author = "Zhou, Qirui  and
      Peng, Shaohui  and
      Xiong, Weiqiang  and
      Chen, Haixin  and
      Wen, Yuanbo  and
      Li, Haochen  and
      Li, Ling  and
      Guo, Qi  and
      Zhao, Yongwei  and
      Gao, Ke  and
      Chen, Ruizhi  and
      Wu, Yanjun  and
      Chen, Zhao  and
      Chen, Yunji",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.446/",
    pages = "8491--8505",
    ISBN = "979-8-89176-256-5",
    abstract = "The attention operator remains a critical performance bottleneck in large language models (LLMs), particularly for long-context scenarios. While FlashAttention is the most widely used and effective GPU-aware acceleration algorithm, it must require time-consuming and hardware-specific manual implementation, limiting adaptability across GPU architectures. Existing LLMs have shown a lot of promise in code generation tasks, but struggle to generate high-performance attention code. The key challenge is it cannot comprehend the complex data flow and computation process of the attention operator and utilize low-level primitive to exploit GPU performance.To address the above challenge, we propose an LLM-friendly Thinking Language (LLM-TL) to help LLMs decouple the generation of high-level optimization logic and low-level implementation on GPU, and enhance LLMs' understanding of attention operator.Along with a 2-stage reasoning workflow, TL-Code generation and translation, the LLMs can automatically generate FlashAttention implementation on diverse GPUs, establishing a self-optimizing paradigm for generating high-performance attention operators in attention-centric algorithms.Verified on A100, RTX8000, and T4 GPUs, the performance of our methods significantly outshines that of vanilla LLMs, achieving a speed-up of up to $35.16\times$.Besides, our method not only surpasses human-optimized libraries (cuDNN and official library) in most scenarios but also extends support to unsupported hardware and data types, reducing development time from months to minutes compared with human experts."
}