@inproceedings{ji-etal-2025-towards,
    title = "Towards Economical Inference: Enabling {D}eep{S}eek{'}s Multi-Head Latent Attention in Any Transformer-based {LLM}s",
    author = "Ji, Tao  and
      Guo, Bin  and
      Wu, Yuanbin  and
      Guo, Qipeng  and
      Shenlixing, Shenlixing  and
      Chenzhan, Chenzhan  and
      Qiu, Xipeng  and
      Zhang, Qi  and
      Gui, Tao",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1597/",
    pages = "33313--33328",
    ISBN = "979-8-89176-251-0",
    abstract = "Multi-head Latent Attention (MLA) is an innovative architecture proposed by DeepSeek, designed to ensure efficient and economical inference by significantly compressing the Key-Value (KV) cache into a latent vector. Compared to MLA, standard LLMs employing Multi-Head Attention (MHA) and its variants such as Grouped-Query Attention (GQA) exhibit significant cost disadvantages. Enabling well-trained LLMs (e.g., Llama) to rapidly adapt to MLA without pre-training from scratch is both meaningful and challenging. This paper proposes the first data-efficient fine-tuning method for transitioning from MHA to MLA (**MHA2MLA**), which includes two key components: for *partial-RoPE*, we remove RoPE from dimensions of queries and keys that contribute less to the attention scores, for *low-rank approximation*, we introduce joint SVD approximations based on the pre-trained parameters of keys and values. These carefully designed strategies enable MHA2MLA to recover performance using only a small fraction (0.6{\%} to 1{\%}) of the data, significantly reducing inference costs while seamlessly integrating with compression techniques such as KV cache quantization. For example, the KV cache size of Llama2-7B is reduced by 92.19{\%}, with only a 1{\%} drop in LongBench performance. Our source code is publicly available at https://github.com/JT-Ushio/MHA2MLA."
}