@inproceedings{zhou-etal-2026-deputy,
    title = "Deputy: Accelerating Large Language Model Inference with Dynamic Low-Rank Substitution",
    author = "Zhou, Yuhua  and
      Weng, Shichao  and
      Zhou, Changhai  and
      Wu, Yuhan  and
      Qiao, Qian  and
      Gao, Jun  and
      Yang, Fei  and
      Pan, Aimin",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.991/",
    pages = "19791--19810",
    ISBN = "979-8-89176-395-1",
    abstract = "While the massive scale of modern LLMs enables remarkable performance, their static, input-agnostic computational graph incurs substantial resource wastage and high latency during inference. Existing dynamic schemes, such as early-exit and layer-drop reduce FLOPs but break batch processing or introduce KV-cache inconsistency. We propose Deputy, a dynamic low-rank substitution framework that employs a lightweight decision module at each layer to dynamically determine the execution branch for different tokens: Attention layers choose between full and low-rank computation to mitigate the KV cache issue, while FFN layers additionally support skipping to further reduce computation. We fine-tune the LLM with LoRA and then derive an additional low-rank matrix C via a least-squares fit $\textbf{BC} \approx \textbf{W}_{\text{pre}}$, where B is the shared LoRA matrix, so that only one extra low-rank matrix is introduced, effectively reducing memory overhead. Moreover, a hybrid KV cache strategy stores KV values generated by the low-rank branch, achieving a 38{\%} reduction in cache storage. Experiments on Llama models demonstrate that Deputy reduces computation by approximately 40{\%} compared to the original dense model while outperforming existing baseline methods."
}