@inproceedings{van-nguyen-etal-2026-lizard,
    title = "Lizard: An Efficient Linearization Framework for Large Language Models",
    author = "Van Nguyen, Chien  and
      Nguyen, Huy Huu  and
      Zhang, Ruiyi  and
      Deilamsalehy, Hanieh  and
      Mathur, Puneet  and
      Lai, Viet Dac  and
      Wang, Haoliang  and
      Subramanian, Jayakumar  and
      Rossi, Ryan A.  and
      Bui, Trung  and
      Vlassis, Nikos  and
      Dernoncourt, Franck  and
      Nguyen, Thien Huu",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1613/",
    pages = "34935--34948",
    ISBN = "979-8-89176-390-6",
    abstract = "We propose Lizard, a linearization framework that transforms pretrained Transformer-based Large Language Models (LLMs) into subquadratic architectures. Transformers faces severe computational and memory bottlenecks with long sequences due to the quadratic complexity of softmax attention and the growing Key-Value (KV) cache that makes inference memory-bound by context length. Lizard addresses these limitations by introducing a subquadratic attention mechanism that closely approximates softmax attention while preserving model quality. Unlike prior linearization methods constrained by fixed, non-adaptive structures, Lizard augments the architecture with compact, learnable modules that enable adaptive memory control and robust length generalization. Moreover, we introduce a hardware-aware algorithm that solves numerical instability in gated attention to accelerate training. Extensive experiments show that Lizard achieves near-lossless recovery of its teacher model{'}s performance, significantly outperforming previous methods by up to 9.4 - 24.5 points on the 5-shot MMLU benchmark and demonstrating superior associative recall."
}