@inproceedings{du-etal-2026-native,
    title = "Native Hybrid Attention for Efficient Sequence Modeling",
    author = "Du, Jusen  and
      Hu, Jiaxi  and
      Tao, Zhang  and
      Sun, Weigao  and
      Cheng, Yu",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.176/",
    pages = "3826--3842",
    ISBN = "979-8-89176-390-6",
    abstract = "Transformers excel at sequence modeling but face quadratic complexity, while linear attention offers improved efficiency but often compromises recall accuracy over long contexts. In this work, we introduce \textbf{Native Hybrid Attention (NHA)}, a novel hybrid architecture of linear and full attention that integrates both intra inter-layer hybridization into a unified layer design. NHA maintains long-term context in key{--}value slots updated by a linear RNN, and augments them with short-term tokens from a sliding window. A single operation is then applied over all keys and values, enabling per-token and per-head context-dependent weighting without requiring additional fusion parameters. The inter-layer behavior is controlled through a single hyperparameter, the sliding window size, which allows smooth adjustment between purely linear and full attention while keeping all layers structurally uniform. Experimental results show that NHA surpasses Transformers and other hybrid baselines on recall-intensive and commonsense reasoning tasks. Furthermore, pretrained LLMs can be structurally \textbf{hybridized} with NHA, achieving competitive accuracy while delivering significant efficiency gains. Code is available at \url{https://github.com/JusenD/NHA}."
}