@inproceedings{mei-etal-2026-gated,
    title = "Gated Differentiable Working Memory for Long-Context Language Modeling",
    author = "Mei, Lingrui  and
      Liu, Shenghua  and
      Wang, Yiwei  and
      Ge, Yuyao  and
      Bi, Baolong  and
      Yao, Jiayu  and
      Wan, Jun  and
      Yin, Ziling  and
      Guo, Jiafeng  and
      Cheng, Xueqi",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1471/",
    pages = "31885--31913",
    ISBN = "979-8-89176-390-6",
    abstract = "Long contexts break transformers: attention scores dilute across thousands of tokens, critical information gets lost in the middle, and the model cannot adapt to novel patterns at inference time. Recent work on test-time adaptation addresses this by maintaining a form of working memory{---}transient parameters updated on the current context{---}but existing approaches employ uniform write policies that waste computation on low-value regions and suffer from high gradient variance across semantically heterogeneous contexts. In this work, we reframe test-time adaptation as a budget-constrained memory consolidation problem, asking: given limited computational budget, which parts of the context should be consolidated into working memory? We propose GDWM ($\textbf{G}ated$ $\textbf{D}ifferentiable$ $\textbf{W}orking$ $\textbf{M}emory$), a framework that introduces a Write Controller to gate the memory consolidation process. Our controller estimates Contextual Utility{---}an information-theoretic measure quantifying how much each region depends on long-range context{---}and allocates gradient steps accordingly, subject to a coverage constraint that ensures global representation. Theoretically, we prove that our chunk-restricted sampling strategy reduces gradient variance by eliminating inter-chunk variance via the Law of Total Variance. Experiments on ZeroSCROLLS and LongBench v2 benchmarks demonstrate that GDWM achieves comparable or superior performance with 4 {\texttimes}fewer gradient steps compared to uniform baselines{---}excelling on sparse-information tasks (+6{--}13{\%} on Qasper, +5{--}13{\%} on GovReport for smaller models) while revealing principled trade-offs on dense-coverage tasks, establishing a new efficiency-performance Pareto frontier for test-time adaptation."
}