@inproceedings{chen-etal-2026-sparse,
    title = "Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping",
    author = "Chen, Yao  and
      Chen, Yilong  and
      Yang, Yinqi  and
      Shang, Junyuan  and
      Zhang, Zhenyu  and
      Zhang, Zefeng  and
      Nie, Shuaiyi  and
      Wang, Shuohuan  and
      Sun, Yu  and
      Wu, Hua  and
      Wang, Haifeng  and
      Liu, Tingwen",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.307/",
    pages = "6168--6193",
    ISBN = "979-8-89176-395-1",
    abstract = "Existing approaches to increasing the effective depth of Transformers predominantly rely on parameter reuse, extending computation through recursive execution.Under this paradigm, the network structure remains static along the training timeline, and additional computational depth is uniformly assigned to entire blocks at the parameter level.This rigidity across training time and parameter space leads to substantial computational redundancy during training.In contrast, we argue that depth allocation during training should not be a static preset, but rather a progressively growing structural process. Our systematic analysis reveals a deep-to-shallow maturation trajectory across layers, where high-entropy attention heads play a crucial role in semantic integration. Motivated by this observation, we introduce the Sparse Growing Transformer (SGT).SGT is a training-time sparse depth allocation framework that progressively extends recurrence from deeper to shallower layers via targeted attention looping on informative heads. This mechanism induces structural sparsity by selectively increasing depth only for a small subset of parameters as training evolves.Extensive experiments across multiple parameter scales demonstrate that SGT consistently outperforms training-time static block-level looping baselines under comparable settings, while reducing the additional training FLOPs overhead from approximately 16{--}20{\%} to only 1{--}3{\%} relative to a standard Transformer backbone."
}