@inproceedings{he-etal-2026-grokking,
    title = "Is Grokking Worthwhile? Functional Analysis and Transferability of Generalization Circuits in Transformers",
    author = "He, Kaiyu  and
      Zhang, Mian  and
      Wu, Peilin  and
      Du, Xinya  and
      Chen, Zhiyu",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1697/",
    pages = "33993--34001",
    ISBN = "979-8-89176-395-1",
    abstract = "While Large Language Models (LLMs) excel at factual retrieval, they often struggle with the ``curse of two-hop reasoning'' in compositional tasks. Recent research suggests that parameter-sharing transformers can bridge this gap by forming a ``Generalization Circuit'' during a prolonged ``grokking'' phase. A fundamental question arises: Is a grokked model truly superior to its non-grokked counterparts? Furthermore, is the extensive computational cost of waiting for the grokking phase worthwhile? In this work, we conduct a mechanistic study to evaluate the Generalization Circuit{'}s role in knowledge assimilation and transfer. We demonstrate that: (i) The inference paths established by non-grokked and grokked models for in-distribution compositional queries are identical. This suggests that the ``Generalization Circuit'' does not represent the sudden acquisition of a new reasoning paradigm. Instead, we argue that grokking is the process of integrating memorized atomic facts into an easy-acquire, naturally established reasoning path. (ii) Achieving high accuracy on unseen cases after prolonged training and the formation of a certain reasoning path are not bound; they can occur independently under specific data regimes. (iii) Even a mature circuit exhibits limited transferability when integrating new knowledge, suggesting that ``grokked'' Transformers do not achieve a full mastery of compositional logic."
}