@inproceedings{bhaskar-etal-2024-heuristic,
    title = "The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models",
    author = "Bhaskar, Adithya  and
      Friedman, Dan  and
      Chen, Danqi",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Add-Cong-Liu-Florida-Atlantic-University-author-id/2024.acl-long.774/",
    doi = "10.18653/v1/2024.acl-long.774",
    pages = "14351--14368",
    abstract = "Prior work has found that pretrained language models (LMs) fine-tuned with different random seeds can achieve similar in-domain performance but generalize differently on tests of syntactic generalization. In this work, we show that, even within a single model, we can find multiple subnetworks that perform similarly in-domain, but generalize vastly differently. To better understand these phenomena, we investigate if they can be understood in terms of {\textquotedblleft}competing subnetworks{\textquotedblright}: the model initially represents a variety of distinct algorithms, corresponding to different subnetworks, and generalization occurs when it ultimately converges to one. This explanation has been used to account for generalization in simple algorithmic tasks ({\textquotedblleft}grokking{\textquotedblright}). Instead of finding competing subnetworks, we find that all subnetworks{---}whether they generalize or not{---}share a set of attention heads, which we refer to as the {\_}heuristic core{\_}. Further analysis suggests that these attention heads emerge early in training and compute shallow, non-generalizing features. The model learns to generalize by incorporating additional attention heads, which depend on the outputs of the {\textquotedblleft}heuristic{\textquotedblright} heads to compute higher-level features. Overall, our results offer a more detailed picture of the mechanisms for syntactic generalization in pre-trained LMs."
}