@inproceedings{crosbie-shutova-2025-induction,
    title = "Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning",
    author = "Crosbie, Joy  and
      Shutova, Ekaterina",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.283/",
    pages = "5034--5096",
    ISBN = "979-8-89176-195-7",
    abstract = "Large language models (LLMs) have shown a remarkable ability to learn and perform complex tasks through in-context learning (ICL). However, a comprehensive understanding of its internal mechanisms is still lacking. This paper explores the role of induction heads in a few-shot ICL setting. We analyse two state-of-the-art models, Llama-3-8B and InternLM2-20B on abstract pattern recognition and NLP tasks. Our results show that even a minimal ablation of induction heads leads to ICL performance decreases of up to {\textasciitilde}32{\%} for abstract pattern recognition tasks, bringing the performance close to random. For NLP tasks, this ablation substantially decreases the model{'}s ability to benefit from examples, bringing few-shot ICL performance close to that of zero-shot prompts. We further use attention knockout to disable specific induction patterns, and present fine-grained evidence for the role that the induction mechanism plays in ICL."
}