@inproceedings{ou-etal-2025-llms,
    title = "How Do {LLM}s Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training",
    author = "Ou, Yixin  and
      Yao, Yunzhi  and
      Zhang, Ningyu  and
      Jin, Hui  and
      Sun, Jiacheng  and
      Deng, Shumin  and
      Li, Zhenguo  and
      Chen, Huajun",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.findings-acl.1021/",
    doi = "10.18653/v1/2025.findings-acl.1021",
    pages = "19889--19913",
    ISBN = "979-8-89176-256-5",
    abstract = "Despite exceptional capabilities in knowledge-intensive tasks, Large Language Models (LLMs) face a critical gap in understanding how they internalize new knowledge, particularly how acquired knowledge becomes structurally embedded in their neural computations. We address this issue through the lens of knowledge circuit evolution, identifying computational subgraphs that facilitate knowledge storage and processing. Our systematic analysis of circuit evolution throughout continual pre-training reveals several key findings: (1) the acquisition of new knowledge is influenced by its relevance to pre-existing knowledge; (2) the evolution of knowledge circuits exhibits a distinct phase shift from formation to optimization; (3) the evolution of knowledge circuits follows a deep-to-shallow pattern. These insights not only advance our theoretical understanding of the mechanisms of new knowledge acquisition in LLMs, but also provide potential implications for improving continual pre-training strategies to enhance model performance."
}