@inproceedings{deshpande-etal-2026-knowledge,
    title = "Knowledge Localization and Editability in Small Language Models: A Multi-Stage Experimental Study",
    author = "Deshpande, Pranamya Nilesh  and
      Konavoor, Aiswarya  and
      Panat, Sreedath",
    editor = "Chen, Canyu  and
      Zhang, Yuji  and
      Li, Zoey Sha  and
      Wang, Zihan  and
      Wang, Qineng  and
      Su, Jinyan  and
      Kargupta, Priyanka  and
      Marjanovi{\'c}, Sara Vera  and
      Pan, Jeff Z.  and
      Bansal, Mohit  and
      Augenstein, Isabelle  and
      Han, Jiawei  and
      Ji, Heng  and
      Li, Manling",
    booktitle = "Proceedings of the 4th Workshop on Towards Knowledgeable Foundation Models ({K}now{FM} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.knowfm-1.13/",
    pages = "165--172",
    ISBN = "979-8-89176-403-3",
    abstract = "The internal mechanisms by which transformer-based language models encode and retrieve factual knowledge remain poorly understood, particularly for small language models (SLMs) operating in the 2{--}3 billion parameter range. This paper presents a systematic, multi-stage empirical investigation into knowledge localization, compression effects, and knowledge editability across four SLMs{---}Gemma-2B, Llama-3.2-3B-Instruct, Qwen-2.5-3B-Instruct, and Phi-2{---}with Meta-Llama-3-8B serving as a large-model baseline. Stage 1 employs causal tracing with activation patching on the CounterFact dataset ({\textasciitilde}450{--}500 validated facts per model) to identify the layer or layers most causally responsible for factual recall. Stage 2 compares knowledge density, layer concentration, and redundancy between the 2{--}3B models and the 8B baseline to quantify the structural effects of model compression on knowledge storage. Stage 3 applies the Rank-One Model Editing (ROME) algorithm at the causally identified layers to assess whether localized knowledge can be reliably overwritten. Our results demonstrate that (i) factual knowledge in SLMs concentrates in upper-to-final transformer layers, with Llama-3B exhibiting extreme concentration in layer 28; (ii) compressed models store knowledge more densely per parameter but with substantially lower redundancy (Llama-3B: 0.047 vs. Llama-8B: 0.468); and (iii) editing success correlates strongly with architectural concentration rather than model size, with Llama-3B achieving 85.7{\%} editing success versus 33{\%} for Gemma-2B. These findings carry direct implications for interpretability, model editing, and the design of future small language model architectures."
}