@inproceedings{endy-etal-2025-mamba,
    title = "Mamba Knockout for Unraveling Factual Information Flow",
    author = "Endy, Nir  and
      Grosbard, Idan Daniel  and
      Ran-Milo, Yuval  and
      Slutzky, Yonatan  and
      Tshuva, Itay  and
      Giryes, Raja",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1143/",
    pages = "23457--23477",
    ISBN = "979-8-89176-251-0",
    abstract = "This paper investigates the flow of factual information in Mamba State-Space Model (SSM)-based language models. We rely on theoretical and empirical connections to Transformer-based architectures and their attention mechanisms. Exploiting this relationship, we adapt attentional interpretability techniques originally developed for Transformers{---}specifically, the Attention Knockout methodology{---}to both Mamba-1 and Mamba-2. Using them we trace how information is transmitted and localized across tokens and layers, revealing patterns of subject-token information emergence and layer-wise dynamics. Notably, some phenomena vary between mamba models and Transformer based models, while others appear universally across all models inspected{---}hinting that these may be inherent to LLMs in general. By further leveraging Mamba{'}s structured factorization, we disentangle how distinct ``features'' either enable token-to-token information exchange or enrich individual tokens, thus offering a unified lens to understand Mamba internal operations."
}