@inproceedings{armengol-estape-etal-2024-statically,
    title = "Can We Statically Locate Knowledge in Large Language Models? Financial Domain and Toxicity Reduction Case Studies",
    author = "Armengol-Estap{\'e}, Jordi  and
      Li, Lingyu  and
      Gehrmann, Sebastian  and
      Gopal, Achintya  and
      Rosenberg, David S  and
      Mann, Gideon S.  and
      Dredze, Mark",
    editor = "Belinkov, Yonatan  and
      Kim, Najoung  and
      Jumelet, Jaap  and
      Mohebbi, Hosein  and
      Mueller, Aaron  and
      Chen, Hanjie",
    booktitle = "Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP",
    month = nov,
    year = "2024",
    address = "Miami, Florida, US",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Add-Cong-Liu-Florida-Atlantic-University-author-id/2024.blackboxnlp-1.9/",
    doi = "10.18653/v1/2024.blackboxnlp-1.9",
    pages = "140--176",
    abstract = "Current large language model (LLM) evaluations rely on benchmarks to assess model capabilities and their encoded knowledge. However, these evaluations cannot reveal where a model encodes its knowledge, and thus little is known about which weights contain specific information. We propose a method to statically (without forward or backward passes) locate topical knowledge in the weight space of an LLM, building on a prior insight that parameters can be decoded into interpretable tokens. If parameters can be mapped into the embedding space, it should be possible to directly search for knowledge via embedding similarity. We study the validity of this assumption across several LLMs for a variety of concepts in the financial domain and a toxicity detection setup. Our analysis yields an improved understanding of the promises and limitations of static knowledge location in real-world scenarios."
}