@inproceedings{shihab-etal-2026-beyond,
    title = "Beyond Variance: Knowledge-Aware {LLM} Compression via Fisher-Aligned Subspace Diagnostics",
    author = "Shihab, Ibne Farabi  and
      Akter, Sanjeda  and
      Sharma, Anuj",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.2214/",
    pages = "47938--47954",
    ISBN = "979-8-89176-390-6",
    abstract = "Post-training activation compression is essential for deploying Large Language Models (LLMs) on resource-constrained hardware. However, standard methods like Singular Value Decomposition (SVD) are gradient-blind: they preserve high-variance dimensions regardless of their impact on factual knowledge preservation. We introduce Fisher-Aligned Subspace Compression (FASC), a knowledge-aware compression framework that selects subspaces by directly modeling activation-gradient coupling, minimizing a second-order surrogate of the loss function. FASC leverages the Fisher Information Matrix to identify dimensions critical for factual knowledge, which often reside in low-variance but high-gradient-sensitivity subspaces. We propose the Dependence Violation Score ({\ensuremath{\rho}}) as a general-purpose diagnostic metric that quantifies activation-gradient coupling, revealing where factual knowledge is stored within transformer architectures. Extensive experiments on Mistral-7B and Llama-3-8B demonstrate that FASC preserves 6{--}8{\%} more accuracy on knowledge-intensive benchmarks (MMLU, LAMA) compared to variance-based methods at 50{\%} rank reduction, effectively enabling a 7B model to match the factual recall of a 13B uncompressed model. Our analysis reveals that {\ensuremath{\rho}} serves as a fundamental signal of stored knowledge, with high-{\ensuremath{\rho}} layers emerging only when models internalize factual associations during training"
}