@inproceedings{gong-etal-2025-beyond,
    title = "Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation",
    author = "Gong, Guoqiang  and
      Wang, Jiaxing  and
      Xu, Jin  and
      Xiang, Deping  and
      Zhang, Zicheng  and
      Shen, Leqi  and
      Zhang, Yifeng  and
      JunhuaShu, JunhuaShu  and
      ZhaolongXing, ZhaolongXing  and
      Chen, Zhen  and
      Liu, Pengzhang  and
      Zhang, Ke",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1125/",
    pages = "23067--23077",
    ISBN = "979-8-89176-251-0",
    abstract = "Knowledge distillation (KD) compresses large language models (LLMs), known as teacher models, into lightweight versions called student models, enabling efficient inference and downstream applications. However, prevailing approaches accomplish this by predominantly focusing on matching the final output distributions of student/teacher models. Drawing on the perspective that transformers can be viewed as discretizing ordinary differential equation (ODEs) on integer time steps (corresponding to layer indices), where intermediate features evolve across layers, we argue that effective KD requires aligning the entire feature dynamics between teacher and student models, which we call feature dynamics distillation (FDD). This alignment involves matching both the feature trajectory and its first-order derivative, rather than just the final states. Our approach extends the original KD objective with two additional loss terms: layer-wise feature KD, which matches discretized feature trajectory, and layer feature delta KD, which matches first-order changes in features across adjacent layers. Extensive experiments on various tasks validate the effectiveness of our distillation method."
}