@inproceedings{anshumann-etal-2025-sparse,
    title = "Sparse Logit Sampling: Accelerating Knowledge Distillation in {LLM}s",
    author = "Anshumann, Anshumann  and
      Zaidi, Mohd Abbas  and
      Kedia, Akhil  and
      Ahn, Jinwoo  and
      Kwon, Taehwak  and
      Lee, Kangwook  and
      Lee, Haejun  and
      Lee, Joohyung",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.885/",
    pages = "18085--18108",
    ISBN = "979-8-89176-251-0",
    abstract = "Knowledge distillation can be a cost-effective technique to distill knowledge in Large Language Models, if the teacher output logits can be pre-computed and cached. However, successfully applying this to pre-training remains largely unexplored. In this work, we prove that naive approaches for sparse knowledge distillation such as caching Top-K probabilities, while intuitive, provide biased estimates of teacher probability distribution to the student, resulting in suboptimal performance and calibration. We propose an importance-sampling-based method `Random Sampling Knowledge Distillation', which provides unbiased estimates, preserves the gradient in expectation, and requires storing significantly sparser logits. Our method enables faster training of student models with marginal overhead ({\ensuremath{<}}10{\%}) compared to cross-entropy based training, while maintaining competitive performance compared to full distillation, across a range of model sizes from 300M to 3B."
}