@inproceedings{zhang-etal-2026-fast,
    title = "Fast and Effective On-Policy Distillation from Reasoning Prefixes",
    author = "Zhang, Dongxu  and
      Yang, Zhichao  and
      Janghorbani, Sepehr  and
      Han, Jun  and
      II, Andrew Ressler  and
      Qian, Qian  and
      Lyng, Gregory D  and
      Batra, Sanjit Singh  and
      Tillman, Robert E.",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1276/",
    pages = "25553--25569",
    ISBN = "979-8-89176-395-1",
    abstract = "On-policy distillation (OPD), which samples trajectories from the student model and supervises them with a teacher at the token-level, avoids relying solely on verifiable terminal rewards and can yield better generalization than off-policy distillation. However, OPD requires expensive on-the-fly sampling of the student policy during training, which substantially increases training cost, especially for responses with long reasoning traces. Our initial analysis shows that, during OPD, training signals are stronger in the prefix of each output reasoning trace, and that even a short teacher-generated prefix can significantly help the student produce the correct answer. Motivated by these observations, we propose a simple yet effective modification of OPD: we apply the distillation objective only to prefixes of student-generated outputs and terminate each sampling early during distillation. Experiments on a suite of AI-for-Math and out-of-domain reasoning benchmarks show that on-policy prefix distillation matches the performance of full OPD in long reasoning outputs while reducing training FLOP by 2x{--}40x."
}