@inproceedings{sermsri-panboonyuen-2026-gatekd,
    title = "{G}ate{KD}: Confidence-Gated Closed-Loop Distillation for Robust Reasoning",
    author = "Sermsri, Kasidit  and
      Panboonyuen, Teerapong",
    editor = "Chang, Kai-Wei  and
      Mehrabi, Ninareh  and
      Krishna, Satyapriya  and
      Das, Anubrata  and
      Dhamala, Jwala  and
      Cao, Yang Trista  and
      Kumarage, Tharindu  and
      Ramakrishna, Anil  and
      Christodoulopoulos, Christos  and
      Wan, Yixin  and
      Galystan, Aram  and
      Kumar, Anoop  and
      Gupta, Rahul",
    booktitle = "Proceedings of the 6th Workshop on Trustworthy {NLP} ({T}rust{NLP} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.trustnlp-main.49/",
    pages = "637--652",
    ISBN = "979-8-89176-418-7",
    abstract = "Distilling multi-step reasoning abilities from large language models (LLMs) into compact student models remains challenging due to noisy rationales, hallucinated supervision, and static teacher{--}student interactions. Existing reasoning distillation methods, including mentor-based approaches, predominantly operate in an open-loop manner, implicitly assuming uniform teacher reliability and consequently propagating erroneous intermediate reasoning. We propose \textbf{GateKD}, a confidence-gated closed-loop distillation framework that enables robust reasoning transfer by treating the teacher as a dynamic gatekeeper rather than a static oracle. GateKD introduces three complementary mechanisms: (i) confidence-gated soft supervision that selectively distills reliable predictive signals, (ii) gated hidden-state evolution that aligns intermediate representations only when teacher confidence is high, and (iii) reliability-filtered attention distillation that preserves stable reasoning structures while suppressing noisy patterns. These components jointly form a closed feedback loop in which teacher confidence continuously modulates the distillation process, reducing hallucination transfer and stabilizing student reasoning. Extensive experiments across commonsense, logical, and symbolic reasoning benchmarks, using T5 and Flan-T5 backbones of varying sizes, demonstrate that GateKD consistently outperforms strong open-loop distillation baselines. Notably, GateKD yields substantial gains in logical and symbolic reasoning, remains robust under low-resource distillation settings, and shows clear performance degradation when any gating component is removed. Our results highlight that confidence-gated closed-loop supervision is critical for building reliable and scalable small reasoning models."
}