@inproceedings{waheed-etal-2025-udistil,
    title = "u{D}istil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes",
    author = "Waheed, Abdul  and
      Kadaoui, Karima  and
      Raj, Bhiksha  and
      Abdul-Mageed, Muhammad",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.296/",
    pages = "5750--5767",
    ISBN = "979-8-89176-189-6",
    abstract = "Recent work on distilling Whisper{'}s knowledge into small models using pseudo-labels shows promising performance while reducing the size by up to 50{\%}. This results in small, efficient, and dedicated models. However, a critical step of distillation using pseudo-labels involves filtering high-quality predictions and using only those during training. This step requires ground truth labels to compare with and filter low-quality examples, making the process dependent on human labels. Additionally, the distillation process requires a large amount of data thereby limiting its applicability in low-resource settings. To address this, we propose a distillation framework that does not require \textit{any} labeled data. Through experimentation, we show that our best-distilled models outperform the teacher model by 5-7 WER points and are on par with or outperform similar supervised data filtering setups. When scaling the data, our models significantly outperform all zero-shot and supervised models. Our models are also 25-50{\%} more compute- and memory-efficient while maintaining performance equal to or better than that of the teacher model. For more details about our models, dataset, and other resources, please visit our GitHub page: \url{https://github.com/UBC-NLP/uDistilWhisper}."
}