@inproceedings{boateng-etal-2025-concept,
    title = "Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting",
    author = "Boateng, Emmanuel Aboah  and
      Becker, Cassiano O  and
      Asghar, Nabiha  and
      Walia, Kabir  and
      Srinivasan, Ashwin  and
      Nosakhare, Ehi  and
      Srinivasan, Soundararajan  and
      Dibia, Victor",
    editor = "Chen, Weizhu  and
      Yang, Yi  and
      Kachuee, Mohammad  and
      Fu, Xue-Yong",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2025.naacl-industry.52/",
    pages = "638--654",
    ISBN = "979-8-89176-194-0",
    abstract = "Hand-crafting high quality prompts to optimize the performance of language models is a complicated and labor-intensive process. Furthermore, when migrating to newer, smaller, or weaker models (possibly due to latency or cost gains), prompts need to be updated to re-optimize the task performance. We propose Concept Distillation (CD), an automatic prompt optimization technique for enhancing weaker models on complex tasks. CD involves: (1) collecting mistakes made by weak models with a base prompt (initialization), (2) using a strong model to generate reasons for these mistakes and create rules/concepts for weak models (induction), and (3) filtering these rules based on validation set performance and integrating them into the base prompt (deduction/verification). We evaluated CD on NL2Code and mathematical reasoning tasks, observing significant performance boosts for small and weaker language models. Notably, Mistral-7B`s accuracy on Multi-Arith increased by 20{\%}, and Phi-3-mini-3.8B`s accuracy on HumanEval rose by 34{\%}. Compared to other automated methods, CD offers an effective, cost-efficient strategy for improving weak models' performance on complex tasks and enables seamless workload migration across different language models without compromising performance."
}