@inproceedings{pangakis-wolken-2024-knowledge,
    title = "Knowledge Distillation in Automated Annotation: Supervised Text Classification with {LLM}-Generated Training Labels",
    author = "Pangakis, Nicholas  and
      Wolken, Sam",
    editor = "Card, Dallas  and
      Field, Anjalie  and
      Hovy, Dirk  and
      Keith, Katherine",
    booktitle = "Proceedings of the Sixth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS 2024)",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.nlpcss-1.9/",
    doi = "10.18653/v1/2024.nlpcss-1.9",
    pages = "113--131",
    abstract = "Computational social science (CSS) practitioners often rely on human-labeled data to fine-tune supervised text classifiers. We assess the potential for researchers to augment or replace human-generated training data with surrogate training labels from generative large language models (LLMs). We introduce a recommended workflow and test this LLM application by replicating 14 classification tasks and measuring performance. We employ a novel corpus of English-language text classification data sets from recent CSS articles in high-impact journals. Because these data sets are stored in password-protected archives, our analyses are less prone to issues of contamination. For each task, we compare supervised classifiers fine-tuned using GPT-4 labels against classifiers fine-tuned with human annotations and against labels from GPT-4 and Mistral-7B with few-shot in-context learning. Our findings indicate that supervised classification models fine-tuned on LLM-generated labels perform comparably to models fine-tuned with labels from human annotators. Fine-tuning models using LLM-generated labels can be a fast, efficient and cost-effective method of building supervised text classifiers."
}