@inproceedings{sharma-etal-2026-council,
    title = "Council of {LLM}s: Evaluating Capability of Large Language Models to Annotate Propaganda",
    author = "Sharma, Vivek  and
      Jain, Shweta  and
      Shokri, Mohammad  and
      Levitan, Sarah Ita  and
      Filatova, Elena",
    editor = "Barnes, Jeremy  and
      Barriere, Valentin  and
      De Clercq, Orph{\'e}e  and
      Klinger, Roman  and
      Nouri, C{\'e}lia  and
      Nozza, Debora  and
      Singh, Pranaydeep",
    booktitle = "The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis ({WASSA} 2026)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.wassa-1.1/",
    pages = "1--12",
    ISBN = "979-8-89176-378-4",
    abstract = "Data annotation is essential for supervised natural language processing tasks but remains labor-intensive and expensive. Large language models (LLMs) have emerged as promising alternatives, capable of generating high-quality annotations either autonomously or in collaboration with human annotators. However their use in autonomous annotations is often questioned for their ethical take on subjective matters. This study investigates the effectiveness of LLMs in a autonomous, and hybrid annotation setups in propaganda detection. We evaluate GPT and open-source models on two datasets from different domains, namely, Propaganda Techniques Corpus (PTC) for news articles and the Journalist Media Bias on X (JMBX) for social media. Our results show that LLMs, in general, exhibit high recall but lower precision in detecting propaganda, often over-predicting persuasive content. Multi-annotator setups did not outperform the best models in single-annotator setting although it helped reasoning models boost their performance. Hybrid annotation, combining LLMs and human input, achieved the highest overall accuracy than LLM-only settings. We further analyze misclassifications and found that LLM have higher sensitivity towards certain propaganda techniques like loaded language, name calling, and doubt. Finally, using error typology analysis, we explore the reasoning provided on misclassifications by the LLM. Our result shows that although some studies report LLM outperforming manual annotations and it could prove useful in hybrid annotation, its incorporation in the human annotation pipeline must be implemented with caution."
}