@inproceedings{blodgett-etal-2025-illuminating,
    title = "Illuminating Logical Fallacies with the {CAMPFIRE} Corpus",
    author = "Blodgett, Austin  and
      Bonial, Claire  and
      Pellegrin, Taylor A.  and
      Torgbi, Melissa  and
      Tayyar Madabushi, Harish",
    editor = "Peng, Siyao  and
      Rehbein, Ines",
    booktitle = "Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.law-1.24/",
    pages = "285--296",
    ISBN = "979-8-89176-262-6",
    abstract = "Misinformation detection remains today a challenging task for both annotators and computer systems. While there are many known markers of misinformation{---}e.g., logical fallacies, propaganda techniques, and improper use of sources{---}labeling these markers in practice has been shown to produce low agreement as it requires annotators to make several subjective judgments and rely on their own knowledge, external to the text, which may vary between annotators. In this work, we address these challenges with a collection of linguistically-inspired litmus tests. We annotate a schema of 25 logical fallacies, each of which is defined with rigorous tests applied during annotation. Our annotation methodology results in a comparatively high IAA on this task: Cohen{'}s kappa in the range .69-.86. We release a corpus of 12 documents from various domains annotated with fallacy labels. Additionally, we experiment with a large language model baseline showing that the largest, most advanced models struggle on this challenging task, achieving an F1-score with our gold standard of .08 when excluding non-fallacious examples, compared to human performance of .59-.73. However, we find that prompting methodologies requiring the model to work through our litmus tests improves performance. Our work contributes a robust fallacy annotation schema and annotated corpus, which advance capabilities in this critical research area."
}