@inproceedings{pustet-etal-2024-detection,
    title = "Detection of Conspiracy Theories Beyond Keyword Bias in {G}erman-Language Telegram Using Large Language Models",
    author = "Pustet, Milena  and
      Steffen, Elisabeth  and
      Mihaljevic, Helena",
    editor = {Chung, Yi-Ling  and
      Talat, Zeerak  and
      Nozza, Debora  and
      Plaza-del-Arco, Flor Miriam  and
      R{\"o}ttger, Paul  and
      Mostafazadeh Davani, Aida  and
      Calabrese, Agostina},
    booktitle = "Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024)",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.woah-1.2/",
    doi = "10.18653/v1/2024.woah-1.2",
    pages = "13--27",
    abstract = "The automated detection of conspiracy theories online typically relies on supervised learning. However, creating respective training data requires expertise, time and mental resilience, given the often harmful content. Moreover, available datasets are predominantly in English and often keyword-based, introducing a token-level bias into the models. Our work addresses the task of detecting conspiracy theories in German Telegram messages. We compare the performance of supervised fine-tuning approaches using BERT-like models with prompt-based approaches using Llama2, GPT-3.5, and GPT-4 which require little or no additional training data. We use a dataset of {\ensuremath{\sim}}4, 000 messages collected during the COVID-19 pandemic, without the use of keyword filters.Our findings demonstrate that both approaches can be leveraged effectively: For supervised fine-tuning, we report an F1 score of {\ensuremath{\sim}} 0.8 for the positive class, making our model comparable to recent models trained on keyword-focused English corpora. We demonstrate our model{'}s adaptability to intra-domain temporal shifts, achieving F1 scores of {\ensuremath{\sim}}0.7. Among prompting variants, the best model is GPT-4, achieving an F1 score of {\ensuremath{\sim}}0.8 for the positive class in a zero-shot setting and equipped with a custom conspiracy theory definition."
}