@inproceedings{greschner-klinger-2025-fearful,
    title = "Fearful Falcons and Angry Llamas: Emotion Category Annotations of Arguments by Humans and {LLM}s",
    author = "Greschner, Lynn  and
      Klinger, Roman",
    editor = {H{\"a}m{\"a}l{\"a}inen, Mika  and
      {\"O}hman, Emily  and
      Bizzoni, Yuri  and
      Miyagawa, So  and
      Alnajjar, Khalid},
    booktitle = "Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities",
    month = may,
    year = "2025",
    address = "Albuquerque, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2025.nlp4dh-1.52/",
    pages = "628--646",
    ISBN = "979-8-89176-234-3",
    abstract = "Arguments evoke emotions, influencing the effect of the argument itself. Not only the emotional intensity but also the category influences the argument`s effects, for instance, the willingness to adapt stances. While binary emotionality has been studied in argumentative texts, there is no work on discrete emotion categories (e.g., {\textquoteleft}anger') in such data. To fill this gap, we crowdsource subjective annotations of emotion categories in a German argument corpus and evaluate automatic LLM-based labeling methods. Specifically, we compare three prompting strategies (zero-shot, one-shot, chain-of-thought) on three large instruction-tuned language models (Falcon-7b-instruct, Llama-3.1-8B-instruct, GPT-4o-mini). We further vary the definition of the output space to be binary (is there emotionality in the argument?), closed-domain (which emotion from a given label set is in the argument?), or open-domain (which emotion is in the argument?). We find that emotion categories enhance the prediction of emotionality in arguments, emphasizing the need for discrete emotion annotations in arguments. Across all prompt settings and models, automatic predictions show a high recall but low precision for predicting anger and fear, indicating a strong bias toward negative emotions."
}