Abstract
Counterspeech has been proposed as a solution to the proliferation of online hate. Research has shown that natural language processing (NLP) approaches could generate such counterspeech automatically, but there are competing ideas for how NLP models might be used for this task and a variety of evaluation metrics whose relationship to one another is unclear. We test three different approaches and collect ratings of the generated counterspeech for 1,740 tweet-participant pairs to systematically compare the counterspeech on three aspects: quality, effectiveness and user preferences. We examine which model performs best at which metric and which aspects of counterspeech predict user preferences. A free-form text generation approach using ChatGPT performs the most consistently well, though its generations are occasionally unspecific and repetitive. In our experiment, participants’ preferences for counterspeech are predicted by the quality of the counterspeech, not its perceived effectiveness. The results can help future research approach counterspeech evaluation more systematically.- Anthology ID:
- 2023.cs4oa-1.5
- Volume:
- Proceedings of the 1st Workshop on CounterSpeech for Online Abuse (CS4OA)
- Month:
- September
- Year:
- 2023
- Address:
- Prague, Czechia
- Editors:
- Yi-Ling Chung, Helena Bonaldi, Gavin Abercrombie, Marco Guerini
- Venues:
- CS4OA | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 62–71
- Language:
- URL:
- https://aclanthology.org/2023.cs4oa-1.5
- DOI:
- Cite (ACL):
- Yi Zheng, Björn Ross, and Walid Magdy. 2023. What Makes Good Counterspeech? A Comparison of Generation Approaches and Evaluation Metrics. In Proceedings of the 1st Workshop on CounterSpeech for Online Abuse (CS4OA), pages 62–71, Prague, Czechia. Association for Computational Linguistics.
- Cite (Informal):
- What Makes Good Counterspeech? A Comparison of Generation Approaches and Evaluation Metrics (Zheng et al., CS4OA-WS 2023)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2023.cs4oa-1.5.pdf