Human-in-the-Loop Hate Speech Classification in a Multilingual Context
Ana Kotarcic, Dominik Hangartner, Fabrizio Gilardi, Selina Kurer, Karsten Donnay
Abstract
The shift of public debate to the digital sphere has been accompanied by a rise in online hate speech. While many promising approaches for hate speech classification have been proposed, studies often focus only on a single language, usually English, and do not address three key concerns: post-deployment performance, classifier maintenance and infrastructural limitations. In this paper, we introduce a new human-in-the-loop BERT-based hate speech classification pipeline and trace its development from initial data collection and annotation all the way to post-deployment. Our classifier, trained using data from our original corpus of over 422k examples, is specifically developed for the inherently multilingual setting of Switzerland and outperforms with its F1 score of 80.5 the currently best-performing BERT-based multilingual classifier by 5.8 F1 points in German and 3.6 F1 points in French. Our systematic evaluations over a 12-month period further highlight the vital importance of continuous, human-in-the-loop classifier maintenance to ensure robust hate speech classification post-deployment.- Anthology ID:
- 2022.findings-emnlp.548
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 7414–7442
- Language:
- URL:
- https://aclanthology.org/2022.findings-emnlp.548
- DOI:
- Cite (ACL):
- Ana Kotarcic, Dominik Hangartner, Fabrizio Gilardi, Selina Kurer, and Karsten Donnay. 2022. Human-in-the-Loop Hate Speech Classification in a Multilingual Context. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7414–7442, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Human-in-the-Loop Hate Speech Classification in a Multilingual Context (Kotarcic et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2022.findings-emnlp.548.pdf