What If Ground Truth Is Subjective? Personalized Deep Neural Hate Speech Detection
Kamil Kanclerz, Marcin Gruza, Konrad Karanowski, Julita Bielaniewicz, Piotr Milkowski, Jan Kocon, Przemyslaw Kazienko
Abstract
A unified gold standard commonly exploited in natural language processing (NLP) tasks requires high inter-annotator agreement. However, there are many subjective problems that should respect users individual points of view. Therefore in this paper, we evaluate three different personalized methods on the task of hate speech detection. The user-centered techniques are compared to the generalizing baseline approach. We conduct our experiments on three datasets including single-task and multi-task hate speech detection. For validation purposes, we introduce a new data-split strategy, preventing data leakage between training and testing. In order to better understand the model behavior for individual users, we carried out personalized ablation studies. Our experiments revealed that all models leveraging user preferences in any case provide significantly better results than most frequently used generalized approaches. This supports our overall observation that personalized models should always be considered in all subjective NLP tasks, including hate speech detection.- Anthology ID:
- 2022.nlperspectives-1.6
- Volume:
- Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022
- Month:
- June
- Year:
- 2022
- Address:
- Marseille, France
- Editors:
- Gavin Abercrombie, Valerio Basile, Sara Tonelli, Verena Rieser, Alexandra Uma
- Venue:
- NLPerspectives
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 37–45
- Language:
- URL:
- https://aclanthology.org/2022.nlperspectives-1.6
- DOI:
- Cite (ACL):
- Kamil Kanclerz, Marcin Gruza, Konrad Karanowski, Julita Bielaniewicz, Piotr Milkowski, Jan Kocon, and Przemyslaw Kazienko. 2022. What If Ground Truth Is Subjective? Personalized Deep Neural Hate Speech Detection. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages 37–45, Marseille, France. European Language Resources Association.
- Cite (Informal):
- What If Ground Truth Is Subjective? Personalized Deep Neural Hate Speech Detection (Kanclerz et al., NLPerspectives 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.nlperspectives-1.6.pdf
- Code
- clarin-pl/personalized-nlp