What If Ground Truth Is Subjective? Personalized Deep Neural Hate Speech Detection

Kamil Kanclerz, Marcin Gruza, Konrad Karanowski, Julita Bielaniewicz, Piotr Milkowski, Jan Kocon, Przemyslaw Kazienko


Abstract
A unified gold standard commonly exploited in natural language processing (NLP) tasks requires high inter-annotator agreement. However, there are many subjective problems that should respect users individual points of view. Therefore in this paper, we evaluate three different personalized methods on the task of hate speech detection. The user-centered techniques are compared to the generalizing baseline approach. We conduct our experiments on three datasets including single-task and multi-task hate speech detection. For validation purposes, we introduce a new data-split strategy, preventing data leakage between training and testing. In order to better understand the model behavior for individual users, we carried out personalized ablation studies. Our experiments revealed that all models leveraging user preferences in any case provide significantly better results than most frequently used generalized approaches. This supports our overall observation that personalized models should always be considered in all subjective NLP tasks, including hate speech detection.
Anthology ID:
2022.nlperspectives-1.6
Volume:
Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022
Month:
June
Year:
2022
Address:
Marseille, France
Venue:
NLPerspectives
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
37–45
Language:
URL:
https://aclanthology.org/2022.nlperspectives-1.6
DOI:
Bibkey:
Cite (ACL):
Kamil Kanclerz, Marcin Gruza, Konrad Karanowski, Julita Bielaniewicz, Piotr Milkowski, Jan Kocon, and Przemyslaw Kazienko. 2022. What If Ground Truth Is Subjective? Personalized Deep Neural Hate Speech Detection. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages 37–45, Marseille, France. European Language Resources Association.
Cite (Informal):
What If Ground Truth Is Subjective? Personalized Deep Neural Hate Speech Detection (Kanclerz et al., NLPerspectives 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/author-url/2022.nlperspectives-1.6.pdf
Code
 clarin-pl/personalized-nlp