In Data We Trust: A Critical Analysis of Hate Speech Detection Datasets

Kosisochukwu Madukwe, Xiaoying Gao, Bing Xue


Abstract
Recently, a few studies have discussed the limitations of datasets collected for the task of detecting hate speech from different viewpoints. We intend to contribute to the conversation by providing a consolidated overview of these issues pertaining to the data that debilitate research in this area. Specifically, we discuss how the varying pre-processing steps and the format for making data publicly available result in highly varying datasets that make an objective comparison between studies difficult and unfair. There is currently no study (to the best of our knowledge) focused on comparing the attributes of existing datasets for hate speech detection, outlining their limitations and recommending approaches for future research. This work intends to fill that gap and become the one-stop shop for information regarding hate speech datasets.
Anthology ID:
2020.alw-1.18
Volume:
Proceedings of the Fourth Workshop on Online Abuse and Harms
Month:
November
Year:
2020
Address:
Online
Venue:
ALW
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
150–161
Language:
URL:
https://aclanthology.org/2020.alw-1.18
DOI:
10.18653/v1/2020.alw-1.18
Bibkey:
Cite (ACL):
Kosisochukwu Madukwe, Xiaoying Gao, and Bing Xue. 2020. In Data We Trust: A Critical Analysis of Hate Speech Detection Datasets. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 150–161, Online. Association for Computational Linguistics.
Cite (Informal):
In Data We Trust: A Critical Analysis of Hate Speech Detection Datasets (Madukwe et al., ALW 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2020.alw-1.18.pdf
Video:
 https://slideslive.com/38939521
Data
CIFAR-10CIFAR-100