Abstract
Increasingly, language models and machine translation are becoming valuable tools to help people communicate with others from diverse cultural backgrounds. However, current language models lack cultural awareness because they are trained on data representing only the culture within the dataset. This presents a problem in the context of hate speech classification, where cultural awareness is especially critical. This study aims to quantify the cultural insensitivity of three monolingual (Korean, English, Arabic) hate speech classifiers by evaluating their performance on translated datasets from the other two languages. Our research has revealed that hate speech classifiers evaluated on datasets from other cultures yield significantly lower F1 scores, up to almost 50%. In addition, they produce considerably higher false negative rates, with a magnitude up to five times greater, demonstrating the extent of the cultural gap. The study highlights the severity of cultural insensitivity of language models in hate speech classification.- Anthology ID:
- 2023.c3nlp-1.5
- Volume:
- Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Sunipa Dev, Vinodkumar Prabhakaran, David Adelani, Dirk Hovy, Luciana Benotti
- Venue:
- C3NLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 35–46
- Language:
- URL:
- https://aclanthology.org/2023.c3nlp-1.5
- DOI:
- 10.18653/v1/2023.c3nlp-1.5
- Cite (ACL):
- Nayeon Lee, Chani Jung, and Alice Oh. 2023. Hate Speech Classifiers are Culturally Insensitive. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 35–46, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Hate Speech Classifiers are Culturally Insensitive (Lee et al., C3NLP 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-3/2023.c3nlp-1.5.pdf