Abstract
Text classifiers are applied at scale in the form of one-size-fits-all solutions. Nevertheless, many studies show that classifiers are biased regarding different languages and dialects. When measuring and discovering these biases, some gaps present themselves and should be addressed. First, “Does language, dialect, and topical content vary across geographical regions?” and secondly “If there are differences across the regions, do they impact model performance?”. We introduce a novel dataset called GeoOLID with more than 14 thousand examples across 15 geographically and demographically diverse cities to address these questions. We perform a comprehensive analysis of geographical-related content and their impact on performance disparities of offensive language detection models. Overall, we find that current models do not generalize across locations. Likewise, we show that while offensive language models produce false positives on African American English, model performance is not correlated with each city’s minority population proportions. Warning: This paper contains offensive language.- Anthology ID:
- 2022.coling-1.574
- Volume:
- Proceedings of the 29th International Conference on Computational Linguistics
- Month:
- October
- Year:
- 2022
- Address:
- Gyeongju, Republic of Korea
- Editors:
- Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, Seung-Hoon Na
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 6600–6616
- Language:
- URL:
- https://aclanthology.org/2022.coling-1.574
- DOI:
- Cite (ACL):
- Brandon Lwowski, Paul Rad, and Anthony Rios. 2022. Measuring Geographic Performance Disparities of Offensive Language Classifiers. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6600–6616, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
- Cite (Informal):
- Measuring Geographic Performance Disparities of Offensive Language Classifiers (Lwowski et al., COLING 2022)
- PDF:
- https://preview.aclanthology.org/fix-dup-bibkey/2022.coling-1.574.pdf
- Data
- OLID