Abstract
We investigate in this paper how distributions of occupations with respect to gender is reflected in pre-trained language models. Such distributions are not always aligned to normative ideals, nor do they necessarily reflect a descriptive assessment of reality. In this paper, we introduce an approach for measuring to what degree pre-trained language models are aligned to normative and descriptive occupational distributions. To this end, we use official demographic information about gender–occupation distributions provided by the national statistics agencies of France, Norway, United Kingdom, and the United States. We manually generate template-based sentences combining gendered pronouns and nouns with occupations, and subsequently probe a selection of ten language models covering the English, French, and Norwegian languages. The scoring system we introduce in this work is language independent, and can be used on any combination of template-based sentences, occupations, and languages. The approach could also be extended to other dimensions of national census data and other demographic variables.- Anthology ID:
- 2023.eacl-main.164
- Volume:
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2242–2248
- Language:
- URL:
- https://aclanthology.org/2023.eacl-main.164
- DOI:
- 10.18653/v1/2023.eacl-main.164
- Cite (ACL):
- Samia Touileb, Lilja Øvrelid, and Erik Velldal. 2023. Measuring Normative and Descriptive Biases in Language Models Using Census Data. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2242–2248, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Measuring Normative and Descriptive Biases in Language Models Using Census Data (Touileb et al., EACL 2023)
- PDF:
- https://preview.aclanthology.org/revert-3132-ingestion-checklist/2023.eacl-main.164.pdf