Abstract
Social biases are encoded in word embeddings. This presents a unique opportunity to study society historically and at scale, and a unique danger when embeddings are used in downstream applications. Here, we investigate the extent to which publicly-available word embeddings accurately reflect beliefs about certain kinds of people as measured via traditional survey methods. We find that biases found in word embeddings do, on average, closely mirror survey data across seventeen dimensions of social meaning. However, we also find that biases in embeddings are much more reflective of survey data for some dimensions of meaning (e.g. gender) than others (e.g. race), and that we can be highly confident that embedding-based measures reflect survey data only for the most salient biases.- Anthology ID:
- 2020.acl-main.405
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4392–4415
- Language:
- URL:
- https://aclanthology.org/2020.acl-main.405
- DOI:
- 10.18653/v1/2020.acl-main.405
- Cite (ACL):
- Kenneth Joseph and Jonathan Morgan. 2020. When do Word Embeddings Accurately Reflect Surveys on our Beliefs About People?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4392–4415, Online. Association for Computational Linguistics.
- Cite (Informal):
- When do Word Embeddings Accurately Reflect Surveys on our Beliefs About People? (Joseph & Morgan, ACL 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2020.acl-main.405.pdf
- Code
- kennyjoseph/embedding_impressions