Abstract
To reveal ableism (i.e., bias against persons with disabilities) in large language models (LLMs), we introduce a novel approach involving multi-turn conversations, enabling a comparative assessment. Initially, we prompt the LLM to elaborate short biographies, followed by a request to incorporate information about a disability. Finally, we employ several methods to identify the top words that distinguish the disability-integrated biographies from those without. This comparative setting helps us uncover how LLMs handle disability-related information and reveal underlying biases. We observe that LLMs tend to highlight disabilities in a manner that can be perceived as patronizing or as implying that overcoming challenges is unexpected due to the disability.- Anthology ID:
- 2024.nlp4pi-1.18
- Volume:
- Proceedings of the Third Workshop on NLP for Positive Impact
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Daryna Dementieva, Oana Ignat, Zhijing Jin, Rada Mihalcea, Giorgio Piatti, Joel Tetreault, Steven Wilson, Jieyu Zhao
- Venue:
- NLP4PI
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 202–210
- Language:
- URL:
- https://aclanthology.org/2024.nlp4pi-1.18
- DOI:
- 10.18653/v1/2024.nlp4pi-1.18
- Cite (ACL):
- Guojun Wu and Sarah Ebling. 2024. Investigating Ableism in LLMs through Multi-turn Conversation. In Proceedings of the Third Workshop on NLP for Positive Impact, pages 202–210, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- Investigating Ableism in LLMs through Multi-turn Conversation (Wu & Ebling, NLP4PI 2024)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2024.nlp4pi-1.18.pdf