Eric Ragan
Fixing paper assignments
- Please select all papers that do not belong to this person.
- Indicate below which author they should be assigned to.
TODO: "submit" and "cancel" buttons here
2025
That Ain’t Right: Assessing LLM Performance on QA in African American and West African English Dialects
William Coggins
|
Jasmine McKenzie
|
Sangpil Youm
|
Pradham Mummaleti
|
Juan Gilbert
|
Eric Ragan
|
Bonnie J Dorr
Proceedings of the 9th Widening NLP Workshop
As Large Language Models (LLMs) gain mainstream public usage, understanding how users interact with them becomes increasingly important. Limited variety in training data raises concerns about LLM reliability across different language inputs. To explore this, we test several LLMs using functionally equivalent prompts expressed in different English sublanguages. We frame this analysis using Question-Answer (QA) pairs, which allow us to detect and evaluate appropriate and anomalous model behavior. We contribute a cross-LLM testing method and a new QA dataset translated into AAVE and WAPE variants. Early results reveal a notable drop in accuracy for one sublanguage relative to the baseline.
Search
Fix data
Co-authors
- William Coggins 1
- Bonnie Dorr 1
- Juan Gilbert 1
- Jasmine McKenzie 1
- Pradham Mummaleti 1
- show all...