Identifying Biases in Large Language Model Assessment of Linguistically Diverse Texts
Lionel Hsien Meng, Shamya Karumbaiah, Vivek Saravanan, Daniel Bolt
Abstract
The development of Large Language Models (LLMs) to assess student text responses is rapidly progressing but evaluating whether LLMs equitably assess multilingual learner responses is an important precursor to adoption. Our study provides an example procedure for identifying and quantifying bias in LLM assessment of student essay responses.- Anthology ID:
- 2025.aimecon-wip.25
- Volume:
- Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Works in Progress
- Month:
- October
- Year:
- 2025
- Address:
- Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States
- Editors:
- Joshua Wilson, Christopher Ormerod, Magdalen Beiting Parrish
- Venue:
- AIME-Con
- SIG:
- Publisher:
- National Council on Measurement in Education (NCME)
- Note:
- Pages:
- 204–210
- Language:
- URL:
- https://preview.aclanthology.org/ingest-emnlp/2025.aimecon-wip.25/
- DOI:
- Cite (ACL):
- Lionel Hsien Meng, Shamya Karumbaiah, Vivek Saravanan, and Daniel Bolt. 2025. Identifying Biases in Large Language Model Assessment of Linguistically Diverse Texts. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Works in Progress, pages 204–210, Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME).
- Cite (Informal):
- Identifying Biases in Large Language Model Assessment of Linguistically Diverse Texts (Meng et al., AIME-Con 2025)
- PDF:
- https://preview.aclanthology.org/ingest-emnlp/2025.aimecon-wip.25.pdf