Operational Alignment of Confidence-Based Flagging Methods in Automated Scoring

Corey Palermo, Troy Chen, Arianto Wibowo


Abstract
In hybrid scoring systems, confidence thresholds determine which responses receive human review. This study evaluates a relative (within-batch) thresholding method against an absolute benchmark across ten items. Results show near-perfect agreement and modest distributional differences, supporting the relative method’s validity as a scalable, operationally viable approach for flagging low-confidence responses.
Anthology ID:
2025.aimecon-sessions.6
Volume:
Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Coordinated Session Papers
Month:
October
Year:
2025
Address:
Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States
Editors:
Joshua Wilson, Christopher Ormerod, Magdalen Beiting Parrish
Venue:
AIME-Con
SIG:
Publisher:
National Council on Measurement in Education (NCME)
Note:
Pages:
56–60
Language:
URL:
https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.aimecon-sessions.6/
DOI:
Bibkey:
Cite (ACL):
Corey Palermo, Troy Chen, and Arianto Wibowo. 2025. Operational Alignment of Confidence-Based Flagging Methods in Automated Scoring. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Coordinated Session Papers, pages 56–60, Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME).
Cite (Informal):
Operational Alignment of Confidence-Based Flagging Methods in Automated Scoring (Palermo et al., AIME-Con 2025)
Copy Citation:
PDF:
https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.aimecon-sessions.6.pdf