Predicting Initial Essay Quality Scores to Increase the Efficiency of Comparative Judgment Assessments
Michiel De Vrindt, Anaïs Tack, Renske Bouwer, Wim Van Den Noortgate, Marije Lesterhuis
Abstract
Comparative judgment (CJ) is a method that can be used to assess the writing quality of student essays based on repeated pairwise comparisons by multiple assessors. Although the assessment method is known to have high validity and reliability, it can be particularly inefficient, as assessors must make many judgments before the scores become reliable. Prior research has investigated methods to improve the efficiency of CJ, yet these methods introduce additional challenges, notably stemming from the initial lack of information at the start of the assessment, which is known as a cold-start problem. This paper reports on a study in which we predict the initial quality scores of essays to establish a warm start for CJ. To achieve this, we construct informative prior distributions for the quality scores based on the predicted initial quality scores. Through simulation studies, we demonstrate that our approach increases the efficiency of CJ: On average, assessors need to make 30% fewer judgments for each essay to reach an overall reliability level of 0.70.- Anthology ID:
- 2024.bea-1.12
- Volume:
- Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Ekaterina Kochmar, Marie Bexte, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Anaïs Tack, Victoria Yaneva, Zheng Yuan
- Venue:
- BEA
- SIG:
- SIGEDU
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 125–136
- Language:
- URL:
- https://aclanthology.org/2024.bea-1.12
- DOI:
- Cite (ACL):
- Michiel De Vrindt, Anaïs Tack, Renske Bouwer, Wim Van Den Noortgate, and Marije Lesterhuis. 2024. Predicting Initial Essay Quality Scores to Increase the Efficiency of Comparative Judgment Assessments. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 125–136, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- Predicting Initial Essay Quality Scores to Increase the Efficiency of Comparative Judgment Assessments (De Vrindt et al., BEA 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.bea-1.12.pdf