Utilizing Machine Learning to Predict Question Difficulty and Response Time for Enhanced Test Construction

Rishikesh Fulari, Jonathan Rusert

Abstract
In this paper, we present the details of ourcontribution to the BEA Shared Task on Automated Prediction of Item Difficulty and Response Time. Participants in this collaborativeeffort are tasked with developing models to predict the difficulty and response time of multiplechoice items within the medical domain. Theseitems are sourced from the United States Medical Licensing Examination® (USMLE®), asignificant medical assessment. In order toachieve this, we experimented with two featurization techniques, one using lingusitic features and the other using embeddings generated by BERT fine-tuned over MS-MARCOdataset. Further, we tried several different machine learning models such as Linear Regression, Decision Trees, KNN and Boosting models such as XGBoost and GBDT. We found thatout of all the models we experimented withRandom Forest Regressor trained on Linguisticfeatures gave the least root mean squared error.
Anthology ID:
2024.bea-1.45
Volume:
Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Ekaterina Kochmar, Marie Bexte, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Anaïs Tack, Victoria Yaneva, Zheng Yuan
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
528–533
Language:
URL:
https://aclanthology.org/2024.bea-1.45
DOI:
Bibkey:
Cite (ACL):
Rishikesh Fulari and Jonathan Rusert. 2024. Utilizing Machine Learning to Predict Question Difficulty and Response Time for Enhanced Test Construction. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 528–533, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Utilizing Machine Learning to Predict Question Difficulty and Response Time for Enhanced Test Construction (Fulari & Rusert, BEA 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-checklist/2024.bea-1.45.pdf
PDF Search