Abstract
Prior knowledge of item characteristics, such as difficulty and response time, without pretesting items can substantially save time and cost in high-standard test development. Using a variety of machine learning (ML) algorithms, the present study explored several (non-)linguistic features (such as Coh-Metrix indices) along with MPNet word embeddings to predict the difficulty and response time of a sample of medical test items. In both prediction tasks, the contribution of embeddings to models already containing other features was found to be extremely limited. Moreover, a comparison of feature importance scores across the two prediction tasks revealed that cohesion-based features were the strongest predictors of difficulty, while the prediction of response time was primarily dependent on length-related features.- Anthology ID:
- 2024.bea-1.48
- Volume:
- Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Ekaterina Kochmar, Marie Bexte, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Anaïs Tack, Victoria Yaneva, Zheng Yuan
- Venue:
- BEA
- SIG:
- SIGEDU
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 551–560
- Language:
- URL:
- https://aclanthology.org/2024.bea-1.48
- DOI:
- Cite (ACL):
- Mehrdad Yousefpoori-Naeim, Shayan Zargari, and Zahra Hatami. 2024. Using Machine Learning to Predict Item Difficulty and Response Time in Medical Tests. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 551–560, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- Using Machine Learning to Predict Item Difficulty and Response Time in Medical Tests (Yousefpoori-Naeim et al., BEA 2024)
- PDF:
- https://preview.aclanthology.org/ingestion-checklist/2024.bea-1.48.pdf