Jump-Starting Item Parameters for Adaptive Language Tests
Arya D. McCarthy, Kevin P. Yancey, Geoffrey T. LaFlair, Jesse Egbert, Manqian Liao, Burr Settles
Abstract
A challenge in designing high-stakes language assessments is calibrating the test item difficulties, either a priori or from limited pilot test data. While prior work has addressed ‘cold start’ estimation of item difficulties without piloting, we devise a multi-task generalized linear model with BERT features to jump-start these estimates, rapidly improving their quality with as few as 500 test-takers and a small sample of item exposures (≈6 each) from a large item bank (≈4,000 items). Our joint model provides a principled way to compare test-taker proficiency, item difficulty, and language proficiency frameworks like the Common European Framework of Reference (CEFR). This also enables new item difficulty estimates without piloting them first, which in turn limits item exposure and thus enhances test item security. Finally, using operational data from the Duolingo English Test, a high-stakes English proficiency test, we find that the difficulty estimates derived using this method correlate strongly with lexico-grammatical features that correlate with reading complexity.- Anthology ID:
- 2021.emnlp-main.67
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 883–899
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.67
- DOI:
- 10.18653/v1/2021.emnlp-main.67
- Cite (ACL):
- Arya D. McCarthy, Kevin P. Yancey, Geoffrey T. LaFlair, Jesse Egbert, Manqian Liao, and Burr Settles. 2021. Jump-Starting Item Parameters for Adaptive Language Tests. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 883–899, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Jump-Starting Item Parameters for Adaptive Language Tests (McCarthy et al., EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2021.emnlp-main.67.pdf