Toward Automatically Measuring Learner Ability from Human-Machine Dialog Interactions using Novel Psychometric Models

Vikram Ramanarayanan, Michelle LaMar

Abstract
While dialog systems have been widely deployed for computer-assisted language learning (CALL) and formative assessment systems in recent years, relatively limited work has been done with respect to the psychometrics and validity of these technologies in evaluating and providing feedback regarding student learning and conversational ability. This paper formulates a Markov decision process based measurement model, and applies it to text chat data collected from crowdsourced native and non-native English language speakers interacting with an automated dialog agent. We investigate how well the model measures speaker conversational ability, and find that it effectively captures the differences in how native and non-native speakers of English accomplish the dialog task. Such models could have important implications for CALL systems of the future that effectively combine dialog management with measurement of learner conversational ability in real-time.
Anthology ID:
W18-0512
Volume:
Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Joel Tetreault, Jill Burstein, Ekaterina Kochmar, Claudia Leacock, Helen Yannakoudakis
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
117–126
Language:
URL:
https://aclanthology.org/W18-0512
DOI:
10.18653/v1/W18-0512
Bibkey:
Cite (ACL):
Vikram Ramanarayanan and Michelle LaMar. 2018. Toward Automatically Measuring Learner Ability from Human-Machine Dialog Interactions using Novel Psychometric Models. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 117–126, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
Toward Automatically Measuring Learner Ability from Human-Machine Dialog Interactions using Novel Psychometric Models (Ramanarayanan & LaMar, BEA 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/W18-0512.pdf
PDF Search