Measuring Optimal Challenge: Trajectory-Based Difficulty Alignment in Open-Ended Language Tutoring

Ziqi Shu, Shuman Wang, Michael Hardy


Abstract
Conversational English as a Foreign Language (EFL) tutoring relies on dynamically generated exercises rather than fixed item banks, so traditional difficulty estimation cannot verify whether a task is appropriately calibrated to a learner. We propose a framework that measures difficulty alignment directly from observable interactional behavior, classifying each exercise into one of three states (Under-Challenged, Optimally Challenged, or Over-Challenged) based on turn-level sequences of student attempts, errors, confusion, and tutor scaffolding. Using 1,566 exercises from the Teacher-Student Chatroom Corpus, we validate the classification against human annotation (Cohen’s kappa = 0.79 at the state level) and show that a learner’s cumulative trajectory of these states predicts success on subsequent exercises. Aggregating these predictions into a within-session capability-shift proxy, we find that sessions with higher proportions of over-challenging exercises systematically yield lower estimated shifts, while optimally challenging interactions are significantly associated with greater improvement than under-challenging ones — patterns consistent with Krashen’s Input Hypothesis.
Anthology ID:
2026.bea-1.46
Volume:
Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)
Month:
July
Year:
2026
Address:
San Diego, California, USA
Editors:
Ekaterina Kochmar, Bashar Alhafni, Stefano Bannò, Marie Bexte, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Anais Tack, Victoria Yaneva, Zheng Yuan
Venues:
BEA | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
651–667
Language:
URL:
https://preview.aclanthology.org/ingest-acl-workshops/2026.bea-1.46/
DOI:
Bibkey:
Cite (ACL):
Ziqi Shu, Shuman Wang, and Michael Hardy. 2026. Measuring Optimal Challenge: Trajectory-Based Difficulty Alignment in Open-Ended Language Tutoring. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), pages 651–667, San Diego, California, USA. Association for Computational Linguistics.
Cite (Informal):
Measuring Optimal Challenge: Trajectory-Based Difficulty Alignment in Open-Ended Language Tutoring (Shu et al., BEA 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-workshops/2026.bea-1.46.pdf