@inproceedings{soliar-etal-2026-using,
    title = "Using Interaction Log Data to Evaluate and Improve Feedback Accuracy in an Intelligent Language Tutoring System",
    author = "Soliar, Mariia  and
      Colling, Leona  and
      Bodnar, Stephen  and
      Meurers, Detmar",
    editor = "Kochmar, Ekaterina  and
      Alhafni, Bashar  and
      Bann{\`o}, Stefano  and
      Bexte, Marie  and
      Burstein, Jill  and
      Horbach, Andrea  and
      Laarmann-Quante, Ronja  and
      Tack, Anais  and
      Yaneva, Victoria  and
      Yuan, Zheng",
    booktitle = "Proceedings of the 21st Workshop on Innovative Use of {NLP} for Building Educational Applications ({BEA} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.bea-1.14/",
    pages = "190--207",
    ISBN = "979-8-89176-409-5",
    abstract = "Intelligent Tutoring Systems (ITS) can record learner interactions in fine-grained detail at scale. This opens the door to data-driven methods for investigating system performance and identifying points for improvement. In this paper, we draw on authentic log data from an English language ITS (N{\_}logs = 5646, N{\_}students = 368) to investigate the performance of its feedback algorithm. In step 1 of our analysis, we profiled feedback accuracy by exploring how well the system provided error-specific feedback to malformed student answers in gap-filling grammar exercises using an expert-created set of feedback generation rules. We then identified frequently occurring student errors that triggered incorrect or unspecific feedback and refined the rule set used to detect and respond to these errors with correct specific feedback. In step 2, we validated the rule modifications on an unseen dataset. Comparing the performance of the initial and updated rule sets, we find significant improvement that generalizes to unseen data. Our study thus illustrates how an empirical evaluation of authentic data can complement feedback creators' expertise by informing rule refinement decisions that yield significant and generalizable performance improvements to feedback in ITS systems."
}