Multiple Instance Learning for Content Feedback Localization without Annotation

Scott Hellman, William Murray, Adam Wiemerslage, Mark Rosenstein, Peter Foltz, Lee Becker, Marcia Derr

Abstract
Automated Essay Scoring (AES) can be used to automatically generate holistic scores with reliability comparable to human scoring. In addition, AES systems can provide formative feedback to learners, typically at the essay level. In contrast, we are interested in providing feedback specialized to the content of the essay, and specifically for the content areas required by the rubric. A key objective is that the feedback should be localized alongside the relevant essay text. An important step in this process is determining where in the essay the rubric designated points and topics are discussed. A natural approach to this task is to train a classifier using manually annotated data; however, collecting such data is extremely resource intensive. Instead, we propose a method to predict these annotation spans without requiring any labeled annotation data. Our approach is to consider AES as a Multiple Instance Learning (MIL) task. We show that such models can both predict content scores and localize content by leveraging their sentence-level score predictions. This capability arises despite never having access to annotation training data. Implications are discussed for improving formative feedback and explainable AES models.
Anthology ID:
2020.bea-1.3
Volume:
Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications
Month:
July
Year:
2020
Address:
Seattle, WA, USA → Online
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
30–40
Language:
URL:
https://aclanthology.org/2020.bea-1.3
DOI:
10.18653/v1/2020.bea-1.3
Bibkey:
Cite (ACL):
Scott Hellman, William Murray, Adam Wiemerslage, Mark Rosenstein, Peter Foltz, Lee Becker, and Marcia Derr. 2020. Multiple Instance Learning for Content Feedback Localization without Annotation. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 30–40, Seattle, WA, USA → Online. Association for Computational Linguistics.
Cite (Informal):
Multiple Instance Learning for Content Feedback Localization without Annotation (Hellman et al., BEA 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/auto-file-uploads/2020.bea-1.3.pdf
PDF Search