Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays
Zarah Weiss, Anja Riemenschneider, Pauline Schröter, Detmar Meurers
Abstract
Computational linguistic research on the language complexity of student writing typically involves human ratings as a gold standard. However, educational science shows that teachers find it difficult to identify and cleanly separate accuracy, different aspects of complexity, contents, and structure. In this paper, we therefore explore the use of computational linguistic methods to investigate how task-appropriate complexity and accuracy relate to the grading of overall performance, content performance, and language performance as assigned by teachers. Based on texts written by students for the official school-leaving state examination (Abitur), we show that teachers successfully assign higher language performance grades to essays with higher task-appropriate language complexity and properly separate this from content scores. Yet, accuracy impacts teacher assessment for all grading rubrics, also the content score, overemphasizing the role of accuracy. Our analysis is based on broad computational linguistic modeling of German language complexity and an innovative theory- and data-driven feature aggregation method inferring task-appropriate language complexity.- Anthology ID:
- W19-4404
- Volume:
- Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications
- Month:
- August
- Year:
- 2019
- Address:
- Florence, Italy
- Editors:
- Helen Yannakoudakis, Ekaterina Kochmar, Claudia Leacock, Nitin Madnani, Ildikó Pilán, Torsten Zesch
- Venue:
- BEA
- SIG:
- SIGEDU
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 30–45
- Language:
- URL:
- https://aclanthology.org/W19-4404
- DOI:
- 10.18653/v1/W19-4404
- Cite (ACL):
- Zarah Weiss, Anja Riemenschneider, Pauline Schröter, and Detmar Meurers. 2019. Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 30–45, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays (Weiss et al., BEA 2019)
- PDF:
- https://preview.aclanthology.org/landing_page/W19-4404.pdf