Abstract
Previous work has shown that automated essay scoring systems, in particular machine learning-based systems, are not capable of assessing the quality of essays, but are relying on essay length, a factor irrelevant to writing proficiency. In this work, we first show that state-of-the-art systems, recent neural essay scoring systems, might be also influenced by the correlation between essay length and scores in a standard dataset. In our evaluation, a very simple neural model shows the state-of-the-art performance on the standard dataset. To consider essay content without taking essay length into account, we introduce a simple neural model assessing the similarity of content between an input essay and essays assigned different scores. This neural model achieves performance comparable to the state of the art on a standard dataset as well as on a second dataset. Our findings suggest that neural essay scoring systems should consider the characteristics of datasets to focus on text quality.- Anthology ID:
- 2021.sustainlp-1.4
- Volume:
- Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Virtual
- Editors:
- Nafise Sadat Moosavi, Iryna Gurevych, Angela Fan, Thomas Wolf, Yufang Hou, Ana Marasović, Sujith Ravi
- Venue:
- sustainlp
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 32–38
- Language:
- URL:
- https://aclanthology.org/2021.sustainlp-1.4
- DOI:
- 10.18653/v1/2021.sustainlp-1.4
- Cite (ACL):
- Sungho Jeon and Michael Strube. 2021. Countering the Influence of Essay Length in Neural Essay Scoring. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pages 32–38, Virtual. Association for Computational Linguistics.
- Cite (Informal):
- Countering the Influence of Essay Length in Neural Essay Scoring (Jeon & Strube, sustainlp 2021)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2021.sustainlp-1.4.pdf
- Code
- sdeva14/sustai21-counter-neural-essay-length
- Data
- ASAP-AES