Content Modeling for Automated Oral Proficiency Scoring System

Su-Youn Yoon, Chong Min Lee

Abstract
We developed an automated oral proficiency scoring system for non-native English speakers’ spontaneous speech. Automated systems that score holistic proficiency are expected to assess a wide range of performance categories, and the content is one of the core performance categories. In order to assess the quality of the content, we trained a Siamese convolutional neural network (CNN) to model the semantic relationship between key points generated by experts and a test response. The correlation between human scores and Siamese CNN scores was comparable to human-human agreement (r=0.63), and it was higher than the baseline content features. The inclusion of Siamese CNN-based feature to the existing state-of-the-art automated scoring model achieved a small but statistically significant improvement. However, the new model suffered from score inflation for long atypical responses with serious content issues. We investigated the reasons of this score inflation by analyzing the associations with linguistic features and identifying areas strongly associated with the score errors.
Anthology ID:
W19-4441
Volume:
Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications
Month:
August
Year:
2019
Address:
Florence, Italy
Editors:
Helen Yannakoudakis, Ekaterina Kochmar, Claudia Leacock, Nitin Madnani, Ildikó Pilán, Torsten Zesch
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
394–401
Language:
URL:
https://aclanthology.org/W19-4441
DOI:
10.18653/v1/W19-4441
Bibkey:
Cite (ACL):
Su-Youn Yoon and Chong Min Lee. 2019. Content Modeling for Automated Oral Proficiency Scoring System. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 394–401, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Content Modeling for Automated Oral Proficiency Scoring System (Yoon & Lee, BEA 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-2023-videos/W19-4441.pdf
PDF Search