Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency
Xinhao Wang, Keelan Evanini, Matthew Mulholland, Yao Qian, James V. Bruno
Abstract
This study aims to build an automatic system for the detection of plagiarized spoken responses in the context of an assessment of English speaking proficiency for non-native speakers. Classification models were trained to distinguish between plagiarized and non-plagiarized responses with two different types of features: text-to-text content similarity measures, which are commonly used in the task of plagiarism detection for written documents, and speaking proficiency measures, which were specifically designed for spontaneous speech and extracted using an automated speech scoring system. The experiments were first conducted on a large data set drawn from an operational English proficiency assessment across multiple years, and the best classifier on this heavily imbalanced data set resulted in an F1-score of 0.761 on the plagiarized class. This system was then validated on operational responses collected from a single administration of the assessment and achieved a recall of 0.897. The results indicate that the proposed system can potentially be used to improve the validity of both human and automated assessment of non-native spoken English.- Anthology ID:
- W19-4445
- Volume:
- Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications
- Month:
- August
- Year:
- 2019
- Address:
- Florence, Italy
- Editors:
- Helen Yannakoudakis, Ekaterina Kochmar, Claudia Leacock, Nitin Madnani, Ildikó Pilán, Torsten Zesch
- Venue:
- BEA
- SIG:
- SIGEDU
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 435–443
- Language:
- URL:
- https://aclanthology.org/W19-4445
- DOI:
- 10.18653/v1/W19-4445
- Cite (ACL):
- Xinhao Wang, Keelan Evanini, Matthew Mulholland, Yao Qian, and James V. Bruno. 2019. Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 435–443, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency (Wang et al., BEA 2019)
- PDF:
- https://preview.aclanthology.org/corrections-2024-07/W19-4445.pdf