Xinhao Wang


2019

pdf
Using Rhetorical Structure Theory to Assess Discourse Coherence for Non-native Spontaneous Speech
Xinhao Wang | Binod Gyawali | James V. Bruno | Hillary R. Molloy | Keelan Evanini | Klaus Zechner
Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019

This study aims to model the discourse structure of spontaneous spoken responses within the context of an assessment of English speaking proficiency for non-native speakers. Rhetorical Structure Theory (RST) has been commonly used in the analysis of discourse organization of written texts; however, limited research has been conducted to date on RST annotation and parsing of spoken language, in particular, non-native spontaneous speech. Due to the fact that the measurement of discourse coherence is typically a key metric in human scoring rubrics for assessments of spoken language, we conducted research to obtain RST annotations on non-native spoken responses from a standardized assessment of academic English proficiency. Subsequently, automatic parsers were trained on these annotations to process non-native spontaneous speech. Finally, a set of features were extracted from automatically generated RST trees to evaluate the discourse structure of non-native spontaneous speech, which were then employed to further improve the validity of an automated speech scoring system.

pdf
Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency
Xinhao Wang | Keelan Evanini | Matthew Mulholland | Yao Qian | James V. Bruno
Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

This study aims to build an automatic system for the detection of plagiarized spoken responses in the context of an assessment of English speaking proficiency for non-native speakers. Classification models were trained to distinguish between plagiarized and non-plagiarized responses with two different types of features: text-to-text content similarity measures, which are commonly used in the task of plagiarism detection for written documents, and speaking proficiency measures, which were specifically designed for spontaneous speech and extracted using an automated speech scoring system. The experiments were first conducted on a large data set drawn from an operational English proficiency assessment across multiple years, and the best classifier on this heavily imbalanced data set resulted in an F1-score of 0.761 on the plagiarized class. This system was then validated on operational responses collected from a single administration of the assessment and achieved a recall of 0.897. The results indicate that the proposed system can potentially be used to improve the validity of both human and automated assessment of non-native spoken English.

2018

pdf bib
Word-Embedding based Content Features for Automated Oral Proficiency Scoring
Su-Youn Yoon | Anastassia Loukina | Chong Min Lee | Matthew Mulholland | Xinhao Wang | Ikkyu Choi
Proceedings of the Third Workshop on Semantic Deep Learning

In this study, we develop content features for an automated scoring system of non-native English speakers’ spontaneous speech. The features calculate the lexical similarity between the question text and the ASR word hypothesis of the spoken response, based on traditional word vector models or word embeddings. The proposed features do not require any sample training responses for each question, and this is a strong advantage since collecting question-specific data is an expensive task, and sometimes even impossible due to concerns about question exposure. We explore the impact of these new features on the automated scoring of two different question types: (a) providing opinions on familiar topics and (b) answering a question about a stimulus material. The proposed features showed statistically significant correlations with the oral proficiency scores, and the combination of new features with the speech-driven features achieved a small but significant further improvement for the latter question type. Further analyses suggested that the new features were effective in assigning more accurate scores for responses with serious content issues.

2017

pdf
Discourse Annotation of Non-native Spontaneous Spoken Responses Using the Rhetorical Structure Theory Framework
Xinhao Wang | James Bruno | Hillary Molloy | Keelan Evanini | Klaus Zechner
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

The availability of the Rhetorical Structure Theory (RST) Discourse Treebank has spurred substantial research into discourse analysis of written texts; however, limited research has been conducted to date on RST annotation and parsing of spoken language, in particular, non-native spontaneous speech. Considering that the measurement of discourse coherence is typically a key metric in human scoring rubrics for assessments of spoken language, we initiated a research effort to obtain RST annotations of a large number of non-native spoken responses from a standardized assessment of academic English proficiency. The resulting inter-annotator kappa agreements on the three different levels of Span, Nuclearity, and Relation are 0.848, 0.766, and 0.653, respectively. Furthermore, a set of features was explored to evaluate the discourse structure of non-native spontaneous speech based on these annotations; the highest performing feature resulted in a correlation of 0.612 with scores of discourse coherence provided by expert human raters.

pdf
SoccEval: An Annotation Schema for Rating Soccer Players
Jose Ramirez | Matthew Garber | Xinhao Wang
Proceedings of ACL 2017, Student Research Workshop

2015

pdf
Automated Scoring of Picture-based Story Narration
Swapna Somasundaran | Chong Min Lee | Martin Chodorow | Xinhao Wang
Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications

2014

pdf
Automatic detection of plagiarized spoken responses
Keelan Evanini | Xinhao Wang
Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications

pdf
Automated scoring of speaking items in an assessment for teachers of English as a Foreign Language
Klaus Zechner | Keelan Evanini | Su-Youn Yoon | Lawrence Davis | Xinhao Wang | Lei Chen | Chong Min Lee | Chee Wee Leong
Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications

2013

pdf
Coherence Modeling for the Automated Assessment of Spontaneous Spoken Responses
Xinhao Wang | Keelan Evanini | Klaus Zechner
Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

pdf
Automated Content Scoring of Spoken Responses in an Assessment for Teachers of English
Klaus Zechner | Xinhao Wang
Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications

2008

pdf
An Improved CRF based Chinese Language Processing System for SIGHAN Bakeoff 2007
Xihong Wu | Xiaojun Lin | Xinhao Wang | Chunyao Wu | Yaozhong Zhang | Dianhai Yu
Proceedings of the Sixth SIGHAN Workshop on Chinese Language Processing

pdf
Integrating Multi-level Linguistic Knowledge with a Unified Framework for Mandarin Speech Recognition
Xinhao Wang | Jiazhong Nie | Dingsheng Luo | Xihong Wu
Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing

2006

pdf
Chinese Word Segmentation with Maximum Entropy and N-gram Language Model
Xinhao Wang | Xiaojun Lin | Dianhai Yu | Hao Tian | Xihong Wu
Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing