Automated Content Analysis: A Case Study of Computer Science Student Summaries

Yanjun Gao, Patricia M. Davies, Rebecca J. Passonneau


Abstract
Technology is transforming Higher Education learning and teaching. This paper reports on a project to examine how and why automated content analysis could be used to assess precis writing by university students. We examine the case of one hundred and twenty-two summaries written by computer science freshmen. The texts, which had been hand scored using a teacher-designed rubric, were autoscored using the Natural Language Processing software, PyrEval. Pearson’s correlation coefficient and Spearman rank correlation were used to analyze the relationship between the teacher score and the PyrEval score for each summary. Three content models automatically constructed by PyrEval from different sets of human reference summaries led to consistent correlations, showing that the approach is reliable. Also observed was that, in cases where the focus of student assessment centers on formative feedback, categorizing the PyrEval scores by examining the average and standard deviations could lead to novel interpretations of their relationships. It is suggested that this project has implications for the ways in which automated content analysis could be used to help university students improve their summarization skills.
Anthology ID:
W18-0531
Volume:
Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Joel Tetreault, Jill Burstein, Ekaterina Kochmar, Claudia Leacock, Helen Yannakoudakis
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
264–272
Language:
URL:
https://aclanthology.org/W18-0531
DOI:
10.18653/v1/W18-0531
Bibkey:
Cite (ACL):
Yanjun Gao, Patricia M. Davies, and Rebecca J. Passonneau. 2018. Automated Content Analysis: A Case Study of Computer Science Student Summaries. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 264–272, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
Automated Content Analysis: A Case Study of Computer Science Student Summaries (Gao et al., BEA 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-bitext-workshop/W18-0531.pdf