Abstract
Technology is transforming Higher Education learning and teaching. This paper reports on a project to examine how and why automated content analysis could be used to assess precis writing by university students. We examine the case of one hundred and twenty-two summaries written by computer science freshmen. The texts, which had been hand scored using a teacher-designed rubric, were autoscored using the Natural Language Processing software, PyrEval. Pearson’s correlation coefficient and Spearman rank correlation were used to analyze the relationship between the teacher score and the PyrEval score for each summary. Three content models automatically constructed by PyrEval from different sets of human reference summaries led to consistent correlations, showing that the approach is reliable. Also observed was that, in cases where the focus of student assessment centers on formative feedback, categorizing the PyrEval scores by examining the average and standard deviations could lead to novel interpretations of their relationships. It is suggested that this project has implications for the ways in which automated content analysis could be used to help university students improve their summarization skills.- Anthology ID:
- W18-0531
- Volume:
- Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications
- Month:
- June
- Year:
- 2018
- Address:
- New Orleans, Louisiana
- Editors:
- Joel Tetreault, Jill Burstein, Ekaterina Kochmar, Claudia Leacock, Helen Yannakoudakis
- Venue:
- BEA
- SIG:
- SIGEDU
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 264–272
- Language:
- URL:
- https://aclanthology.org/W18-0531
- DOI:
- 10.18653/v1/W18-0531
- Cite (ACL):
- Yanjun Gao, Patricia M. Davies, and Rebecca J. Passonneau. 2018. Automated Content Analysis: A Case Study of Computer Science Student Summaries. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 264–272, New Orleans, Louisiana. Association for Computational Linguistics.
- Cite (Informal):
- Automated Content Analysis: A Case Study of Computer Science Student Summaries (Gao et al., BEA 2018)
- PDF:
- https://preview.aclanthology.org/ingest-bitext-workshop/W18-0531.pdf