Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions
Hannah Craighead, Andrew Caines, Paula Buttery, Helen Yannakoudakis
Abstract
We address the task of automatically grading the language proficiency of spontaneous speech based on textual features from automatic speech recognition transcripts. Motivated by recent advances in multi-task learning, we develop neural networks trained in a multi-task fashion that learn to predict the proficiency level of non-native English speakers by taking advantage of inductive transfer between the main task (grading) and auxiliary prediction tasks: morpho-syntactic labeling, language modeling, and native language identification (L1). We encode the transcriptions with both bi-directional recurrent neural networks and with bi-directional representations from transformers, compare against a feature-rich baseline, and analyse performance at different proficiency levels and with transcriptions of varying error rates. Our best performance comes from a transformer encoder with L1 prediction as an auxiliary task. We discuss areas for improvement and potential applications for text-only speech scoring.- Anthology ID:
- 2020.acl-main.206
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2258–2269
- Language:
- URL:
- https://aclanthology.org/2020.acl-main.206
- DOI:
- 10.18653/v1/2020.acl-main.206
- Cite (ACL):
- Hannah Craighead, Andrew Caines, Paula Buttery, and Helen Yannakoudakis. 2020. Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2258–2269, Online. Association for Computational Linguistics.
- Cite (Informal):
- Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions (Craighead et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2020.acl-main.206.pdf