Overcoming the bottleneck in traditional assessments of verbal memory: Modeling human ratings and classifying clinical group membership
Chelsea Chandler, Peter W. Foltz, Jian Cheng, Jared C. Bernstein, Elizabeth P. Rosenfeld, Alex S. Cohen, Terje B. Holmlund, Brita Elvevåg
Abstract
Verbal memory is affected by numerous clinical conditions and most neuropsychological and clinical examinations evaluate it. However, a bottleneck exists in such endeavors because traditional methods require expert human review, and usually only a couple of test versions exist, thus limiting the frequency of administration and clinical applications. The present study overcomes this bottleneck by automating the administration, transcription, analysis and scoring of story recall. A large group of healthy participants (n = 120) and patients with mental illness (n = 105) interacted with a mobile application that administered a wide range of assessments, including verbal memory. The resulting speech generated by participants when retelling stories from the memory task was transcribed using automatic speech recognition tools, which was compared with human transcriptions (overall word error rate = 21%). An assortment of surface-level and semantic language-based features were extracted from the verbal recalls. A final set of three features were used to both predict expert human ratings with a ridge regression model (r = 0.88) and to differentiate patients from healthy individuals with an ensemble of logistic regression classifiers (accuracy = 76%). This is the first ‘outside of the laboratory’ study to showcase the viability of the complete pipeline of automated assessment of verbal memory in naturalistic settings.- Anthology ID:
- W19-3016
- Volume:
- Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology
- Month:
- June
- Year:
- 2019
- Address:
- Minneapolis, Minnesota
- Editors:
- Kate Niederhoffer, Kristy Hollingshead, Philip Resnik, Rebecca Resnik, Kate Loveys
- Venue:
- CLPsych
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 137–147
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/W19-3016/
- DOI:
- 10.18653/v1/W19-3016
- Cite (ACL):
- Chelsea Chandler, Peter W. Foltz, Jian Cheng, Jared C. Bernstein, Elizabeth P. Rosenfeld, Alex S. Cohen, Terje B. Holmlund, and Brita Elvevåg. 2019. Overcoming the bottleneck in traditional assessments of verbal memory: Modeling human ratings and classifying clinical group membership. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, pages 137–147, Minneapolis, Minnesota. Association for Computational Linguistics.
- Cite (Informal):
- Overcoming the bottleneck in traditional assessments of verbal memory: Modeling human ratings and classifying clinical group membership (Chandler et al., CLPsych 2019)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/W19-3016.pdf