Automated coherence measures fail to index thought disorder in individuals at risk for psychosis
Kasia Hitczenko, Henry Cowan, Vijay Mittal, Matthew Goldrick
Abstract
Thought disorder – linguistic disturbances including incoherence and derailment of topic – is seen in individuals both with and at risk for psychosis. Methods from computational linguistics have increasingly sought to quantify thought disorder to detect group differences between clinical populations and healthy controls. While previous work has been quite successful at these classification tasks, the lack of interpretability of the computational metrics has made it unclear whether they are in fact measuring thought disorder. In this paper, we dive into these measures to try to better understand what they reflect. While we find group differences between at-risk and healthy control populations, we also find that the measures mostly do not correlate with existing measures of thought disorder symptoms (what they are intended to measure), but rather correlate with surface properties of the speech (e.g., sentence length) and sociodemographic properties of the speaker (e.g., race). These results highlight the importance of considering interpretability and front and center as the field continues to grow. Ethical use of computational measures like those studied here – especially in the high-stakes context of clinical care – requires us to devote substantial attention to potential biases in our measures.- Anthology ID:
- 2021.clpsych-1.16
- Volume:
- Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access
- Month:
- June
- Year:
- 2021
- Address:
- Online
- Venue:
- CLPsych
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 129–150
- Language:
- URL:
- https://aclanthology.org/2021.clpsych-1.16
- DOI:
- 10.18653/v1/2021.clpsych-1.16
- Cite (ACL):
- Kasia Hitczenko, Henry Cowan, Vijay Mittal, and Matthew Goldrick. 2021. Automated coherence measures fail to index thought disorder in individuals at risk for psychosis. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 129–150, Online. Association for Computational Linguistics.
- Cite (Informal):
- Automated coherence measures fail to index thought disorder in individuals at risk for psychosis (Hitczenko et al., CLPsych 2021)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2021.clpsych-1.16.pdf