Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts
Jenny Chim, Adam Tsakalidis, Dimitris Gkoumas, Dana Atzil-Slonim, Yaakov Ophir, Ayah Zirikly, Philip Resnik, Maria Liakata
Abstract
We present the overview of the CLPsych 2024 Shared Task, focusing on leveraging open source Large Language Models (LLMs) for identifying textual evidence that supports the suicidal risk level of individuals on Reddit. In particular, given a Reddit user, their pre- determined suicide risk level (‘Low’, ‘Mod- erate’ or ‘High’) and all of their posts in the r/SuicideWatch subreddit, we frame the task of identifying relevant pieces of text in their posts supporting their suicidal classification in two ways: (a) on the basis of evidence highlighting (extracting sub-phrases of the posts) and (b) on the basis of generating a summary of such evidence. We annotate a sample of 125 users and introduce evaluation metrics based on (a) BERTScore and (b) natural language inference for the two sub-tasks, respectively. Finally, we provide an overview of the system submissions and summarise the key findings.- Anthology ID:
- 2024.clpsych-1.15
- Original:
- 2024.clpsych-1.15v1
- Version 2:
- 2024.clpsych-1.15v2
- Volume:
- Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024)
- Month:
- March
- Year:
- 2024
- Address:
- St. Julians, Malta
- Editors:
- Andrew Yates, Bart Desmet, Emily Prud’hommeaux, Ayah Zirikly, Steven Bedrick, Sean MacAvaney, Kfir Bar, Molly Ireland, Yaakov Ophir
- Venues:
- CLPsych | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 177–190
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2024.clpsych-1.15/
- DOI:
- Cite (ACL):
- Jenny Chim, Adam Tsakalidis, Dimitris Gkoumas, Dana Atzil-Slonim, Yaakov Ophir, Ayah Zirikly, Philip Resnik, and Maria Liakata. 2024. Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 177–190, St. Julians, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts (Chim et al., CLPsych 2024)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2024.clpsych-1.15.pdf