Abstract
This paper presents our approach to the CLPsych 2024 shared task: utilizing large language models (LLMs) for finding supporting evidence about an individual’s suicide risk level in Reddit posts. Our framework is constructed around an LLM with knowledge self-generation and output refinement. The knowledge self-generation process produces task-related knowledge which is generated by the LLM and leads to accurate risk predictions. The output refinement process, later, with the selected best set of LLM-generated knowledge, refines the outputs by prompting the LLM repeatedly with different knowledge instances interchangeably. We achieved highly competitive results comparing to the top-performance participants with our official recall of 93.5%, recall–precision harmonic-mean of 92.3%, and mean consistency of 96.1%.- Anthology ID:
- 2024.clpsych-1.16
- Volume:
- Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024)
- Month:
- March
- Year:
- 2024
- Address:
- St. Julians, Malta
- Editors:
- Andrew Yates, Bart Desmet, Emily Prud’hommeaux, Ayah Zirikly, Steven Bedrick, Sean MacAvaney, Kfir Bar, Molly Ireland, Yaakov Ophir
- Venues:
- CLPsych | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 191–196
- Language:
- URL:
- https://aclanthology.org/2024.clpsych-1.16
- DOI:
- Cite (ACL):
- Vu Tran and Tomoko Matsui. 2024. Team ISM at CLPsych 2024: Extracting Evidence of Suicide Risk from Reddit Posts with Knowledge Self-Generation and Output Refinement using A Large Language Model. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 191–196, St. Julians, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Team ISM at CLPsych 2024: Extracting Evidence of Suicide Risk from Reddit Posts with Knowledge Self-Generation and Output Refinement using A Large Language Model (Tran & Matsui, CLPsych-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.clpsych-1.16.pdf