@inproceedings{ghanadian-etal-2023-chatgpt,
    title = "{C}hat{GPT} for Suicide Risk Assessment on Social Media: Quantitative Evaluation of Model Performance, Potentials and Limitations",
    author = "Ghanadian, Hamideh  and
      Nejadgholi, Isar  and
      Al Osman, Hussein",
    editor = "Barnes, Jeremy  and
      De Clercq, Orph{\'e}e  and
      Klinger, Roman",
    booktitle = "Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, {\&} Social Media Analysis",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2023.wassa-1.16/",
    doi = "10.18653/v1/2023.wassa-1.16",
    pages = "172--183",
    abstract = "This paper presents a novel framework for quantitatively evaluating the interactive ChatGPT model in the context of suicidality assessment from social media posts, utilizing the University of Maryland Reddit suicidality dataset. We conduct a technical evaluation of ChatGPT`s performance on this task using Zero-Shot and Few-Shot experiments and compare its results with those of two fine-tuned transformer-based models. Additionally, we investigate the impact of different temperature parameters on ChatGPT`s response generation and discuss the optimal temperature based on the inconclusiveness rate of ChatGPT. Our results indicate that while ChatGPT attains considerable accuracy in this task, transformer-based models fine-tuned on human-annotated datasets exhibit superior performance. Moreover, our analysis sheds light on how adjusting the ChatGPT`s hyperparameters can improve its ability to assist mental health professionals in this critical task."
}