@inproceedings{kian-etal-2025-using,
    title = "Using Linguistic Entrainment to Evaluate Large Language Models for Use in Cognitive Behavioral Therapy",
    author = "Kian, Mina  and
      Shrestha, Kaleen  and
      Fischer, Katrin  and
      Zhu, Xiaoyuan  and
      Ong, Jonathan  and
      Trehan, Aryan  and
      Wang, Jessica  and
      Chang, Gloria  and
      Arnold, S{\'e}b  and
      Mataric, Maja",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.430/",
    pages = "7724--7743",
    ISBN = "979-8-89176-195-7",
    abstract = "Entrainment, the responsive communication between interacting individuals, is a crucial process in building a strong relationship between a mental health therapist and their client, leading to positive therapeutic outcomes. However, so far entrainment has not been investigated as a measure of efficacy of large language models (LLMs) delivering mental health therapy. In this work, we evaluate the linguistic entrainment of an LLM (ChatGPT 3.5-turbo) in a mental health dialog setting. We first validate computational measures of linguistic entrainment with two measures of the quality of client self-disclosures: intimacy and engagement ($p < 0.05$). We then compare the linguistic entrainment of the LLM to trained therapists and non-expert online peer supporters in a cognitive behavioral therapy (CBT) setting. We show that the LLM is outperformed by humans with respect to linguistic entrainment ($p < 0.001$). These results support the need to be cautious in using LLMs out-of-the-box for mental health applications."
}