@inproceedings{campione-etal-2026-language,
    title = "Language-Based Detection of Adherence to Evidence-Based Psychotherapy Scripts",
    author = "Campione, Samuel  and
      Stade, Elizabeth  and
      Losavio, Stefanie  and
      Singhvi, Shreya  and
      Xuan, William  and
      Bui, Tony  and
      Martin Lopez, Maria  and
      Subrahmanya, Shashanka  and
      Schuhmann, Bailee  and
      Worley, Courtney  and
      Wiltsey Stirman, Shannon  and
      Eichstaedt, Johannes  and
      Schwartz, H. Andrew",
    editor = "Zirikly, Aya  and
      Bar, Kfir  and
      MacAvaney, Sean  and
      Ireland, Molly  and
      Ophir, Yaakov  and
      Atzil-Slonim, Dana  and
      Varadarajan, Vasudha  and
      Bedrick, Steven  and
      Desmet, Bart",
    booktitle = "Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology ({CLP}sych 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.clpsych-1.20/",
    pages = "250--257",
    ISBN = "979-8-89176-421-7",
    abstract = "Some psychotherapies, such as written exposure therapy for posttraumatic stress disorder, utilize ``scripts'' during parts of treatment, but verifying script adherence to ensure engagement of key mechanisms of change is a time-consuming step for therapy supervisors. Here, we formalize therapy script adherence as an NLP task, and evaluate several simple (text similarity) and more complex (few-shot LLM) approaches. Over 351 annotated therapist utterance-script pairs, we find text similarity approaches to be highly competitive with LLMs and produce fewer false positives. ROUGE-L recall achieves F1 = 0.973, and BLEU achieves F1 = 0.972 with full precision and zero false positives. GPT-5.2 achieves F1 = 0.935 and GPT-4o-mini achieves F1 = 0.876. Given that the text similarity techniques are multiple orders of magnitude less complex, our results underscore the ability for simpler NLP techniques to still be effective in the age of LLMs for tasks that are more textual in nature, suggesting that aspects of therapist fidelity to evidence-based treatments can be assessed without using cloud API calls."
}