Recent advances in large language models (LLMs) have enabled the creation of highly realistic digital patients across a broad range of clinical scenarios, yet systematic evaluation of such simulations remains challenging due to a lack of standardised methodology. This paper investigates the faithfulness of LLM-simulated patients within motivational interviewing contexts. We directly compare the properties of data generated by simulated and human patients given identical profiles, rather than relying on subjective user experiences. Our findings reveal that while simulated and human patients produce semantically similar content and engage with comparable topics, their modes of expression differ substantially. LLM-simulated patients struggle to reproduce the full complexity of human behaviours and attitudes. While human patients exhibit a mix of positive and negative responses, LLM patients skew toward uniformly ones.

Within Motivational Interviewing (MI), client utterances are coded as for or against a certain behaviour change, along with commitment strength; this is essential to ensure therapists soften rather than persisting goal-related actions in the face of resistance. Prior works in MI agents have been scripted or semi-scripted, limiting users’ natural language expressions. With the aim of automating the MI interactions, we propose and explore the task of automated identification of client motivational language. Employing Large Language Models (LLMs), we compare in-context learning (ICL) and instruction fine-tuning (IFT) with varying training sizes for this identification task. Our experiments show that both approaches can learn under low-resourced settings. Our results demonstrate that IFT, though cheaper, is more stable to prompt choice, and yields better performance with more data. Given the detected motivation, we further present an approach to the analysis of therapists’ strategies for balancing building rapport with clients with advancing the treatment plan. A framework of MI agents is developed using insights from the data and the psychotherapy literature.