@inproceedings{kim-etal-2026-doctor,
    title = "The Doctor Will Agree With You Now: Sycophancy of Large Language Models in Multi-Turn Medical Conversations",
    author = "Kim, Taeil Matthew  and
      Luo, Luyang  and
      Kim, Sung Eun  and
      Manrai, Arjun Kumar  and
      Topol, Eric  and
      Rajpurkar, Pranav",
    editor = {Danilova, Vera  and
      Kurfal{\i}, Murathan  and
      S{\"o}derfeldt, Ylva  and
      Reed, Julia  and
      Burchell, Andrew},
    booktitle = "Proceedings of the 1st Workshop on Linguistic Analysis for Health ({H}ea{L}ing 2026)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.healing-1.2/",
    pages = "19--34",
    ISBN = "979-8-89176-367-8",
    abstract = "Large language models (LLMs) increasingly exhibit sycophancy{---}the tendency to conform to user beliefs rather than provide factually accurate information{---}posing significant risks in healthcare applications where reliability is paramount. We evaluate sycophantic behavior in ten LLMs from OpenAI, Google, and Anthropic across multi-turn medical conversations using an escalatory pushback framework. To enable fine-grained analysis, we introduce Resistance, a metric that measures nonconformity to user stances at each conversational turn, providing insights beyond existing flip-based metrics. Evaluating on MedCaseReasoning (open-ended diagnostic questions) and PubMedQA (clear-answer biomedical questions), we find that Gemini models exhibit the highest Resistance, followed by OpenAI and Claude models. We further observe that response patterns ({''}Yes, but...'' vs. ``Yes, and...'') may be more predictive of sycophancy than specific phrases. Notably, all models are more easily persuaded to change their answers on clear multiple-choice questions than on ambiguous diagnostic cases. Our findings highlight critical vulnerabilities in deploying LLMs for clinical decision support and suggest that training toward contradiction-maintaining response patterns may serve as a potential mitigation strategy."
}