@inproceedings{wang-etal-2025-towards-adapting,
    title = "Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation",
    author = "Wang, Hanyin  and
      Gao, Chufan  and
      Liu, Bolun  and
      Xu, Qiping  and
      Hussein, Guleid  and
      Labban, Mohamad El  and
      Iheasirim, Kingsley  and
      Korsapati, Hariprasad Reddy  and
      Outcalt, Chuck  and
      Sun, Jimeng",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.626/",
    pages = "12084--12117",
    ISBN = "979-8-89176-256-5",
    abstract = "Proprietary Large Language Models (LLMs) such as GPT-4 and Gemini have demonstrated promising capabilities in clinical text summarization tasks. However, due to patient data privacy concerns and computational costs, many healthcare providers prefer using small, locally-hosted models over external generic LLMs. This study presents a comprehensive domain- and task-specific adaptation process for the open-source LLaMA-2 13 billion parameter model, enabling it to generate high-quality clinical notes from outpatient patient-doctor dialogues. Our process incorporates continued pre-training, supervised fine-tuning, and reinforcement learning from both AI and human feedback. We introduced a new approach, DistillDirect, for performing on-policy reinforcement learning with Gemini 1.0 Pro as the teacher model. Our resulting model, LLaMA-Clinic, can generate clinical notes comparable in quality to those authored by physicians. In a blinded physician reader study, the majority (92.8{\%}) of individual evaluations rated the notes generated by LLaMA-Clinic as ``acceptable'' or higher across all three criteria: real-world readiness, completeness, and accuracy. In the more challenging ``Assessment and Plan'' section, LLaMA-Clinic received the same score as the notes authored by physicians. We highlight key considerations for future clinical note-generation tasks, emphasizing the importance of pre-defining a best-practice note format, rather than relying on LLMs to determine this for clinical practice."
}