@inproceedings{pakull-etal-2025-preliminary,
    title = "Preliminary Evaluation of an Open-Source {LLM} for Lay Translation of {G}erman Clinical Documents",
    author = {Pakull, Tabea  and
      Dada, Amin  and
      Damm, Hendrik  and
      Fleischhauer, Anke  and
      Benson, Sven  and
      Bender, No{\"e}lle  and
      Prasuhn, Nicola  and
      Kaminski, Katharina  and
      Friedrich, Christoph  and
      Horn, Peter  and
      Kleesiek, Jens  and
      Schadendorf, Dirk  and
      Pretzell, Ina},
    editor = "Ananiadou, Sophia  and
      Demner-Fushman, Dina  and
      Gupta, Deepak  and
      Thompson, Paul",
    booktitle = "Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health)",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.cl4health-1.15/",
    pages = "180--192",
    ISBN = "979-8-89176-238-1",
    abstract = "Clinical documents are essential to patient care, but their complexity often makes them inaccessible to patients. Large Language Models (LLMs) are a promising solution to support the creation of lay translations of these documents, addressing the infeasibility of manually creating these translations in busy clinical settings. However, the integration of LLMs into medical practice in Germany is challenging due to data scarcity and privacy regulations. This work evaluates an open-source LLM for lay translation in this data-scarce environment using datasets of German synthetic clinical documents and real tumor board protocols. The evaluation framework used combines readability, semantic, and lexical measures with the G-Eval framework. Preliminary results show that zero-shot prompts significantly improve readability (e.g., FREde: 21.4 {\textrightarrow} 39.3) and few-shot prompts improve semantic and lexical fidelity. However, the results also reveal G-Eval{'}s limitations in distinguishing between intentional omissions and factual inaccuracies. These findings underscore the need for manual review in clinical applications to ensure both accessibility and accuracy in lay translations. Furthermore, the effectiveness of prompting highlights the need for future work to develop applications that use predefined prompts in the background to reduce clinician workload."
}