@inproceedings{diekmann-etal-2025-llms,
    title = "{LLM}s as Medical Safety Judges: Evaluating Alignment with Human Annotation in Patient-Facing {QA}",
    author = "Diekmann, Yella  and
      Fensore, Chase  and
      Carrillo-Larco, Rodrigo  and
      Castejon Rosales, Eduard  and
      Shiromani, Sakshi  and
      Pai, Rima  and
      Shah, Megha  and
      Ho, Joyce",
    editor = "Demner-Fushman, Dina  and
      Ananiadou, Sophia  and
      Miwa, Makoto  and
      Tsujii, Junichi",
    booktitle = "ACL 2025",
    month = aug,
    year = "2025",
    address = "Viena, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.bionlp-1.19/",
    pages = "217--224",
    ISBN = "979-8-89176-275-6",
    abstract = "The increasing deployment of LLMs in patient-facing medical QA raises concerns about the reliability and safety of their responses. Traditional evaluation methods rely on expert human annotation, which is costly, time-consuming, and difficult to scale. This study explores the feasibility of using LLMs as automated judges for medical QA evaluation. We benchmark LLMs against human annotators across eight qualitative safety metrics and introduce adversarial question augmentation to assess LLMs' robustness in evaluating medical responses. Our findings reveal that while LLMs achieve high accuracy in objective metrics such as scientific consensus and grammaticality, they struggle with more subjective categories like empathy and extent of harm. This work contributes to the ongoing discussion on automating safety assessments in medical AI and informs the development of more reliable evaluation methodologies."
}