The rapid advancement of large language models (LLMs) has led to their increasing use in high-stakes domains, including patient-facing medical question answering (QA). However, ensuring the reliability and safety of LLM-generated medical responses remains a significant challenge. Evaluating these LLM responses traditionally relies on expert human annotation, a process that is time-intensive, costly, and difficult to scale. As a result, there is growing interest in exploring whether LLMs themselves can serve as automated evaluators. %, reducing the need for human labor while maintaining robust assessment quality.  

While LLMs have shown promise as judges in various NLP evaluation tasks \citep{gu2025surveyllmasajudge}, their applicability in medical contexts remains underexplored. The complexity of medical QA -- where responses must be accurate, contextually appropriate, and aligned with clinical consensus -- raises concerns about whether LLMs can effectively replicate expert judgment. Medical evaluation requires nuanced assessments across multiple qualitative dimensions, such as scientific validity, completeness, and potential harm, making it unclear how well LLMs align with human annotators in this setting.  

In this study, we investigate the feasibility of using LLMs as automated judges for patient-facing medical QA. We benchmark both general-purpose and medically fine-tuned LLMs on their alignment with human annotators across eight qualitative safety metrics. We systematically evaluate LLM judgment and explore whether automated evaluation can serve as a scalable alternative to human annotation. Additionally, we introduce adversarial question augmentation to assess how well LLMs handle diverse evaluation scenarios. Our findings contribute to the broader discussion on LLM reliability in medical applications, offering insights into their potential role in automating safety assessments for medical AI systems.