\subsection{Problem Statement}
Human annotation presents challenges in terms of time duration and scalability. %, as it is both time-intensive and difficult to scale across large datasets. 
To address these limitations, this paper investigates the feasibility of using LLMs as automated judges. We benchmark both medically fine-tuned and general-purpose LLMs on their alignment with human annotators when evaluating a patient-facing QA dataset annotated across eight qualitative metrics. Additionally, to enhance the diversity of the evaluation set, we generate negative examples tailored to each metric, allowing for a more comprehensive analysis of LLM judgment and potential biases.

\subsection{Dataset}
To evaluate the alignment of LLMs with human annotators for patient-facing QA, we sought a dataset that not only contain patient-facing QA pairs but are also pre-annotated. We leverage our previous work \citep{pmlr-v259-diekmann25a} that provides 
%Our previous work offers 
two relevant annotated datasets in this context: TREC LiveQA 2017 \citep{LiveMedQA2017} and the CDC subset of MedQuAD \citep{nguyenMedRedQAMedicalConsumer2023}.

For this study, we focus on the MedQuAD dataset.
MedQuAD presents significantly simpler and more concisely phrased questions (average question length of 54.59 characters) compared to TREC LiveQA (average question length of 239.94 characters). This characteristic is particularly advantageous when using LLMs as evaluators, as longer and more complex questions may introduce challenges in judgment responses, potentially consuming a large portion of the context window. By selecting a dataset with shorter and more straightforward questions, we aim to minimize these constraints and improve the reliability of our evaluations of LLM-as-a-Judge.

\citet{pmlr-v259-diekmann25a} used 270 QA pairs in MedQuAD sourced from the CDC website. Each question was answered by four different LLMs: Meditron-7B \citep{chen2023meditron70bscalingmedicalpretraining}, PMC-LLama 13B \citep{wu2023pmcllamabuildingopensourcelanguage}, Me-LLama 13B \citep{physionet_MeLLaMA}, and Meta-Llama-3-8B-Instruct \citep{llama3modelcard}.
This resulted in a total of 1,080 generated model answers. In our previous study \citep{pmlr-v259-diekmann25a}, each of these responses was annotated by a single medical doctor across eight qualitative metrics, adapted from \citep{singhal2023large} and \citep{finch_UnifiedDialogueSystem2020}: Scientific Consensus, Inappropriate and/or Incorrect Content, Missing Content, Extent of Possible Harm, Likelihood of Possible Harm, Possibility of Bias, Empathy, and Grammaticality.
Each metric was assessed using a predefined categorical scale with two to three severity levels. For example, Scientific Consensus was categorized as No Consensus, Opposed to Consensus, or Aligned with Consensus. This structured annotation process allowed for a standardized and granular evaluation of model-generated answers.
The LLM responses and annotations were publicly available on GitHub.\footnote{The annotation dataset was downloaded from \url{https://github.com/yella1603/LLM-Safety-For-PatientQA}.} 

To expand upon the previously generated annotations (i.e., only a single annotator was used for MedQuAD responses) and improve ground truth reliability, we introduced an additional layer of human evaluation. Three additional annotators with clinical or public health training—two holding MD degrees and one holding an MBBS—each reviewed 720 responses. The questions were assigned in a round-robin fashion to ensure that each sample received two additional independent annotations. This resulted in a total of three annotations per response, thereby strengthening the reliability of the ground truth labels.

\subsection{Model Selection}  
Models were selected based on prior work in LLM-as-a-Judge research, ensuring coverage of both general-purpose and medically fine-tuned models. The chosen models include Meta-Llama-3-70B-Instruct \citep{llama3modelcard}, Llama3-OpenBioLLM-70B \citep{OpenBioLLMs}, Prometheus 2 \citep{kim2024prometheus}, Llama3-Med42-8B \citep{med42v2}, and Mixtral-8x7B-Instruct-v0.1 \citep{jiang2024mixtralexperts}. These models were selected to assess how well different types of LLMs align with human annotations, particularly in the context of evaluating patient-facing QA. All models were downloaded from Hugging Face and evaluated locally. Experiments were conducted using an NVIDIA H100 Tensor Core GPU or an NVIDIA Titan RTX GPU to ensure efficient execution and evaluation across all selected models.

\subsection{LLM-as-a-Judge Process}  
Each model was prompted to evaluate every sample in the dataset across all eight qualitative metrics, selecting one of the predefined categorical labels per metric. Since some metrics are inherently related (e.g., Missing Content and Extent of Possible Harm), all judgments for a given sample were generated in a single pass to ensure internal consistency.

Our prompting approach closely followed prior work but required iterative refinement for some models. This process involved manual trial and error to tune prompt phrasing and formatting until the models reliably produced valid categorical outputs. No held-out validation set was used; instead, prompt performance was assessed qualitatively during development. Notably, Prometheus 2 outputs values on a 1–5 Likert scale, which did not directly align with our categorical labels (typically two or three classes). To reconcile this, we implemented a threshold-based mapping strategy that converted Likert responses to the corresponding predefined categories.

For all models, the prompt included the question and the previously generated model answer but excluded the reference answer. This ensured that model judgments relied solely on their own knowledge and reasoning rather than comparison-based scoring. Appendix \ref{sec:appendix_A} contains an example of the final prompt.

\subsection{Adversarial Questions}
While the primary dataset provides a structured evaluation framework, it may not fully capture edge cases where LLM judges default to overly generic or overly lenient outputs. To address this, we introduced an adversarial question set designed to probe model robustness—specifically in the dimension of Scientific Consensus. This metric was chosen because it offers relatively objective correctness criteria, making it more suitable for controlled adversarial testing than more subjective axes like Empathy.

We generated 35 adversarial questions using ChatGPT, with the goal of constructing samples that, under proper evaluation, should be classified in the most severe category for Scientific Consensus. Using a powerful LLM as the adversarial generator (or "attacker") is a common practice in stress-testing safety evaluators \citep{ness2024medfuzzexploringrobustnesslarge, 10487363}. These adversarial questions were deliberately subtle and designed to require non-superficial medical reasoning to identify as misaligned with consensus. 

Our approach involved generating answers that are specifically opposed to scientific consensus while maintaining surface plausibility to challenge automated evaluation systems. The key insight is that effective adversarial examples in medical domains must balance factual incorrectness with believability---answers that are obviously wrong would not effectively test model robustness, while answers that are too subtle might not constitute clear evaluation failures. We focused on creating systematic errors in four critical areas: pathogen identification (e.g., misclassifying parasitic infections as viral), treatment recommendations (e.g., suggesting antibiotics for parasitic conditions), diagnostic methods (e.g., proposing inappropriate testing procedures), and risk factor assessment (e.g., misidentifying vulnerable populations).

The generation prompt was: \textit{``Generate answers to CDC medical questions that are specifically opposed to scientific consensus but sound plausible enough to potentially fool an LLM judge. The answers should contain factually incorrect medical information that requires specialized medical knowledge to identify as wrong, while remaining detectable by human experts. Focus on creating subtle errors in pathogen identification, treatment recommendations, diagnostic methods, and risk factors that an automated system might miss but a qualified reviewer would catch.''}

\subsection{Evaluation}
For the original generated responses, we measure LLM judge alignment with human annotators by treating human annotations as the ground truth and evaluating the model’s accuracy in aligning with human judgments. Each question is annotated by three human annotators (1 from the previous dataset and 2 of our 3 medical annotators). When there is disagreement, we use the median value as the ground truth.
Accuracy is then calculated as the proportion of cases where the LLM judge's evaluation matches this ground truth. %, using sklearn’s accuracy\_score function \citep{scikit-learn}.
For the adversarial questions, we measure the percentage of correctly classified questions from the most severe category. %-- in the most severe category -- classified questions.