% \subsection{LLM-as-a-Judge Approaches}
\section{Additional Related Work Details}
\label{sec:long_related_work}

\begin{table*}[t]
    \centering
    \input{latex/example_prompt}
    \label{example_prompt}
\end{table*}

\begin{table*}[t]
    \centering
    \input{latex/annotator_agreement}
    \label{tab:compact-agreement}
\end{table*}

\citet{gu2025surveyllmasajudge} conducted a comprehensive survey on LLM-as-a-Judge methodologies, highlighting optimization strategies such as in-context learning, scoring scale selection, and model choice. They decomposed evaluation steps using chain-of-thought frameworks and identified various biases including self-enhancement, length, and position bias. Their benchmarking, using pairwise comparison, revealed that while GPT-4 was the best closed-source model and Qwen2.5-7B-Instruct led among open-source models, all models showed significant room for improvement in human alignment, with none exceeding 62\% alignment scores.

In a large-scale empirical study across 20 NLP evaluation tasks, \citet{bavaresco2024llmsinsteadhumanjudges} found that LLM-as-a-Judge performance varies substantially across models. Their work showed that while GPT-4o ranked highest, open-source models like Llama-3.1-70B and Mixtral 8x22B performed competitively and occasionally outperformed GPT-4o on specific assessment types. Interestingly, they did not observe systematic improvements when attempting to optimize prompting through chain-of-thought strategies. Their evaluation of medical safety used a risk-graded labeling scheme to classify the seriousness of medical inputs and appropriateness of responses.

\citet{szymanski2024limitationsllmasajudgeapproachevaluating} investigated limitations of the LLM-as-a-Judge approach in medical fields. Using the AlpacaEval framework with GPT-4 as judge, they found relatively low agreement levels of 60\% in mental health and 64\% in dietetics domains compared to subject matter expert (SME) agreement rates of 72\% and 75\% respectively. Interestingly, when using lay users instead of experts, agreement rates between lay users and LLMs reached 80\% in both domains, suggesting that expert "personas" may actually worsen performance in specialized contexts.

In the context of evaluating medical safety, \citet{han2024medsafetybenchevaluatingimprovingmedical} introduced MedSafetyBench, which uniquely focused on the safety of LLMs in medical domains. Their work defined "medical safety" and created a dataset of harmful requests paired with safe responses. They employed GPT-3.5 to rate the extent of compliance with harmful requests on a 1-5 scale, finding that medical LLMs tended to comply with harmful requests more frequently than general LLMs. 

\begin{table*}[ht]
    \centering
    \input{latex/distribution}
\end{table*}

\citet{kanithi2024mediccomprehensiveframeworkevaluating} proposed MEDIC, a framework designed to evaluate LLMs in clinical applications. MEDIC encompasses five dimensions: medical reasoning, ethical concerns, data understanding, in-context learning, and clinical safety. Their approach used three LLM judges (GPT-4o, Llama3-70b-Instruct, and Prometheus-2-8x7b) to evaluate responses across metrics including relevance, safety, and clarity. They found high alignment between judges (up to 78.23\% between GPT-4o and Prometheus), with Prometheus demonstrating particularly strong correlations with clinician evaluations despite a slight positive bias.

Similarly, \citet{krolik2024leveraginglargelanguagemodels} evaluated whether LLMs can be leveraged for automated medical Q\&A evaluation. Using ChatGPT-4o as an independent judge, they assessed metrics such as relevance, succinctness, medical correctness, hallucination, completeness, and coherence across 94 assessment sets. Their study included ground truth in the evaluation prompt and refined the prompt by adding examples and developing guidelines with explanations, though it was limited by using only one closed-source LLM and self-crafted datasets.



\citet{zheng2023judgingllmasajudgemtbenchchatbot} directly evaluated the LLM-as-a-Judge approach by comparing to human evaluations using MT-Bench (80 multi-turn questions) and Chatbot Arena (a crowdsourced platform). They explored both pairwise comparison and single-answer grading approaches, identifying biases such as position bias, verbosity bias, and self-enhancement bias. Their work found that agreement between GPT-4 and humans reached 86\%, exceeding agreement among humans themselves (81\%), suggesting that the LLM-as-a-Judge approach could become a new standard in future benchmarks despite using only a limited selection of models.

% \subsection{Patient-Facing Medical QA Safety}

% This study builds on our previous work \citep{pmlr-v259-diekmann25a}, which evaluated the safety of LLM-generated responses in patient-facing medical QA. While that work relied on human annotation, we now explore whether LLMs can serve as automated judges for such evaluations.

