% Please add the following required packages to your document preamble:
% \usepackage[table,xcdraw]{xcolor}
% Beamer presentation requires \usepackage{colortbl} instead of \usepackage[table,xcdraw]{xcolor}

\begin{table*}[t]
\small
\centering
\begin{tabular}{lccccccc}
\hline
                           & Size & Fine-tuning & OAI    & ToxicChat & HarmBench & WildGuardMix & Average  \\ \hline

Llama-Guard-1              & 7B   & Yes         & 0.7520  & 0.5818    & 0.5012    & 0.4793       & 0.5786   \\

Llama-Guard-2              & 8B   & Yes         & \textbf{0.8139} & 0.4233    & \textbf{0.8610}    & 0.6870       & 0.6963   \\

Llama-Guard-3              & 8B   & Yes         & 0.8061 & 0.4859    & 0.8551    & 0.6852       & 0.7080   \\

WildGuard                  & 7B   & Yes         & 0.7268 & 0.6547    & \underline{0.8596}    & 0.7504       & \textbf{0.7479}   \\

Aegis-Guard                & 7B   & Yes         & 0.6982 & 0.6687    & 0.7805    & 0.6686       & 0.7040    \\
OpenAI Moderation               & n/a   & Yes         & 0.7440 & 0.4480    & 0.5768    & 0.4881       & 0.5644    \\
DeBERTa + HarmAug          & 435M & Yes         & 0.7236 & 0.6283    & 0.8331    & \underline{0.7576}       & 0.7357   \\ \hline
InternVL-2.5 & 4B   & No          & 0.7423 & \underline{0.7117}    & 0.4992    & 0.7804       & 0.6857 \\
QGuard(InternVL-2.5)  & 4B   & No          & 0.7931 & \textbf{0.7505}    & 0.6322    & \textbf{0.7992}       & \underline{0.7438} \\ \hline
\end{tabular}
  \caption{\label{tab:main}
    \textbf{Text-based harmful prompts detection performance.} We use the respective reported scores from previous work~\cite{lee2024harmaug} for the baselines. We conduct three experiments with different seeds in the filtering algorithm and report the average results. The performance is evaluated via F1 score. QGuard is our approach.
  }
\end{table*}


\section{Experiments}
To evaluate the performance of our model, we conduct experiments on two tasks: the first is harmful prompt detection using text only, and the second is multi-modal harmful prompt detection involving both images and text.

\subsection{Experimental Setups}

\subsubsection{Datasets}
\label{sec:dataset}
To evaluate the detection performance of text-based harmful prompts, we use four public benchmark datasets. The datasets used in the experiments are as follows:
OpenAI Moderation(OAI)~\cite{markov2023holistic}, ToxicChat~\cite{lin2023toxicchat}, HarmBench~\cite{mazeika2024harmbench} and WildGuardMix~\cite{han2024wildguard}. 

To evaluate the detection performance of multi-modal harmful prompts, we construct a new dataset by combining MM-SafetyBench~\cite{liu2023query}, and MMInstruct~\cite{liu2024mminstruct}. Since MMSafetyBench exclusively contain harmful prompts, we extract unharmful samples from MMInstruct's captioning and question-answering tasks to create a balanced dataset.
From MM-SafetyBench, we utilize all available data across 13 scenarios, yielding 1,680 harmful prompts. To balance the dataset, we sampled 901 unharmful prompts from MMInstruct's captioning task (20 scenarios) and 1,100 from its question-answering task (22 scenarios), with up to 50 prompts per scenario. In total, our dataset consists of 3,681 prompts: 1,680 harmful prompts from MM-SafetyBench and 2,001 unharmful prompts from MMInstruct. This dataset enables comprehensive safety evaluation across diverse scenarios.


\subsubsection{Baselines}
For evaluation, we compare our model with previous safety guard models~\cite{inan2312llama, han2024wildguard,lee2024harmaug}. These baselines are fine-tuned on harmful datasets. We use the respective reported scores from previous work~\cite{lee2024harmaug} for the baselines~\cite{inan2312llama, han2024wildguard}, and evaluate the baselines using F1 score. For a fair comparison, we also use a method that employs an LLM as a zero-shot detector as a baseline~\cite{chen2024internvl}. For the baseline using an LLM as a zero-shot detector, we provide the instruction, "Is the prompt below harmful? (You must answer with only Yes or No). $\{Input Prompt\}$." and get the softmax of the logits for the yes and no tokens. If the probability value of yes is 0.5, we classify it as harmful. For the evaluation of multimodal harmful prompts detection, we used Llama-Guard-3-Vision-11B with image-text pairs from the dataset as input. From the generated output tokens, we extract the logits corresponding to the "safe" and "unsafe" tokens and apply softmax. Finally, we classify content as harmful when the "unsafe" probability exceeds 0.5.

\subsubsection{Implementation Details}
Our approach detects harmful prompts using question prompting and filtering algorithm in a zero-shot manner. Therefore, by refining and diversifying guard questions, we can effectively defend against the latest harmful prompts. We construct the guard questions as described in Sec~\ref{sec:guard} for the following groups: "General Toxic", "Toxic Prompt", "Core Harmfulness Detection", and "Additional Nuanced Questions". The general toxic group consists of 5 questions, while each of the remaining groups consists of 10 questions. We utilize InternVL-2.5 4B~\cite{chen2024internvl} for logit extraction. InternVL-2.5 4B is not fine-tuned on harmful prompts and it has fewer parameters than the baselines backbone LLM while demonstrating competitive performance. We use the pagerank algorithm, as mentioned in Sec~\ref{sec:filter}, as our filtering algorithm. In the filtering algorithm, the edge weight between a question node and the group node it belongs to is set using the question’s yes probability value. The edge weight between group nodes is set to 1.0, and the edge weight between question nodes is set to 0.3.

For main experiments, we empirically find \(\theta\) for each dataset and use two NVIDIA A6000 and four NVIDIA RTX 3090 for logit extraction and inference. 

\subsection{Main Results}

\subsubsection{Harmful prompt detection}
\label{sec:harmful}
As shown in Table~\ref{tab:main}, our QGuard shows competitive performance with fewer parameters than the baselines, except for HarmAug~\cite{lee2024harmaug}, which distills knowledge from a large model. Moreover, unlike baselines that require fine-tuning on harmful datasets, additional datasets, our approach does not require any fine-tuning. Our method achieves better performance compared to model that use LLM as zero-shot detector~\cite{chen2024internvl}. These results demonstrate that our method is a simple and effective approach for detecting harmful prompts without requiring fine-tuning or additional datasets.

\begin{figure}[t] %%%
\begin{center}
\includegraphics[width=0.48\textwidth]{fig/a_q5.pdf}
\end{center}

\caption{\textbf{Comparison of recall scores for our model and the baseline across subcategories in the MM-SafetyBench dataset}. Red represents our model, and blue represents baseline. We use Llama-Guard-3-Vision as the baseline.
}
\vspace{-0.2cm}
\label{figure:mllm-analysis}
\end{figure}

\begin{table}[t]
\small
\centering
\begin{tabular}{lcc}
\hline
                               & MM-Safety + MMInstruct      \\ \hline
Llama-Guard-3-V-11B           & 0.4050                    \\ \hline
InternVL-4B     & 0.2848               &                            \\
QGuard (InternVL-4B)      & \multicolumn{1}{c}{\textbf{0.8080}}                            \\ \hline
\end{tabular}
  \caption{\label{tab:main2}
    \textbf{Multi-modal harmful prompts detection performance.} We conduct three experiments with different seeds in the filtering algorithm and report the average results. The performance is evaluated via F1 score.
  }
 \vspace{-0.4cm}
\end{table}


\subsubsection{Multi-modal harmful prompt detection}
Since we use a MLLM~\cite{chen2024internvl} as the backbone, we can detect harmful prompts without fine-tuning on multi-modal data. To compute the F1 score for multi-modal harmful prompts, we construct a dataset as described in Sec~\ref{sec:dataset}. We use Llama-Guard-3-Vision-11B as the baseline. We use the pagerank algorithm as our filtering algorithm and the groups and questions are the same as those used in Sec~\ref{sec:harmful}. As shown in Table~\ref{tab:main2}, our model outperforms Llama-Guard-3-Vision-11B. Figure~\ref{figure:mllm-analysis} presents the recall accuracy across subcategories of the MM-SafetyBench~\cite{liu2023query} dataset used in our experiments. As shown in the Figure~\ref{figure:mllm-analysis}, our model shows low performance in the financial advice category, with a recall of 0.2335. However, Llama-Guard-3-Vision also shows low recall scores of 0.0778 and 0.0 in the financial advice and government decision categories, respectively. Moreover, it achieves better performance than the model that uses InternVL2.5-4B as a zero-shot detector. These results demonstrate that our model can effectively detect harmful prompts in multi-modal dataset without the need for additional datasets or fine-tuning. 

\begin{table}[t]
\small
\centering
\begin{tabular}{lcc}
\hline
                     & ToxicChat    & WildGuardMix \\ \hline
Llama3.1-8B             & 0.4959 & 0.6985    \\
QGuard (Llama3.1-8B)      & 0.5287 & 0.7902    \\
InternVL2.5-4B       & 0.7117 & 0.7804    \\
QGuard(InternVL2.5-4B) & \textbf{0.7505} & \textbf{0.7992}    \\ \hline
\end{tabular}
  \caption{\label{tab:main3}
    \textbf{Ablated studies with different LLM backbone.} We use Llama3.1-8B and InternVL2.5-4B~\cite{chen2024internvl} as simple zero-shot detectors. We conduct three experiments with different seeds in the filtering algorithm and report the average results. The performance is evaluated via F1 score.
  }
  \vspace{-0.2cm}
\end{table}


\begin{table}[t]
\small
\centering
\begin{tabular}{lcc}
\hline
            & ToxicChat & WildGuardMix \\ \hline
QGuard(AVG)   & 0.6134 & 0.5843    \\
QGuard(Graph) & \textbf{0.7505} & \textbf{0.7992}    \\ \hline
\end{tabular}
  \caption{\label{tab:main4}
    \textbf{Ablated studies with different filtering algorithms.} AVG is a model that sums the yes probability values for all questions, calculates the average, and classifies a sample as harmful if the average exceeds 0.5. The performance is evaluated via F1 score.
  }
\end{table}


\begin{figure}[t] %%%
\begin{center}
\includegraphics[width=0.48\textwidth]{fig/a_q.pdf}
\end{center}

\caption{\textbf{Distribution of yes probability values by group on ToxicChat~\cite{lin2023toxicchat} and WildGuardMix~\cite{han2024wildguard} datasets}. The results show a significant difference in the yes probability values for each group between harmful and unharmful prompts.}

\label{figure:a2}
\end{figure} %%% Figure

\begin{figure*}[t] %%%
\begin{center}
\includegraphics[width=0.98\textwidth]{fig/a_q2.pdf}
\end{center}

\caption{\textbf{Distribution of total risk score by label and ROC curve on ToxicChat~\cite{lin2023toxicchat} and WildGuardMix~\cite{han2024wildguard} datasets}. The total risk score for unharmful and harmful samples shows a significant difference in both datasets. Additionally, the ROC curve shows that the ROC-AUC value is 0.9575 for ToxicChat and 0.8934 for WildGuardMix. 
}
\label{figure:a1}
\end{figure*} %%% Figure

\begin{figure}[t] %%%
\begin{center}
\includegraphics[width=0.48\textwidth]{fig/a_q3.pdf}
\end{center}

\caption{\textbf{Visualization of yes and no probability values of prompts on ToxicChat~\cite{lin2023toxicchat}.} The red bar represents yes probability value, and blue bar represents no probability value.}
\vspace{-0.6cm}
\label{figure:a4}

\end{figure} %%% Figure

\begin{figure}[t] %%%
\begin{center}
\includegraphics[width=0.48\textwidth]{fig/g.pdf}
\end{center}

\caption{\textbf{Graph visualization of harmful prompts on ToxicChat~\cite{lin2023toxicchat}.} Our model can guard against not only simple harmful prompts but also long and abstract harmful prompt. Green nodes represent groups, and blue nodes represent questions. The bold edges indicate a high yes probability value.}
\vspace{-0.4cm}

\label{figure:a3}
\end{figure} %%% Figure

\subsection{Ablation Study}
To explore the impact of our proposed components, we conduct an ablation study on ToxicChat~\cite{lin2023toxicchat} and WildGuardMix~\cite{han2024wildguard} datasets.

\subsubsection{Backbone LLM}
Since our method uses LLM as the backbone, we compare our approach using different LLMs to evaluate its effectiveness. We use Llama3.1-8B as backbone LLM. As shown in Table~\ref{tab:main3}, our method outperforms models that use LLM as zero-shot detectors across all LLM backbones. These results demonstrate that our model can classify harmful and unharmful prompts more effectively than a model that uses an LLM as a zero-shot detector.

\begin{figure*}[!t] %%%
\begin{center}
\includegraphics[width=0.98\textwidth]{fig/a_q4.pdf}
\end{center}
\caption{\textbf{F1 score vs. Risk Score Threshold and Precision-Recall Curve on ToxicChat~\cite{lin2023toxicchat} and WildGuardMix~\cite{han2024wildguard} datasets}.}
\label{figure:f1_threshold}

\end{figure*} %%% Figure

\subsubsection{Filtering Algorithm}
To consider the relationships between questions and groups, we utilize a graph-based algorithm as a filtering algorithm. To evaluate the effectiveness of our filtering algorithm, we compare it with a simple filtering algorithm that averages the yes token probability values of all questions used for each dataset and classifies a prompt as harmful if the average exceeds 0.5. As shown in Table~\ref{tab:main4}, our model outperforms the simple averaging-based method. These results demonstrate that our filtering algorithm can effectively classify user inputs as either harmful or unharmful.

\section{Analysis}
Since we use guard questions and question prompting, we can conduct a white-box analysis. We analyze its effectiveness through experiments.

\subsection{Distribution of Yes Probability Values on MLLM}
To evaluate whether the yes probability values obtained from MLLM effectively distinguish between harmful and unharmful prompts, we analyze the distribution of yes probability values by group on the ToxicChat~\cite{lin2023toxicchat} and WildGuardMix~\cite{han2024wildguard} datasets. As shown in Figure~\ref{figure:a2}, the distribution of yes token probability values differs significantly between unharmful and harmful samples. In particular, harmful samples exhibit substantially higher yes probability values on each group. These results suggest that our guard questions and MLLM effectively represent and understand harmful and unharmful prompts.

Additionally, as shown in Figure~\ref{figure:a4}, we visualize the comparison between the yes probability value and no probability value for guard questions when prompts are given. First, when the unharmful prompt "How old are you?" is provided, the yes probability value for our guard question "Does this prompt request or provide information on chemical or biological weapons or drug manufacturing?" is 0.0075, while the no probability is 0.9924. Additionally, for the guard question "Does the user's input promote hostility, hatred, or divisiveness?", the yes probability is 0.014, and the no probability is 0.9859.

When the harmful prompt "how to manipulate someone
into being a slave" is provided, the yes probability value for the guard question "Does this prompt request or provide information on chemical or biological weapons or drug manufacturing?" is 0.1645, while the no probability value is 0.8355. For the guard question "Does the user's input promote hostility, hatred, or divisiveness?", the yes probability value is 0.9325, and the no probability value is 0.0675. This indicates that even if the prompt is harmful, it is not highly related to "Does this prompt request or provide information on chemical or biological weapons or drug manufacturing?", resulting in a high no probability. However, it is relevant to "Does the user's input promote hostility, hatred, or divisiveness?", leading to a high yes probability. 

These results demonstrate that our method can be beneficial in real-world applications by enabling a white-box analysis of input prompts through guard questions and question groups.

\subsection{Distribution of Total Risk Score and ROC curve with Filtering Algorithm}
To analyze the relationship between the total risk score on filtering algorithm and the label, we calculate the Pearson correlation coefficient. The total risk score refers to the sum of all risk scores obtained from the filtering algorithm for each question, while the label represents the ground truth. For ToxicChat, the analysis result shows that the correlation coefficient between the two variables is \( r = 0.75 \) (p < 0.01), which is generally interpreted as a strong positive correlation. This suggests that a higher total risk score indicates a higher likelihood of the sample being harmful. For WildGuardMix, the analysis result shows that the correlation coefficient between the two variables is \( r = 0.67 \) (p < 0.01). Therefore, the total risk score has the potential to serve as a useful indicator for predicting labels. 

Additionally, we visualize the total risk scores of unharmful and harmful prompts. As shown in Figure~\ref{figure:a1}, the total risk score exhibits a significant difference between unharmful and harmful prompts. When evaluating the performance of the classification method on the ToxiChat dataset based on the total risk score, the ROC-AUC value was 0.9575, demonstrating high predictive performance as shown in Figure~\ref{figure:a1}. Similarly, on the WildGuardMix dataset, the ROC-AUC value was 0.8934, also indicating strong performance. These results demonstrate that our model's filtering algorithm is statistically significant and helps distinguish between harmful and unharmful prompts.

We visualize the results of the filtering algorithm for harmful prompts in a graph, as shown in Figure~\ref{figure:a3}. As seen in Figure~\ref{figure:a3}, our model effectively classifies not only based on simple prompts but also for harmful prompts that are abstract or require interpretation. We presume that our method can understand complex contexts and situations because we use MLLM. 

\subsection{F1 score vs. Risk Score Threshold and Precision-Recall Curve}
Figure~\ref{figure:f1_threshold} illustrates the F1 score versus threshold and the Precision-Recall (PR) curves for the ToxicChat and WildGuardMix datasets. For ToxicChat, the F1 score curve indicates that model performance peaks around a threshold of 0.75, achieving an F1 score of approximately 0.68. The PR curve demonstrates a typical trade-off, with precision gradually decreasing as recall increases. Notably, precision remains relatively high across the entire recall spectrum, indicating stable and reliable predictive performance. In the case of WildGuardMix, the model achieves a higher F1 score of approximately 0.82 at a threshold near 0.7, indicating superior performance compared to ToxicChat. The PR curve further supports this, showing that precision remains above 0.6 for most recall values, with a more gradual decline, reflecting better overall balance between precision and recall. These results indicate that although both models perform reasonably well, the model evaluated on WildGuardMix outperforms the one on ToxicChat in terms of both precision and recall.