\begin{figure*}[t] %%%
\begin{center}
\includegraphics[width=0.98\textwidth]{fig/overview.pdf}
\end{center}
\vspace{-0.25cm}
\caption{\textbf{Overview of our method, QGuard}. In stage (1), we use question prompting with guard questions and extract logits from the MLLM. In stage (2), we classify the extracted logits into harmful and unharmful categories using the filtering algorithm.}
\label{figure:overview}
\vspace{-0.5cm}
\end{figure*} %%% Figure

\section{Related Work} \label{sec:related_work}
\subsection{Harmful Prompt Detection}
With the rapid advancement of LLMs, malicious attacks have also been increasing significantly. As a result, extensive research~\cite{caselli2020hatebert, hada2021ruddit, vidgen2020learning, lin2023toxicchat, inan2312llama, mazeika2024harmbench, huang2024harmful} has been conducted to detect harmful, offensive, hate speech, and toxic language. In particular, many studies~\cite{lin2023toxicchat, rottger2023xstest, rottger-etal-2021-hatecheck} have focused on detecting hate speech on social media platforms. For instance, ToxicChat~\cite{lin2023toxicchat} has been proposed as a new benchmark that focuses on detecting unsafe prompts in LLMs using real user queries, rather than content derived from social media. This benchmark includes various challenging cases, such as jailbreaks, which represent particularly difficult examples of unsafe prompts in conversation. 
Additionally, recent works~\cite{inan2312llama,han2024wildguard,xie2024gradsafe,gu2025mllmguard,YouOnlyPromptOnce} have aimed to defend against harmful prompts by constructing dedicated datasets and fine-tuning LLMs. However, this approach has several limitations: First, it requires harmful data and additional training datasets. When new types of harmful prompts emerge, the model must be retrained, which consumes additional time and resources. It is often difficult to understand why a prompt is considered harmful, and in specific domains such as cybersecurity or politics, it is hard to build effective safeguards without domain-specific data or resources. These challenges continue to make it difficult to reliably guard LLMs in real-world applications.

\subsection{Multimodal Harmful Prompt Detection}
As LLMs advance to handle not only text but also various types of data such as images, videos, and audio~\cite{achiam2023gpt, team2023gemini, singer2022make,crossmodal,SafetyofMultimodal}, the importance of multi-modal harmful prompt detection methods is also growing~\cite{ye2025survey, liu2024survey}. Recently, multi-modal harmful datasets~\cite{gu2025mllmguard, liu2024mminstruct} based on social media platforms similar to traditional harmful prompt datasets have been proposed. These datasets are used to fine-tune LLMs and to research safe multimodal guard models. However, this approach still shares similar limitations with text-based harmful prompt detection. First, it requires fine-tuning the LLMs, which can be time-consuming and resource-intensive. Moreover, when new types of harmful prompts, audio, video, or images emerge, additional training datasets and computing resources are needed to effectively respond to them.
