@inproceedings{wang-etal-2025-chain-jailbreak,
    title = "Chain-of-Jailbreak Attack for Image Generation Models via Step by Step Editing",
    author = "Wang, Wenxuan  and
      Gao, Kuiyi  and
      Yuan, Youliang  and
      Huang, Jen-tse  and
      Liu, Qiuzhi  and
      Wang, Shuai  and
      Jiao, Wenxiang  and
      Tu, Zhaopeng",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.findings-acl.571/",
    pages = "10940--10957",
    ISBN = "979-8-89176-256-5",
    abstract = "Text-based image generation models, such as Stable Diffusion and DALL-E 3, hold significant potential in content creation and publishing workflows, making them the focus in recent years. Despite their remarkable capability to generate diverse and vivid images, considerable efforts are being made to prevent the generation of harmful content, such as abusive, violent, or pornographic material. To assess the safety of existing models, we introduce a novel jailbreaking method called Chain-of-Jailbreak (CoJ) attack, which compromises image generation models through a step-by-step editing process. Specifically, for malicious queries that cannot bypass the safeguards with a single prompt, we intentionally decompose the query into multiple sub-queries. The image generation models are then prompted to generate and iteratively edit images based on these sub-queries. To evaluate the effectiveness of our CoJ attack method, we constructed a comprehensive dataset, CoJ-Bench, including nine safety scenarios, three types of editing operations, and three editing elements. Experiments on four widely-used image generation services provided by GPT-4V, GPT-4o, Gemini 1.5 and Gemini 1.5 Pro, demonstrate that our CoJ attack method can successfully bypass the safeguards of models for over 60{\%} cases, which significantly outperforms other jailbreaking methods (i.e., 14{\%}). Further, to enhance these models' safety against our CoJ attack method, we also propose an effective prompting-based method, Think-Twice Prompting, that can successfully defend over 95{\%} of CoJ attack. Our dataset and code are included in the supplementary materials and will be made publicly available upon publication."
}