@inproceedings{hao-etal-2025-making,
    title = "Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical {KV} Equalization",
    author = "Hao, Shuyang  and
      Wang, Yiwei  and
      Hooi, Bryan  and
      Liu, Jun  and
      Chen, Muhao  and
      Huang, Zi  and
      Cai, Yujun",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-luhme/2025.findings-emnlp.618/",
    doi = "10.18653/v1/2025.findings-emnlp.618",
    pages = "11528--11543",
    ISBN = "979-8-89176-335-7",
    abstract = "In the realm of large vision-language models (LVLMs), adversarial jailbreak attacks serve as a red-teaming approach to identify safety vulnerabilities of these models and their associated defense mechanisms. However, we identify a critical limitation: not every adversarial optimization step leads to a positive outcome, and indiscriminately accepting optimization results at each step may reduce the overall attack success rate. To address this challenge, we introduce HKVE (Hierarchical Key-Value Equalization), an innovative jailbreaking framework that selectively accepts gradient optimization results based on the distribution of attention scores across different layers, ensuring that every optimization step positively contributes to the attack. Extensive experiments demonstrate HKVE{'}s significant effectiveness, achieving attack success rates of 75.08{\%} on MiniGPT4, 85.84{\%} on LLaVA and 81.00{\%} on Qwen-VL, substantially outperforming existing methods by margins of 20.43{\%}, 21.01{\%} and 26.43{\%} respectively. Furthermore, making every step effective not only leads to an increase in attack success rate but also allows for a reduction in the number of iterations, thereby lowering computational costs."
}