@inproceedings{huang-etal-2026-jailmeter,
    title = "{J}ail{M}eter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models",
    author = "Huang, Qingjia  and
      Zhang, Jingyu  and
      Wu, Jianguo  and
      Li, Yakai  and
      Zhang, Weijuan  and
      Rong, Yankai  and
      Yao, Junyi  and
      Zhang, Shengzhi  and
      Jia, Xiaoqi",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.786/",
    pages = "16006--16029",
    ISBN = "979-8-89176-395-1",
    abstract = "The assessment of jailbreak attacks against large language models currently suffers from inconsistent evaluation criteria and methods, leading to unreliable estimates of attack success rates. We propose JailMeter, an evidence-based evaluation framework designed to more faithfully measure jailbreak effectiveness. Inspired by the Information Bottleneck theory, JailMeter applies dual-feedback optimization to filter jailbreak noise from model responses while preserving content relevant to the original malicious question. This process produces concise evidence for a rigorous assessment under which an attack is validated only when the response captures the malicious intent and delivers a complete answer, thereby signaling a substantive bypass of model safety alignment. We evaluate JailMeter on JailMeter-Eva, a challenging benchmark containing 330 human-labeled, non-rejected jailbreak instances. JailMeter achieves an accuracy of 97.27{\%}, substantially outperforming existing evaluation methods. To support large-scale evaluation, we further distill JailMeter into a small language model, JailMeter{SLM}, which maintains comparable reliability with significantly reduced computational costs. Code and dataset are available at \url{https://github.com/Magi2B0y/JailMeter}."
}