@inproceedings{zheng-etal-2026-usb,
    title = "{USB}: A {COMPREHENSIVE} {AND} {UNIFIED} {SAFETY} {EVALUATION} {BENCHMARK} {FOR} {MULTIMODAL} {LARGE} {LANGUAGE} {MODELS}",
    author = "Zheng, Baolin  and
      Chen, Guanlin  and
      Teng, Qingyang  and
      Zhong, Hongqiong  and
      Tan, Yingshui  and
      Liu, Zhendong  and
      Wang, Weixun  and
      Liu, Jiaheng  and
      Yang, Jian  and
      Jing, Huiyun  and
      Wei, Jincheng  and
      Su, Wenbo  and
      Zhu, Xiaoyong  and
      Zheng, Bo  and
      Zhang, Kaifu",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.970/",
    pages = "21184--21211",
    ISBN = "979-8-89176-390-6",
    abstract = "Despite their rapid advancement, Multimodal Large Language Models (MLLMs) remain vulnerable to diverse safety risks. Current benchmarks fail to provide reliable assessments due to limited risk coverage, insufficient scale, and the oversight of complex modality combinations (e.g., cross-modal risks). To address this, we introduce the Unified Safety Benchmark (USB), a comprehensive framework covering 61 risk categories across four distinct modality interactions. We first demonstrate that existing benchmarks{---}even when aggregated{---}leave significant coverage gaps. To bridge this, we design a sophisticated data synthesis pipeline that generates complementary data, ensuring balanced coverage across all risk dimensions. Furthermore, beyond evaluating vulnerability to harmful queries, USB incorporates the simultaneous assessment of model over-refusal on benign inputs as an integrated diagnostic suite. Experimental results, evaluating 22 MLLMs across 244 risk-modality intersections, demonstrate that existing MLLMs still struggle with the trade-off between avoiding vulnerabilities and over-refusal. Models are particularly vulnerable to image-only or cross-modal risky inputs, highlighting the persistent need for refined safety mechanisms. Warning: This paper contains unfiltered and potentially harmful content that may be offensive."
}