@inproceedings{geng-etal-2025-vscbench,
    title = "{VSCB}ench: Bridging the Gap in Vision-Language Model Safety Calibration",
    author = "Geng, Jiahui  and
      Li, Qing  and
      Chen, Zongxiong  and
      Wang, Yuxia  and
      Zhu, Derui  and
      Xie, Zhuohan  and
      Lyu, Chenyang  and
      Chen, Xiuying  and
      Nakov, Preslav  and
      Karray, Fakhri",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.158/",
    pages = "3047--3059",
    ISBN = "979-8-89176-256-5",
    abstract = "The rapid advancement of vision-language models (VLMs) has brought a lot of attention to their safety alignment. However, existing methods have primarily focused on model undersafety, where the model responds to hazardous queries, while neglecting oversafety, where the model refuses to answer safe queries. In this paper, we introduce the concept of safety calibration, which systematically addresses both undersafety and oversafety. Specifically, we present VSCBench, a novel dataset of 3,600 image-text pairs that are visually or textually similar but differ in terms of safety, which is designed to evaluate safety calibration across image-centric and text-centric scenarios. Based on our benchmark, we evaluate safety calibration across eleven widely used VLMs. Our extensive experiments revealed major issues with both undersafety and oversafety. We further investigated four approaches to improve the model{'}s safety calibration. We found that even though some methods effectively calibrated the models' safety problems, these methods also lead to the degradation of models' utility. This trade-off underscores the urgent need for advanced calibration methods, and our benchmark provides a valuable tool for evaluating future approaches."
}