@inproceedings{zhang-etal-2026-defeating,
    title = "Defeating Cerberus: Privacy-Leakage Mitigation in Vision Language Models",
    author = "Zhang, Boyang  and
      Akkus, Istemi Ekin  and
      Chen, Ruichuan  and
      Dethise, Alice  and
      Satzke, Klaus  and
      Rimac, Ivica  and
      Zhang, Yang",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.findings-eacl.154/",
    pages = "2952--2965",
    ISBN = "979-8-89176-386-9",
    abstract = "Vision Language Models (VLMs) have demonstrated remarkable capabilities in processing multimodal data, but their advanced abilities also raise significant privacy concerns, particularly regarding Personally Identifiable Information (PII) leakage. While relevant research has been conducted on single-modal language models to some extent, the vulnerabilities in the multimodal setting have yet to be fully investigated. Our work assesses these emerging risks and introduces a concept-guided mitigation approach. By identifying and modifying the model{'}s internal states associated with PII-related content, our method guides VLMs to refuse PII-sensitive tasks effectively and efficiently, without requiring re-training or fine-tuning. We also address the current lack of multimodal PII datasets by constructing various ones that simulate real-world scenarios. Experimental results demonstrate the method can achieve on average 93.3{\%} refusal rate for various PII-related tasks with minimal impact on unrelated model performances. We further examine the mitigation{'}s performance under various conditions to show the adaptability of our proposed method."
}