@inproceedings{woo-etal-2025-black,
    title = "Black-Box Visual Prompt Engineering for Mitigating Object Hallucination in Large Vision Language Models",
    author = "Woo, Sangmin  and
      Zhou, Kang  and
      Zhou, Yun  and
      Wang, Shuai  and
      Guan, Sheng  and
      Ding, Haibo  and
      Cheong, Lin Lee",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.naacl-short.45/",
    pages = "529--538",
    ISBN = "979-8-89176-190-2",
    abstract = "Large Vision Language Models (LVLMs) often suffer from object hallucination, which undermines their reliability. Surprisingly, we find that simple object-based visual prompting{---}overlaying visual cues (e.g., bounding box, circle) on images{---}can significantly mitigate such hallucination; however, different visual prompts (VPs) vary in effectiveness. To address this, we propose Black-Box Visual Prompt Engineering (BBVPE), a framework to identify optimal VPs that enhance LVLM responses without needing access to model internals. Our approach employs a pool of candidate VPs and trains a router model to dynamically select the most effective VP for a given input image. This black-box approach is model-agnostic, making it applicable to both open-source and proprietary LVLMs. Evaluations on benchmarks such as POPE and CHAIR demonstrate that BBVPE effectively reduces object hallucination."
}