@inproceedings{prabhakaran-etal-2025-vade,
    title = "{VADE}: Visual Attention Guided Hallucination Detection and Elimination",
    author = "Prabhakaran, Vishnu  and
      Aggarwal, Purav  and
      Verma, Vinay Kumar  and
      Swamy, Gokul  and
      Saladi, Anoop",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.773/",
    pages = "14949--14965",
    ISBN = "979-8-89176-256-5",
    abstract = "Vision Language Models (VLMs) have achieved significant advancements in complex visual understanding tasks. However, VLMs are prone to hallucinations{---}generating outputs that lack alignment with visual content. This paper addresses hallucination detection in VLMs by leveraging the visual grounding information encoded in transformer attention maps. We identify three primary challenges in this approach: the elective nature of visual grounding for certain tokens, the high-dimensional and noisy nature of attention maps, and the dynamic sequence length of attention on previous tokens. To address these, we propose VADE, a novel sequence modelling approach to effectively learn complex sequential patterns from high-dimensional and noisy attention maps for fine-grained hallucination detection and mitigation. VADE achieves an average PR-AUC of 80{\%} in hallucination detection on M-HalDetect across four different model architectures and an 5{\%} improvement in hallucination mitigation on MSCOCO."
}