@inproceedings{agrawal-etal-2026-towards,
    title = "Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings",
    author = "Agrawal, Aakriti  and
      KV, Gouthaman  and
      Aralikatti, Rohith  and
      Jagatap, Gauri  and
      Yuan, Jiaxin  and
      Baskar, Sarvesh  and
      Kamarshi, Vijay  and
      Fanelli, Andrea  and
      Huang, Furong",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.2086/",
    pages = "42032--42048",
    ISBN = "979-8-89176-395-1",
    abstract = "Hallucinations in Large Vision-Language Models (LVLMs) remain a persistent challenge, often stemming from inadequate integration of visual information during multimodal reasoning. A key cause is the model{'}s over-reliance on textual priors and underutilization of visual cues, leading to outputs that are linguistically fluent but visually inaccurate. For example, given an image of an empty kitchen countertop, an LVLM might hallucinate a ``bowl of fruit'' or ``cup of coffee,'' relying on language associations rather than visual evidence. Most LVLMs incorporate visual features by appending them to the input stream of a pre-trained LLM and training on large-scale vision-language datasets. Our systematic analysis reveals that this strategy often leads to over-dependence on textual information due to the inherent bias of LLMs towards language-dominant representations. This imbalance skews attention towards the text over visual content, weakening the model{'}s ability to ground outputs in visual inputs. To address this, we propose a simple yet effective visual feature incorporation method that encourages the model to learn visually-informed textual embeddings distinct from those of the base LLM and promotes a more balanced attention distribution. Experimental results across multiple hallucination benchmarks demonstrate that our method significantly reduces hallucinations and fosters more balanced multimodal reasoning. Notably, our approach achieves substantial gains, including $\textbf{+9.33}${\%} on MMVP-MLLM, $\textbf{+2.99}${\%} on POPE-AOKVQA, up to $\textbf{+3.4}${\%} on Merlin, and $\textbf{+3}${\%} on the hard-data split of HallusionBench."
}