@inproceedings{su-etal-2026-truth,
    title = "Truth or Dare: Analyzing {LLM} Susceptibility to External Evidence of Varying Factuality",
    author = "Su, Han-Yu  and
      Chu, Kuan-Yu  and
      Li, Yung-Hui  and
      Ku, Lun-Wei",
    editor = "Chang, Kai-Wei  and
      Mehrabi, Ninareh  and
      Krishna, Satyapriya  and
      Das, Anubrata  and
      Dhamala, Jwala  and
      Cao, Yang Trista  and
      Kumarage, Tharindu  and
      Ramakrishna, Anil  and
      Christodoulopoulos, Christos  and
      Wan, Yixin  and
      Galystan, Aram  and
      Kumar, Anoop  and
      Gupta, Rahul",
    booktitle = "Proceedings of the 6th Workshop on Trustworthy {NLP} ({T}rust{NLP} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.trustnlp-main.39/",
    pages = "528--538",
    ISBN = "979-8-89176-418-7",
    abstract = "Modern Large Language Models (LLMs) often rely on Retrieval-Augmented Generation (RAG) to access up-to-date information; however, retrieved corpora may contain misleading, outdated, or incorrect content, raising concerns about how such evidence affects model reliability. In this work, we investigate the susceptibility of LLMs to false external evidence. Existing studies have shown that poisoned external corpora can mislead LLM responses; yet, there is still a lack of studies on the effects of different evidence properties. To bridge this gap, we design comprehensive experiments along three dimensions: styles of evidence, quantity of evidence, and the semantic similarity between external messages and the model{'}s internal belief. We find that instructive-style evidence demonstrates the most severe performance degradation. On the other hand, we observe a steady decline in model response quality as the amount of false evidence accumulates. Finally, we show that LLMs are more susceptible to factually incorrect evidence when their semantic similarity is close to the model{'}s parametric knowledge."
}