@inproceedings{shafiei-etal-2026-truthtrap,
    title = "{T}ruth{T}rap: A Bilingual Benchmark for Evaluating Factually Correct Yet Misleading Information in Question Answering",
    author = "Shafiei, Mohammadamin  and
      Saffari, Hamidreza  and
      Pilehvar, Mohammad Taher  and
      Raganato, Alessandro",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.findings-eacl.155/",
    pages = "2966--2987",
    ISBN = "979-8-89176-386-9",
    abstract = "Large Language Models (LLMs) are increasingly used to answer factual, information-seeking questions (ISQs). While prior work often focuses on false, misleading information, little attention has been paid to true but strategically persuasive content that can derail a model{'}s reasoning. To address this gap, we introduce a new evaluation dataset, TruthTrap, in two languages, i.e., English and Farsi, on Iran-related ISQs, each paired with a correct explanation and a persuasive-yet-misleading true hint. We then evaluate nine diverse LLMs (spanning proprietary and open-source systems) via factuality classification and multiple-choice QA tasks, finding that accuracy drops by 25{\%}, on average, when models encounter these misleading yet factual hints. Also, the models' predictions match the hint-aligned options up to 77 percent of the time. Notably, models often misjudge such hints in isolation yet still integrate them into final answers. Our results highlight a significant limitation in LLM outputs, underscoring the importance of robust fact-verification and emphasizing real-world risks posed by partial truths in domains like social media, education, and policy-making."
}