@article{nguyen-etal-2026-bis,
    title = "{BIS} Reasoning 1.0: The First Large-Scale {J}apanese Benchmark for Belief-Inconsistent Syllogistic Reasoning",
    author = "Nguyen, Ha Thanh  and
      Tachibana, Hideyuki  and
      Liu, Chaoran  and
      Liu, Qianying  and
      Myat Noe, Su  and
      Takeda, Koichi  and
      Kurohashi, Sadao",
    editor = "Piperidis, Stelios  and
      Bel, N{\'u}ria  and
      van den Heuvel, Henk  and
      Ide, Nancy  and
      Krek, Simon  and
      Toral, Antonio",
    journal = "International Conference on Language Resources and Evaluation",
    volume = "main",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resource Association",
    url = "https://preview.aclanthology.org/ingest-lrec/2026.lrec-main.173/",
    pages = "2211--2219",
    abstract = "We present BIS Reasoning 1.0, the first large-scale Japanese dataset of syllogistic reasoning problems explicitly designed to evaluate belief-inconsistent reasoning in large language models (LLMs). Unlike prior resources such as NeuBAROCO and JFLD, which emphasize general or belief-aligned logic, BIS Reasoning 1.0 systematically introduces logically valid yet belief-inconsistent syllogisms to expose belief bias{---}the tendency to accept believable conclusions irrespective of validity. We benchmark a representative suite of cutting-edge models{---}including OpenAI GPT-5 variants, GPT-4o, Qwen, and prominent Japanese LLMs{---}under a uniform, zero-shot protocol. Reasoning-centric models achieve near-perfect accuracy on BIS Reasoning 1.0 (e.g., Qwen3-32B {\ensuremath{\approx}}99{\%} and GPT-5-mini up to {\ensuremath{\approx}}99.7{\%}), while GPT-4o attains around 80{\%}. Earlier Japanese-specialized models underperform, often well below 60{\%}, whereas the latest llm-jp-3.1-13b-instruct4 markedly improves to the mid-80{\%} range. These results indicate that robustness to belief-inconsistent inputs is driven more by explicit reasoning optimization than by language specialization or scale alone. Our analysis further shows that even top-tier systems falter when logical validity conflicts with intuitive or factual beliefs, and that performance is sensitive to prompt design and inference-time reasoning effort. We discuss implications for safety-critical domains{---}law, healthcare, and scientific literature{---}where strict logical fidelity must override intuitive belief to ensure reliability."
}