@inproceedings{ando-etal-2023-evaluating,
    title = "Evaluating Large Language Models with {N}eu{BAROCO}: Syllogistic Reasoning Ability and Human-like Biases",
    author = "Ando, Risako  and
      Morishita, Takanobu  and
      Abe, Hirohiko  and
      Mineshima, Koji  and
      Okada, Mitsuhiro",
    editor = "Chatzikyriakidis, Stergios  and
      de Paiva, Valeria",
    booktitle = "Proceedings of the 4th Natural Logic Meets Machine Learning Workshop",
    month = jun,
    year = "2023",
    address = "Nancy, France",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/build-pipeline-with-new-library/2023.naloma-1.1/",
    pages = "1--11",
    abstract = "This paper investigates whether current large language models exhibit biases in logical reasoning, similar to humans. Specifically, we focus on syllogistic reasoning, a well-studied form of inference in the cognitive science of human deduction. To facilitate our analysis, we introduce a dataset called NeuBAROCO, originally designed for psychological experiments that assess human logical abilities in syllogistic reasoning. The dataset consists of syllogistic inferences in both English and Japanese. We examine three types of biases observed in human syllogistic reasoning: belief biases, conversion errors, and atmosphere effects. Our findings demonstrate that current large language models struggle more with problems involving these three types of biases."
}