<h1 align="center">Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models</h1>

<div style="text-align: center; width: 100%;">
  <!-- Container to align the image and the caption -->
  <div style="display: inline-block; text-align: left; width: 85%;">
    <img src="assets/images/knights_and_knaves.png" style="width: 100%;" alt="knights and knaves">
  </div>
</div>

<br>


Knights and knaves problems represent a classic genre of logical puzzles where characters either tell the truth or lie. The objective is to logically deduce each character's identity based on their statements. The challenge arises from the truth-telling or lying behavior, which influences the logical implications of each statement. Solving these puzzles requires not only direct deductions from individual statements, but the ability to assess the truthfulness of statements by reasoning through various hypothetical scenarios. As such, knights and knaves puzzles serve as compelling examples of suppositional reasoning. In this paper, we introduce \emph{TruthQuest}, a benchmark for suppositional reasoning based on the principles of knights and knaves puzzles. Our benchmark presents problems of varying complexity, considering both the number of characters and the types of logical statements involved. Evaluations on \emph{TruthQuest} show that large language models like Llama 3 and Mixtral-8x7B exhibit significant difficulties solving these tasks. A detailed error analysis of the models' output reveals that lower-performing models exhibit a diverse range of reasoning errors, frequently failing to grasp the concept of truth and lies. In comparison, more proficient models primarily struggle with accurately inferring the logical implications of potentially false statements.


## Table of Contents
- [Setup](#setup)
- [Generate Puzzles](#generate-puzzles)
- [Run Models](#run-models)

## Setup
All code was developed and tested on Ubuntu 22.04 with Python 3.11

To run the current code, we recommend to use Poetry:
```bash
poetry install                          # Install dependencies
poetry shell                            # Activate virtual environment
# Work for a while
deactivate
```

## Generate Puzzles
To generate puzzles, run the following command:
```
python gen_data.py --from-yaml
```

## Run Models
To run models, run the following command:
```
python run.py --model <hf-model-name>
```

## Evaluate Performance
To evaluate the performance of the models, run the following command:
```
python evaluate_conclusion.py
```

For specific command-line arguments, please refer to the code.