# Image, Tell me your story! Predicting the original meta-context of visual misinformation

This repository contains the code and dataset accompanying the paper: "Image, Tell me your story!" Predicting the original meta-context of visual misinformation.

### Abstract 
To assist human fact-checkers, researchers have developed automated approaches for visual misinformation detection. These  methods assign veracity scores by identifying inconsistencies between the image and its caption, or by detecting forgeries in the image.
However, they neglect a crucial point of the human fact-checking process: identifying the original meta-context of the image. By explaining what is actually true about the image, fact-checkers can better detect misinformation, focus their efforts on check-worthy visual content, engage in counter-messaging before misinformation spreads widely, and make their explanation more convincing. Here, we fill this gap by introducing the task of automated image contextualization. We create 5Pils, a dataset of  1,676 fact-checked images with question-answer pairs about their original meta-context. Annotations are based on the 5 Pillars fact-checking framework. We implement a first baseline that grounds the image in its original meta-context using the content of the image and textual evidence retrieved from the open web. Our experiments show promising results while highlighting several open challenges in retrieval and reasoning.

<p align="center">
  <img width="80%" src="assets/introducory_example.png" alt="header" />
</p>

### 5Pils dataset

The 5Pils dataset consists of 1,676 fact-checked images annotated with question-answer pairs based on the 5 Pillars framework for image contextualization.

- The dataset annotations are contained in the train.json, val.json, and test.json files of the dataset folder.
- The URLs of the FC articles from which the annotations were extracted are in dataset/url/article_urls.txt
- The URLs of the retrieved evidence are in dataset/retrieval_results/evidence_url.json

### How to use?

To collec the FC articles and the images of the 5Pils dataset from their urls:

```
$ python scripts/build_dataset_from_url.py
```

We also provide the code to reproduce the collection of the FC articles from the Wayback Machine

```
$ python scripts/data_collection_waybackmachine.py
```

To collect the evidence 
```
$ python scripts/collect_RIS_evidence.py
```

To detect the original versions of manipulated images
```
$ python scripts/get_manipulated_images.py
$ python scripts/get_originals_of_manipulated.py
```

To compute embeddings for evidence ranking and few-shot demonstration selection
```
$ python scripts/get_embeddings.py
```

To generate answers for a specific pillar and evaluate the results
```
$ python scripts/get_5pillars_answers.py
$ python scripts/evaluate_answer_generation.py
```

### Google Vision API and Azure OpenAI service

- Retrieving evidence with the web detection tool of Google Vision API requires an active Google Cloud account with an API key.
- Using GPT4(-Vision) with the Azure OpenAI service requires an Azure account.


### Requirements

```
$ conda create --name 5Pils python=3.9
$ conda activate 5Pils
$ pip install -r requirements.txt
$ python -m spacy download en_core_web_lg
```