This package includes the data and code to replicate results in Table 4 of the submitted paper. Specifically,
- `human_evaluations.json` includes human evaluations results. There are three judgments in "raw" and we take the majority vote as the final vote.
- `{LFRQA|RQA}_{RQA|gpt-4|llama-3|mixtral-large}-eval_by_gpt4-0125-preview.json` is the LLM-based evaluators' results. In these files, you can find RobustQA and LFRQA's annotation, together with LLMs generated answers. As Table 4 states, we sampled 100 queries per domain. So 700 queries in total.

To reproduce results, simply run `python code/compute_correlation.py`.

We will publish all data and code for RAG-QA Arena upon paper acceptance.
