Abstract
This paper describes our 3rd place submission in the AVeriTeC shared task in which we attempted to address the challenge of fact-checking with evidence retrieved in the wild using a simple scheme of Retrieval-Augmented Generation (RAG) designed for the task, leveraging the predictive power of Large Language Models.We release our codebase and explain its two modules - the Retriever and the Evidence & Label generator - in detail, justifying their features such as MMR-reranking and Likert-scale confidence estimation.We evaluate our solution on AVeriTeC dev and test set and interpret the results, picking the GPT-4o as the most appropriate model for our pipeline at the time of our publication, with Llama 3.1 70B being a promising open-source alternative.We perform an empirical error analysis to see that faults in our predictions often coincide with noise in the data or ambiguous fact-checks, provoking further research and data augmentation.- Anthology ID:
- 2024.fever-1.16
- Volume:
- Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER)
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Michael Schlichtkrull, Yulong Chen, Chenxi Whitehouse, Zhenyun Deng, Mubashara Akhtar, Rami Aly, Zhijiang Guo, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal, James Thorne, Andreas Vlachos
- Venue:
- FEVER
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 137–150
- Language:
- URL:
- https://aclanthology.org/2024.fever-1.16
- DOI:
- 10.18653/v1/2024.fever-1.16
- Cite (ACL):
- Herbert Ullrich, Tomáš Mlynář, and Jan Drchal. 2024. AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task. In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), pages 137–150, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task (Ullrich et al., FEVER 2024)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2024.fever-1.16.pdf