Abstract
Recent work on fact-checking addresses a realistic setting where models incorporate evidence retrieved from the web to decide the veracity of claims. A bottleneck in this pipeline is in retrieving relevant evidence: traditional methods may surface documents directly related to a claim, but fact-checking complex claims requires more inferences. For instance, a document about how a vaccine was developed is relevant to addressing claims about what it might contain, even if it does not address them directly. We present Contrastive Fact-Checking Reranker (CFR), an improved retriever for this setting. By leveraging the AVeriTeC dataset, which annotates subquestions for claims with human written answers from evidence documents, we fine-tune Contriever with a contrastive objective based on multiple training signals, including distillation from GPT-4, evaluating subquestion answers, and gold labels in the dataset. We evaluate our model on both retrieval and end-to-end veracity judgments about claims. On the AVeriTeC dataset, we find a 6% improvement in veracity classification accuracy. We also show our gains can be transferred to FEVER, ClaimDecomp, HotpotQA, and a synthetic dataset requiring retrievers to make inferences.- Anthology ID:
- 2024.fever-1.28
- Volume:
- Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER)
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Michael Schlichtkrull, Yulong Chen, Chenxi Whitehouse, Zhenyun Deng, Mubashara Akhtar, Rami Aly, Zhijiang Guo, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal, James Thorne, Andreas Vlachos
- Venue:
- FEVER
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 264–279
- Language:
- URL:
- https://aclanthology.org/2024.fever-1.28
- DOI:
- 10.18653/v1/2024.fever-1.28
- Cite (ACL):
- Aniruddh Sriram, Fangyuan Xu, Eunsol Choi, and Greg Durrett. 2024. Contrastive Learning to Improve Retrieval for Real-World Fact Checking. In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), pages 264–279, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- Contrastive Learning to Improve Retrieval for Real-World Fact Checking (Sriram et al., FEVER 2024)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2024.fever-1.28.pdf