@inproceedings{semnani-etal-2025-detecting,
    title = "Detecting Corpus-Level Knowledge Inconsistencies in {W}ikipedia with Large Language Models",
    author = "Semnani, Sina  and
      Burapacheep, Jirayu  and
      Khatua, Arpandeep  and
      Atchariyachanvanit, Thanawan  and
      Wang, Zheng  and
      Lam, Monica",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-luhme/2025.emnlp-main.1765/",
    doi = "10.18653/v1/2025.emnlp-main.1765",
    pages = "34827--34854",
    ISBN = "979-8-89176-332-6",
    abstract = "Wikipedia is the largest open knowledge corpus, widely used worldwide and serving as a key resource for training large language models (LLMs) and retrieval-augmented generation (RAG) systems. Ensuring its accuracy is therefore critical. But how accurate is Wikipedia, and how can we improve it?We focus on inconsistencies, a specific type of factual inaccuracy, and introduce the task of corpus-level inconsistency detection. We present CLAIRE, an agentic system that combines LLM reasoning with retrieval to surface potentially inconsistent claims along with contextual evidence for human review. In a user study with experienced Wikipedia editors, 87.5{\%} reported higher confidence when using CLAIRE, and participants identified 64.7{\%} more inconsistencies in the same amount of time.Combining CLAIRE with human annotation, we contribute WIKICOLLIDE, the first benchmark of real Wikipedia inconsistencies. Using random sampling with CLAIRE-assisted analysis, we find that at least 3.3{\%} of English Wikipedia facts contradict another fact, with inconsistencies propagating into 7.3{\%} of FEVEROUS and 4.0{\%} of AmbigQA examples. Benchmarking strong baselines on this dataset reveals substantial headroom: the best fully automated system achieves an AUROC of only 75.1{\%}.Our results show that contradictions are a measurable component of Wikipedia and that LLM-based systems like CLAIRE can provide a practical tool to help editors improve knowledge consistency at scale."
}