@inproceedings{vanmechelen-etal-2026-quantifying,
    title = "Quantifying Retrieval Quality in {G}raph{RAG}: A Schema-Agnostic Approach",
    author = "Vanmechelen, Thibaud  and
      Achten, Alexandre  and
      Gabsi, Zaineb  and
      Skhiri, Sabri",
    editor = "S{\'e}rasset, Gilles  and
      Gkirtzou, Katerina  and
      Cochez, Michael  and
      Kalo, Jan-Christoph",
    booktitle = "Proceedings of the Knowledge Graphs and Large Language Models Workshop ({KG}-{LLM}) @ {LREC}26",
    month = may,
    year = "2026",
    address = "Palma, Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/paragraph-normalization/2026.kallm-1.18/",
    doi = "10.63317/5h4oct2t73a2",
    pages = "176--189",
    abstract = "While LLMs have achieved significant success in natural language tasks, their tendency to hallucinate remains a critical challenge. RAG tries to address this issue by grounding models in external data; however, standard vector-based RAGs often fail when working with highly interconnected datasets. GraphRAG has emerged as a superior alternative in this setting by modelling the relational topology, yet evaluating GraphRAGs remains challenging. Current benchmarks predominantly focus on the final LLM-generated output frequently overlooking the structural accuracy of the underlying retrieval process. In this paper, we propose a novel schema-agnostic framework for the automated generation of synthetic evaluation datasets from KGs. Unlike previous approaches, our framework establishes a rigorous, deterministic ground truth to specifically quantify the retriever performance across nine distinct query categories, including multi-hop and aggregation tasks. We demonstrate the utility of this benchmark by applying it to a biochemical KG and evaluating four diverse retrieval architectures. Our results indicate that agentic, LLM-driven retrievers provide the highest recall and reasoning capacity, effectively navigating complex topologies where other methods struggle. This work provides a robust, scalable methodology for performance tracking, shifting the evaluation of GraphRAG toward a more topologically precise standard."
}