@inproceedings{peng-etal-2025-m3gqa,
    title = "{M}{\textthreesuperior}{GQA}: A Multi-Entity Multi-Hop Multi-Setting Graph Question Answering Benchmark",
    author = "Peng, Boci  and
      Liu, Yongchao  and
      Bo, Xiaohe  and
      Guo, Jiaxin  and
      Zhu, Yun  and
      Fan, Xuanbo  and
      Hong, Chuntao  and
      Zhang, Yan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.acl-long.1478/",
    pages = "30594--30620",
    ISBN = "979-8-89176-251-0",
    abstract = "Recently, GraphRAG systems have achieved remarkable progress in enhancing the performance and reliability of large language models (LLMs). However, most previous benchmarks are template-based and primarily focus on few-entity queries, which are monotypic and simplistic, failing to offer comprehensive and robust assessments. Besides, the lack of ground-truth reasoning paths also hinders the assessments of different components in GraphRAG systems. To address these limitations, we propose M{\textthreesuperior}GQA, a complex, diverse, and high-quality GraphRAG benchmark focusing on multi-entity queries, with six distinct settings for comprehensive evaluation. In order to construct diverse data with semantically correct ground-truth reasoning paths, we introduce a novel reasoning-driven four-step data construction method, including tree sampling, reasoning path backtracking, query creation, and multi-stage refinement and filtering. Extensive experiments demonstrate that M{\textthreesuperior}GQA effectively reflects the capabilities of GraphRAG methods, offering valuable insights into the model performance and reliability. By pushing the boundaries of current methods, M{\textthreesuperior}GQA establishes a comprehensive, robust, and reliable benchmark for advancing GraphRAG research."
}