@inproceedings{al-azani-etal-2025-ontologyrag,
    title = "{O}ntology{RAG}-{Q}: Resource Development and Benchmarking for Retrieval-Augmented Question Answering in Qur{'}anic Tafsir",
    author = "Al-Azani, Sadam  and
      Alowaifeer, Maad  and
      Alhunief, Alhanoof  and
      Abdelali, Ahmed",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.784/",
    pages = "15551--15569",
    ISBN = "979-8-89176-332-6",
    abstract = "This paper introduces essential resources for Qur{'}anic studies: an annotated Tafsir ontology, a dataset of approximately 4,200 question-answer pairs, and a collection of 15 structured Tafsir books available in two formats. We present a comprehensive framework for handling sensitive Qur{'}anic Tafsir data that spans the entire pipeline from dataset construction through evaluation and error analysis. Our work establishes new benchmarks for retrieval and question-answering tasks on Qur{'}anic content, comparing performance across state-of-the-art embedding models and large language models (LLMs).We introduce OntologyRAG-Q, a novel retrieval-augmented generation approach featuring our custom Ayat-Ontology chunking method that segments Tafsir content at the verse level using ontology-driven structure. Benchmarking reveals strong performance across various LLMs, with GPT-4 achieving the highest results, followed closely by ALLaM. Expert evaluations show our system achieves 69.52{\%} accuracy and 74.36{\%} correctness overall, though multi-hop and context-dependent questions remain challenging. Our analysis demonstrates that answer position within documents significantly impacts retrieval performance, and among the evaluation metrics tested, BERT-recall and BERT-F1 correlate most strongly with expert assessments. The resources developed in this study are publicly available at \url{https://github.com/sazani/OntologyRAG-Q.git}."
}