Mahmoud Alhirthani
2026
From RAG to Agentic RAG for Faithful Islamic Question Answering
Gagan Bhatia | Hamdy Mubarak | Mustafa Jarrar | George Mikros | Fadi Zaraket | Mahmoud Alhirthani | Mutaz al-Khatib | Logan Cochrane | Kareem Mohamed Darwish | Rashid Yahiaoui | Firoj Alam
Findings of the Association for Computational Linguistics: ACL 2026
Gagan Bhatia | Hamdy Mubarak | Mustafa Jarrar | George Mikros | Fadi Zaraket | Mahmoud Alhirthani | Mutaz al-Khatib | Logan Cochrane | Kareem Mohamed Darwish | Rashid Yahiaoui | Firoj Alam
Findings of the Association for Computational Linguistics: ACL 2026
Large Language Models (LLMs) are increasingly used for Islamic question answering, where ungrounded responses may carry serious religious consequences. Yet standard MCQ/MRC-style evaluations do not capture key real-world failure modes, notably free-form hallucinations and the ability to abstain when evidence is insufficient. To address this gap, we introduce IslamicFaithQA, a 3,810-item bilingual (Arabic/English) **generative** benchmark with atomic single-gold answers, which enables direct measurement of hallucination and abstention. We additionally developed an end-to-end grounded Islamic modeling suite consisting of *(i)* 25K Arabic text-grounded SFT reasoning pairs, *(ii)* 5K bilingual preference samples for reward-guided alignment, and *(iii)* a verse-level Qur’an retrieval corpus of ∼6k atomic *verses* (ayat). Building on these resources, we develop an agentic Quran-grounding framework (agentic RAG) that uses structured tool calls for iterative evidence seeking and answer revision. Experiments across Arabic-centric and multilingual LLMs show that retrieval improves correctness and that agentic RAG yields the largest gains beyond standard RAG, achieving state-of-the-art performance and stronger Arabic–English robustness even with a small model (i.e., Qwen3 4B). We made the datasets are publicly available (https://huggingface.co/datasets/QCRI/IslamicFaithQA).