Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA

Dhruv Agarwal, Rajarshi Das, Sopan Khosla, Rashmi Gangadharaiah

Abstract
We present BYOKG, a universal question-answering (QA) system that can operate on any knowledge graph (KG), requires no human-annotated training data, and can be ready to use within a day—attributes that are out-of-scope for current KGQA systems. BYOKG draws inspiration from the remarkable ability of humans to comprehend information present in an unseen KG through exploration—starting at random nodes, inspecting the labels of adjacent nodes and edges, and combining them with their prior world knowledge. Exploration in BYOKG leverages an LLM-backed symbolic agent that generates a diverse set of query-program exemplars, which are then used to ground a retrieval-augmented reasoning procedure to synthesize programs for arbitrary questions. BYOKG is effective over both small- and large-scale graphs, showing dramatic gains in zero-shot QA accuracy of 27.89 and 59.88 F1 on GrailQA and MetaQA, respectively. We further find that performance of BYOKG reliably improves with continued exploration as well as improvements in the base LLM, notably outperforming a state-of-the-art fine-tuned model by 7.08 F1 on a sub-sampled zero-shot split of GrailQA. Lastly, we verify our universality claim by evaluating BYOKG on a domain-specific materials science KG and show that it improves zero-shot performance by 46.33 F1.
Anthology ID:
2024.findings-naacl.57
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
896–919
Language:
URL:
https://aclanthology.org/2024.findings-naacl.57
DOI:
10.18653/v1/2024.findings-naacl.57
Bibkey:
Cite (ACL):
Dhruv Agarwal, Rajarshi Das, Sopan Khosla, and Rashmi Gangadharaiah. 2024. Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 896–919, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA (Agarwal et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl-24-ws-corrections/2024.findings-naacl.57.pdf
PDF Search