Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction
Samuel Broscheit, Kiril Gashteovski, Yanjie Wang, Rainer Gemulla
Abstract
Open Information Extraction systems extract (“subject text”, “relation text”, “object text”) triples from raw text. Some triples are textual versions of facts, i.e., non-canonicalized mentions of entities and relations. In this paper, we investigate whether it is possible to infer new facts directly from the open knowledge graph without any canonicalization or any supervision from curated knowledge. For this purpose, we propose the open link prediction task,i.e., predicting test facts by completing (“subject text”, “relation text”, ?) questions. An evaluation in such a setup raises the question if a correct prediction is actually a new fact that was induced by reasoning over the open knowledge graph or if it can be trivially explained. For example, facts can appear in different paraphrased textual variants, which can lead to test leakage. To this end, we propose an evaluation protocol and a methodology for creating the open link prediction benchmark OlpBench. We performed experiments with a prototypical knowledge graph embedding model for openlink prediction. While the task is very challenging, our results suggests that it is possible to predict genuinely new facts, which can not be trivially explained.- Anthology ID:
- 2020.acl-main.209
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2296–2308
- Language:
- URL:
- https://aclanthology.org/2020.acl-main.209
- DOI:
- 10.18653/v1/2020.acl-main.209
- Cite (ACL):
- Samuel Broscheit, Kiril Gashteovski, Yanjie Wang, and Rainer Gemulla. 2020. Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2296–2308, Online. Association for Computational Linguistics.
- Cite (Informal):
- Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction (Broscheit et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2020.acl-main.209.pdf
- Code
- samuelbroscheit/open_knowledge_graph_embeddings
- Data
- OLPBENCH