TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering
Andrey Sakhovskiy, Mikhail Salnikov, Irina Nikishina, Aida Usmanova, Angelie Kraft, Cedric Möller, Debayan Banerjee, Junbo Huang, Longquan Jiang, Rana Abdullah, Xi Yan, Dmitry Ustalov, Elena Tutubalina, Ricardo Usbeck, Alexander Panchenko
Abstract
This paper describes the results of the Knowledge Graph Question Answering (KGQA) shared task that was co-located with the TextGraphs 2024 workshop. In this task, given a textual question and a list of entities with the corresponding KG subgraphs, the participating system should choose the entity that correctly answers the question. Our competition attracted thirty teams, four of which outperformed our strong ChatGPT-based zero-shot baseline. In this paper, we overview the participating systems and analyze their performance according to a large-scale automatic evaluation. To the best of our knowledge, this is the first competition aimed at the KGQA problem using the interaction between large language models (LLMs) and knowledge graphs.- Anthology ID:
- 2024.textgraphs-1.9
- Volume:
- Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Dmitry Ustalov, Yanjun Gao, Alexander Panchenko, Elena Tutubalina, Irina Nikishina, Arti Ramesh, Andrey Sakhovskiy, Ricardo Usbeck, Gerald Penn, Marco Valentino
- Venues:
- TextGraphs | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 116–125
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2024.textgraphs-1.9/
- DOI:
- Cite (ACL):
- Andrey Sakhovskiy, Mikhail Salnikov, Irina Nikishina, Aida Usmanova, Angelie Kraft, Cedric Möller, Debayan Banerjee, Junbo Huang, Longquan Jiang, Rana Abdullah, Xi Yan, Dmitry Ustalov, Elena Tutubalina, Ricardo Usbeck, and Alexander Panchenko. 2024. TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering. In Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing, pages 116–125, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering (Sakhovskiy et al., TextGraphs 2024)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2024.textgraphs-1.9.pdf