SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, Parisa Kordjamshidi
Abstract
This paper proposes a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reasoning rules to automatically generate a spatial description of visual scenes and corresponding QA pairs. Experiments show that further pretraining LMs on these automatically generated data significantly improves LMs’ capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster investigations into more sophisticated models for spatial reasoning over text.- Anthology ID:
- 2021.naacl-main.364
- Volume:
- Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Month:
- June
- Year:
- 2021
- Address:
- Online
- Editors:
- Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4582–4598
- Language:
- URL:
- https://aclanthology.org/2021.naacl-main.364
- DOI:
- 10.18653/v1/2021.naacl-main.364
- Cite (ACL):
- Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. 2021. SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4582–4598, Online. Association for Computational Linguistics.
- Cite (Informal):
- SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning (Mirzaee et al., NAACL 2021)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2021.naacl-main.364.pdf
- Code
- HLR/SpartQA-baselines + additional community code
- Data
- SPARTQA -, BoolQ, GLUE, NLVR