Abstract
Spatial reasoning is a fundamental building block of human cognition, used in representing, grounding, and reasoning about physical and abstract concepts. We propose a novel benchmark focused on assessing inferential properties of statements with spatial prepositions. The benchmark includes original datasets in English and Romanian and aims to probe the limits of reasoning about spatial relations in large language models. We use prompt engineering to study the performance of two families of large language models, PaLM and GPT-3, on our benchmark. Our results show considerable variability in the performance of smaller and larger models, as well as across prompts and languages. However, none of the models reaches human performance.- Anthology ID:
- 2023.emnlp-main.1015
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 16328–16335
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.1015
- DOI:
- 10.18653/v1/2023.emnlp-main.1015
- Cite (ACL):
- Iulia Comsa and Srini Narayanan. 2023. A Benchmark for Reasoning with Spatial Prepositions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16328–16335, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- A Benchmark for Reasoning with Spatial Prepositions (Comsa & Narayanan, EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2023.emnlp-main.1015.pdf