Abstract
Semantic parsing plays a pivotal role in advancing the accessibility of human-computer interaction on a large scale. Spider, a widely recognized dataset for text2SQL, contains a wide range of natural language (NL) questions in English and corresponding SQL queries. Original splits of Spider and its adapted to Russian language and improved version, PAUQ, assume independence and identical distribution of training and testing data (i.i.d split). In this work, we propose a target length split and multilingual i.i.d split to measure compositionality and cross-language generalization. We present experimental results of popular text2SQL models on original, multilingual, and target length splits. We also construct a context-free grammar for the evaluation of compositionality in text2SQL in an out-of-distribution setting. We make the splits publicly available on HuggingFace hub via https://huggingface.co/datasets/composite/pauq- Anthology ID:
- 2023.genbench-1.18
- Volume:
- Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Dieuwke Hupkes, Verna Dankers, Khuyagbaatar Batsuren, Koustuv Sinha, Amirhossein Kazemnejad, Christos Christodoulopoulos, Ryan Cotterell, Elia Bruni
- Venues:
- GenBench | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 214–220
- Language:
- URL:
- https://aclanthology.org/2023.genbench-1.18
- DOI:
- 10.18653/v1/2023.genbench-1.18
- Cite (ACL):
- Oleg Somov and Elena Tutubalina. 2023. Shifted PAUQ: Distribution shift in text-to-SQL. In Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP, pages 214–220, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Shifted PAUQ: Distribution shift in text-to-SQL (Somov & Tutubalina, GenBench-WS 2023)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2023.genbench-1.18.pdf