It Ain’t Over: A Multi-aspect Diverse Math Word Problem Dataset
Jiwoo Kim, Youngbin Kim, Ilwoong Baek, JinYeong Bak, Jongwuk Lee
Abstract
The math word problem (MWP) is a complex task that requires natural language understanding and logical reasoning to extract key knowledge from natural language narratives. Previous studies have provided various MWP datasets but lack diversity in problem types, lexical usage patterns, languages, and annotations for intermediate solutions. To address these limitations, we introduce a new MWP dataset, named DMath (Diverse Math Word Problems), offering a wide range of diversity in problem types, lexical usage patterns, languages, and intermediate solutions. The problems are available in English and Korean and include an expression tree and Python code as intermediate solutions. Through extensive experiments, we demonstrate that the DMath dataset provides a new opportunity to evaluate the capability of large language models, i.e., GPT-4 only achieves about 75% accuracy on the DMath dataset.- Anthology ID:
- 2023.emnlp-main.927
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 14984–15011
- Language:
- URL:
- https://preview.aclanthology.org/icon-24-ingestion/2023.emnlp-main.927/
- DOI:
- 10.18653/v1/2023.emnlp-main.927
- Cite (ACL):
- Jiwoo Kim, Youngbin Kim, Ilwoong Baek, JinYeong Bak, and Jongwuk Lee. 2023. It Ain’t Over: A Multi-aspect Diverse Math Word Problem Dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14984–15011, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- It Ain’t Over: A Multi-aspect Diverse Math Word Problem Dataset (Kim et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/icon-24-ingestion/2023.emnlp-main.927.pdf