Abstract
Mathematical reasoning is regarded as a necessary ability for Language Models (LMs). Recent works demonstrate large LMs’ impressive performance in solving math problems. The success is attributed to their Chain-of-Thought (CoT) reasoning abilities, i.e., the ability to decompose complex questions into step-by-step reasoning chains, but such ability seems only to emerge from models with abundant parameters. This work investigates how to incorporate relatively small LMs with the capabilities of multi-step reasoning. We propose to inject such abilities by continually pre-training LMs on a synthetic dataset MsAT which is composed of Multi-step Arithmetic Tasks. Our experiments on four math word problem datasets show the effectiveness of the proposed method in enhancing LMs’ math reasoning abilities.- Anthology ID:
- 2023.acl-short.106
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1229–1238
- Language:
- URL:
- https://aclanthology.org/2023.acl-short.106
- DOI:
- Cite (ACL):
- Tianduo Wang and Wei Lu. 2023. Learning Multi-Step Reasoning by Solving Arithmetic Tasks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1229–1238, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Learning Multi-Step Reasoning by Solving Arithmetic Tasks (Wang & Lu, ACL 2023)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2023.acl-short.106.pdf