Abstract
This paper investigates the performance of Large Language Models (LLMs) and Tool-augmented LLMs in tackling complex mathematical reasoning tasks. We introduce IMR-TIP: Improving Math Reasoning with Tool-augmented Interleaf Prompting, a framework that combines the strengths of both LLMs and Tool-augmented LLMs. IMR-TIP follows the “From Good to Great” concept, collecting multiple potential solutions from both LLMs and their Tool-Augmented counterparts for the same math problem, and then selecting or re-generating the most accurate answer after cross-checking these solutions via tool-augmented interleaf prompting. The framework incorporates two key aspects: self-prompt and tool-augmented interleaf prompting (TIP). The former allows LLMs to autonomously refine and improve an initial prompt related to tool usage, while the latter enables LLMs to derive the final answer by dynamically analyzing the problem, cross-checking potential solutions, and revising previous reasoning hints in an interleaved manner. Experimental analysis shows that IMR-TIP achieves enhanced mathematical capabilities and outperforms traditional LLMs and tool-augmented LLMs in accuracy and reasoning diversity on math reasoning tasks. For instance, IMR-TIP can improve Tool-augmented ChatGPT on GSM8K-Hard from 56.0% to 65.2 %.- Anthology ID:
- 2024.nlrse-1.7
- Volume:
- Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ACL 2024)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Ben Lipkin, Danilo Neves Ribeiro, Lionel Wong, Xi Ye, Wenting Zhao
- Venues:
- NLRSE | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 64–79
- Language:
- URL:
- https://aclanthology.org/2024.nlrse-1.7
- DOI:
- Cite (ACL):
- Nuo Chen, Hongguang Li, Baoyuan Wang, and Jia Li. 2024. From Good to Great: Improving Math Reasoning with Tool-Augmented Interleaf Prompting. In Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ACL 2024), pages 64–79, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- From Good to Great: Improving Math Reasoning with Tool-Augmented Interleaf Prompting (Chen et al., NLRSE-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.nlrse-1.7.pdf