Abstract
Generating intermediate steps, or Chain of Thought (CoT), is an effective way to significantly improve language models’ (LM) multi-step reasoning capability. However, the CoT lengths can grow rapidly with the problem complexity, easily exceeding the maximum context size. Instead of increasing the context limit, which has already been heavily investigated, we explore an orthogonal direction: making LMs divide a problem into multiple contexts. We propose a new inference framework, called Recursion of Thought (RoT), which introduces several special tokens that the models can output to trigger context-related operations. Extensive experiments with multiple architectures including GPT-3 show that RoT dramatically improves LMs’ inference capability to solve problems, whose solution consists of hundreds of thousands of tokens.- Anthology ID:
- 2023.findings-acl.40
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2023
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 623–658
- Language:
- URL:
- https://aclanthology.org/2023.findings-acl.40
- DOI:
- 10.18653/v1/2023.findings-acl.40
- Cite (ACL):
- Soochan Lee and Gunhee Kim. 2023. Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 623–658, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Models (Lee & Kim, Findings 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2023.findings-acl.40.pdf