Reasoning Like Program Executors

Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Qiang Fu, Yan Gao, Jian-Guang Lou, Weizhu Chen

Abstract
Reasoning over natural language is a long-standing goal for the research community. However, studies have shown that existing language models are inadequate in reasoning. To address the issue, we present POET, a novel reasoning pre-training paradigm. Through pre-training language models with programs and their execution results, POET empowers language models to harvest the reasoning knowledge possessed by program executors via a data-driven approach. POET is conceptually simple and can be instantiated by different kinds of program executors. In this paper, we showcase two simple instances POET-Math and POET-Logic, in addition to a complex instance, POET-SQL. Experimental results on six benchmarks demonstrate that POET can significantly boost model performance in natural language reasoning, such as numerical reasoning, logical reasoning, and multi-hop reasoning. POET opens a new gate on reasoning-enhancement pre-training, and we hope our analysis would shed light on the future research of reasoning like program executors.
Anthology ID:
2022.emnlp-main.48
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
761–779
Language:
URL:
https://preview.aclanthology.org/icon-24-ingestion/2022.emnlp-main.48/
DOI:
10.18653/v1/2022.emnlp-main.48
Bibkey:
Cite (ACL):
Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Qiang Fu, Yan Gao, Jian-Guang Lou, and Weizhu Chen. 2022. Reasoning Like Program Executors. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 761–779, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Reasoning Like Program Executors (Pi et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/icon-24-ingestion/2022.emnlp-main.48.pdf
PDF Search Fix data