Symbolic Working Memory Enhances Language Models for Complex Rule Application

Siyuan Wang, Zhongyu Wei, Yejin Choi, Xiang Ren

Abstract
Large Language Models (LLMs) have shown remarkable reasoning performance but struggle with multi-step deductive reasoning involving a series of rule application steps, especially when rules are presented non-sequentially. Our preliminary analysis shows that while LLMs excel in single-step rule application, their performance drops significantly in multi-step scenarios due to the challenge in rule grounding. It requires anchoring the applicable rule and supporting facts at each step, amidst multiple input rules, facts, and inferred facts. To address this, we propose augmenting LLMs with external working memory and introduce a neurosymbolic framework for rule application. The memory stores facts and rules in both natural language and symbolic forms, enabling precise tracking. Utilizing this memory, our framework iteratively performs symbolic rule grounding and LLM-based rule implementation. The former matches predicates and variables of symbolic rules and facts to ground applicable rules at each step. Experiments indicate our framework’s effectiveness in rule application and its robustness across various steps and settings.
Anthology ID:
2024.emnlp-main.974
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
17583–17604
Language:
URL:
https://preview.aclanthology.org/build-pipeline-with-new-library/2024.emnlp-main.974/
DOI:
10.18653/v1/2024.emnlp-main.974
Bibkey:
Cite (ACL):
Siyuan Wang, Zhongyu Wei, Yejin Choi, and Xiang Ren. 2024. Symbolic Working Memory Enhances Language Models for Complex Rule Application. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17583–17604, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Symbolic Working Memory Enhances Language Models for Complex Rule Application (Wang et al., EMNLP 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/build-pipeline-with-new-library/2024.emnlp-main.974.pdf
PDF Search Fix metadata