Abstract
In this paper we examine the limitations of Large Language Models (LLMs) for complex reasoning tasks. While current approaches leverage formal languages as intermediate representation for these reasoning problems, they still struggle with generating intermediate for-mal specifications with great correctness and in refining these representations. To address these issues, this paper proposes Logic-LM++, an improvement on Logic-LM (Pan et al., 2023). It uses the ability of LLMs to do pairwise comparisons, allowing the evaluation of the refinements suggested by the LLM. The paper demonstrates that Logic-LM++ outperforms Logic-LM and LLM based techniques on natural language reasoning tasks on two datasets, FOLIO, ProofWriter and AR-LSAT. Logic-LM++ show an average improvement of 18.5% on standard prompting, 12.3% on chain of thought prompting and 5% on Logic-LM.- Anthology ID:
- 2024.nlrse-1.6
- Volume:
- Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ACL 2024)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Ben Lipkin, Danilo Neves Ribeiro, Lionel Wong, Xi Ye, Wenting Zhao
- Venues:
- NLRSE | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 56–63
- Language:
- URL:
- https://aclanthology.org/2024.nlrse-1.6
- DOI:
- Cite (ACL):
- Shashank Kirtania, Priyanshu Gupta, and Arjun Radhakrishna. 2024. LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations. In Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ACL 2024), pages 56–63, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations (Kirtania et al., NLRSE-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.nlrse-1.6.pdf