PROC2PDDL: Open-Domain Planning Representations from Texts
Tianyi Zhang, Li Zhang, Zhaoyi Hou, Ziyu Wang, Yuling Gu, Peter Clark, Chris Callison-Burch, Niket Tandon
Abstract
Planning in a text-based environment continues to be a significant challenge for AI systems. Recent approaches have utilized language models to predict planning domain definitions (e.g., PDDL) but have only been evaluated in closed-domain simulated environments. To address this, we present Proc2PDDL, the first dataset containing open-domain procedural texts paired with expert-annotated PDDL representations. Using this dataset, we evaluate the task of predicting domain actions (parameters, preconditions, and effects). We experiment with various large language models (LLMs) and prompting mechanisms, including a novel instruction inspired by the zone of proximal development (ZPD), which reconstructs the task as incremental basic skills. Our results demonstrate that Proc2PDDL is highly challenging for end-to-end LLMs, with GPT-3.5’s success rate close to 0% and GPT-4o’s 38%. With ZPD instructions, GPT-4o’s success rate increases to 45%, outperforming regular chain-of-thought prompting’s 34%. Our analysis systematically examines both syntactic and semantic errors, providing insights into the strengths and weaknesses of language models in generating domain-specific programs.- Anthology ID:
- 2024.nlrse-1.2
- Volume:
- Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ACL 2024)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Ben Lipkin, Danilo Neves Ribeiro, Lionel Wong, Xi Ye, Wenting Zhao
- Venues:
- NLRSE | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 13–24
- Language:
- URL:
- https://aclanthology.org/2024.nlrse-1.2
- DOI:
- Cite (ACL):
- Tianyi Zhang, Li Zhang, Zhaoyi Hou, Ziyu Wang, Yuling Gu, Peter Clark, Chris Callison-Burch, and Niket Tandon. 2024. PROC2PDDL: Open-Domain Planning Representations from Texts. In Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ACL 2024), pages 13–24, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- PROC2PDDL: Open-Domain Planning Representations from Texts (Zhang et al., NLRSE-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.nlrse-1.2.pdf