A Picture is Worth a Thousand Words: Language Models Plan from Pixels
Anthony Liu, Lajanugen Logeswaran, Sungryull Sohn, Honglak Lee
Abstract
Planning is an important capability of artificial agents that perform long-horizon tasks in real-world environments. In this work, we explore the use of pre-trained language models (PLMs) to reason about plan sequences from text instructions in embodied visual environments. Prior PLM based approaches for planning either assume observations are available in the form of text by a captioning model, reason about plans from the instruction alone, or incorporate information about the visual environment in limited ways (such as a pre-trained affordance function). In contrast, we show that the PLM can accurately plan even when observations are directly encoded as input prompts for the PLM. We show this simple approach outperforms prior approaches in experiments on the ALFWorld and VirtualHome benchmarks.- Anthology ID:
- 2023.emnlp-main.1025
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 16450–16459
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.1025
- DOI:
- 10.18653/v1/2023.emnlp-main.1025
- Cite (ACL):
- Anthony Liu, Lajanugen Logeswaran, Sungryull Sohn, and Honglak Lee. 2023. A Picture is Worth a Thousand Words: Language Models Plan from Pixels. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16450–16459, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- A Picture is Worth a Thousand Words: Language Models Plan from Pixels (Liu et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/ingest-bitext-workshop/2023.emnlp-main.1025.pdf