WebWISE: Unlocking Web Interface Control for LLMs via Sequential Exploration
Heyi Tao, Sethuraman T V, Michal Shlapentokh-Rothman, Tanmay Gupta, Heng Ji, Derek Hoiem
Abstract
This paper investigates using Large Language Models (LLMs) to automatically perform web software tasks using click, scroll, and text in- put operations. Previous approaches, such as reinforcement learning (RL) or imitation learning, are inefficient to train and task-specific. Our method uses filtered Document Object Model (DOM) elements as observations and performs tasks step-by-step, sequentially generating small programs based on the current observations. We use in-context learning, either benefiting from a single manually provided example, or an automatically generated example based on a successful zero-shot trial. We evaluate our proposed method on the MiniWob++ benchmark. With only one in-context example, our WebWISE method using gpt-3.5-turbo achieves similar or better performance than other methods that require many demonstrations or trials.- Anthology ID:
- 2024.findings-naacl.234
- Volume:
- Findings of the Association for Computational Linguistics: NAACL 2024
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Kevin Duh, Helena Gomez, Steven Bethard
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3693–3711
- Language:
- URL:
- https://aclanthology.org/2024.findings-naacl.234
- DOI:
- Cite (ACL):
- Heyi Tao, Sethuraman T V, Michal Shlapentokh-Rothman, Tanmay Gupta, Heng Ji, and Derek Hoiem. 2024. WebWISE: Unlocking Web Interface Control for LLMs via Sequential Exploration. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3693–3711, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- WebWISE: Unlocking Web Interface Control for LLMs via Sequential Exploration (Tao et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2024.findings-naacl.234.pdf