WebWISE: Unlocking Web Interface Control for LLMs via Sequential Exploration

Heyi Tao, Sethuraman T V, Michal Shlapentokh-Rothman, Tanmay Gupta, Heng Ji, Derek Hoiem

Abstract
This paper investigates using Large Language Models (LLMs) to automatically perform web software tasks using click, scroll, and text in- put operations. Previous approaches, such as reinforcement learning (RL) or imitation learning, are inefficient to train and task-specific. Our method uses filtered Document Object Model (DOM) elements as observations and performs tasks step-by-step, sequentially generating small programs based on the current observations. We use in-context learning, either benefiting from a single manually provided example, or an automatically generated example based on a successful zero-shot trial. We evaluate our proposed method on the MiniWob++ benchmark. With only one in-context example, our WebWISE method using gpt-3.5-turbo achieves similar or better performance than other methods that require many demonstrations or trials.
Anthology ID:
2024.findings-naacl.234
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3693–3711
Language:
URL:
https://aclanthology.org/2024.findings-naacl.234
DOI:
10.18653/v1/2024.findings-naacl.234
Bibkey:
Cite (ACL):
Heyi Tao, Sethuraman T V, Michal Shlapentokh-Rothman, Tanmay Gupta, Heng Ji, and Derek Hoiem. 2024. WebWISE: Unlocking Web Interface Control for LLMs via Sequential Exploration. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3693–3711, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
WebWISE: Unlocking Web Interface Control for LLMs via Sequential Exploration (Tao et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.findings-naacl.234.pdf
PDF Search