@inproceedings{verma-etal-2025-adaptagent,
    title = "{A}dapt{A}gent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations",
    author = "Verma, Gaurav  and
      Kaur, Rachneet  and
      Srishankar, Nishan  and
      Zeng, Zhen  and
      Balch, Tucker  and
      Veloso, Manuela",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1008/",
    pages = "20635--20651",
    ISBN = "979-8-89176-251-0",
    abstract = "State-of-the-art multimodal web agents, powered by Multimodal Large Language Models (MLLMs), can autonomously execute many web tasks by processing user instructions and interacting with graphical user interfaces (GUIs). Current strategies for building web agents rely on (i) the generalizability of underlying MLLMs and their steerability via prompting, and (ii) large-scale fine-tuning of MLLMs on web-related tasks. However, web agents still struggle to automate tasks on unseen websites and domains, limiting their applicability to enterprise-specific and proprietary platforms. Beyond generalization from large-scale pre-training and fine-tuning, we propose building agents for few-shot adaptability using human demonstrations. We introduce the AdaptAgent framework that enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations (up to 2). Our experiments on two popular benchmarks {---} Mind2Web {\&} VisualWebArena {---} show that using in-context demonstrations (for proprietary models) or meta-adaptation demonstrations (for meta-learned open-weights models) boosts task success rate by 3.36{\%} to 7.21{\%} over non-adapted state-of-the-art models, corresponding to a relative increase of 21.03{\%} to 65.75{\%}. Furthermore, our additional analyses (a) show the effectiveness of multimodal demonstrations over text-only ones, (b) illuminate how different meta-learning data selection strategies influence the agent{'}s generalization, and (c) demonstrate how the number of few-shot examples affects the web agent{'}s success rate. Our results offer a complementary axis for developing widely applicable multimodal web agents beyond large-scale pre-training and fine-tuning, emphasizing few-shot adaptability."
}