@inproceedings{jandial-etal-2025-fine,
    title = "On the Fine-Grained Planning Abilities of {VLM} Web Agents",
    author = "Jandial, Surgan  and
      Wang, Yinong Oliver  and
      Bajcsy, Andrea  and
      De la Torre, Fernando",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.1382/",
    doi = "10.18653/v1/2025.findings-emnlp.1382",
    pages = "25347--25380",
    ISBN = "979-8-89176-335-7",
    abstract = "Vision-Language Models (VLMs) have shown promise as web agents, yet their planning{---}the ability to devise strategies or action sequences to complete tasks{---}remains understudied. While prior works focus on VLM{'}s perception and overall success rates (i.e., goal completion), fine-grained investigation of their planning has been overlooked. To address this gap, we examine VLMs' capability to (1) understand temporal relationships within web contexts, and (2) assess plans of actions across diverse scenarios. We design four simple yet effective tests to delve into these nuanced aspects around planning. Our results across nineteen VLMs reveal that these models exhibit limited performance in the aforementioned skills and are not reliable to function as web agents. To facilitate future work, we release our planning evaluations and data, providing a foundation for advancing the future research in this area."
}