@inproceedings{xu-etal-2025-retrieval,
    title = "Retrieval-augmented {GUI} Agents with Generative Guidelines",
    author = "Xu, Ran  and
      Ma, Kaixin  and
      Yu, Wenhao  and
      Zhang, Hongming  and
      Ho, Joyce C.  and
      Yang, Carl  and
      Yu, Dong",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.902/",
    pages = "17877--17886",
    ISBN = "979-8-89176-332-6",
    abstract = "GUI agents powered by vision-language models (VLMs) show promise in automating complex digital tasks. However, their effectiveness in real-world applications is often limited by scarce training data and the inherent complexity of these tasks, which frequently require long-tailed knowledge covering rare, unseen scenarios. We propose RAG-GUI , a lightweight VLM that leverages web tutorials at inferencetime. RAG-GUI is first warm-started via supervised finetuning (SFT) and further refined through self-guided rejection sampling fine-tuning (RSF). Designed to be model-agnostic, RAG-GUI functions as a generic plug-in that enhances any VLM-based agent. Evaluatedacross three distinct tasks, it consistently outperforms baseline agents and surpasses other inference baselines by 2.6{\%} to 13.3{\%} acrosstwo model sizes, demonstrating strong generalization and practical plug-and-play capabilities in real-world scenarios."
}