@inproceedings{zhang-etal-2025-ufo,
    title = "{UFO}: A {UI}-Focused Agent for Windows {OS} Interaction",
    author = "Zhang, Chaoyun  and
      Li, Liqun  and
      He, Shilin  and
      Zhang, Xu  and
      Qiao, Bo  and
      Qin, Si  and
      Ma, Minghua  and
      Kang, Yu  and
      Lin, Qingwei  and
      Rajmohan, Saravan  and
      Zhang, Dongmei  and
      Zhang, Qi",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.naacl-long.26/",
    pages = "597--622",
    ISBN = "979-8-89176-189-6",
    abstract = "We introduce UFO, a UI-Fcused agent designed to fulfill user requests tailored to Windows OS applications by observing and analyzing the GUI and control information of these applications. UFO utilizes a hierarchical dual-agent framework that decomposes user requests using a divide-and-conquer approach, enabling seamless navigation and addressing sub-tasks across multiple applications. It also incorporates a control interaction module tailored for Windows OS, which detects control elements effectively and allows for fully automated execution. As a result, UFO simplifies complex and time-consuming processes into tasks that can be completed with natural language commands.We conducted testing of UFO across 9 popular Windows applications, encompassing a variety of scenarios. The results derived from both quantitative metrics and real-case studies, underscore the superior effectiveness of UFOin fulfilling user requests. To the best of our knowledge, UFO stands as the first UI agent specifically tailored for task completion within the Windows OS."
}