@inproceedings{cheng-etal-2025-os,
    title = "{OS}-Kairos: Adaptive Interaction for {MLLM}-Powered {GUI} Agents",
    author = "Cheng, Pengzhou  and
      Wu, Zheng  and
      Wu, Zongru  and
      Ju, Tianjie  and
      Zhang, Aston  and
      Zhang, Zhuosheng  and
      Liu, Gongshen",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.findings-acl.348/",
    doi = "10.18653/v1/2025.findings-acl.348",
    pages = "6701--6725",
    ISBN = "979-8-89176-256-5",
    abstract = "Autonomous graphical user interface (GUI) agents powered by multimodal large language models have shown great promise. However, a critical yet underexplored issue persists: \textbf{over-execution}, where the agent executes tasks in a fully autonomous way, without adequate assessment of its action confidence to compromise an adaptive human-agent collaboration. This poses substantial risks in complex scenarios, such as those involving ambiguous user instructions, unexpected interruptions, and environmental hijacks. To address the issue, we introduce \textit{OS-Kairos}, an adaptive GUI agent capable of predicting confidence levels at each interaction step and efficiently deciding whether to act autonomously or seek human intervention. \textit{OS-Kairos} is developed through two key mechanisms: (i) collaborative probing that annotates confidence scores at each interaction step; (ii) confidence-driven interaction that leverages these confidence scores to elicit the ability of adaptive interaction. Experimental results show that \textit{OS-Kairos} substantially outperforms existing models on our curated dataset featuring complex scenarios, as well as on established benchmarks such as AITZ and Meta-GUI, with 24.59{\%}{\textasciitilde}87.29{\%} improvements in task success rate. \textit{OS-Kairos} facilitates an adaptive human-agent collaboration, prioritizing effectiveness, generality, scalability, and efficiency for real-world GUI interaction. The dataset and codes are available at Anonymous."
}