@inproceedings{lu-etal-2026-experience,
    title = "Experience-driven Multi-turn Reinforcement Learning for {GUI} Agents",
    author = "Lu, Zhengxi  and
      Ye, Jiabo  and
      Tang, Fei  and
      Shen, Yongliang  and
      Xu, Haiyang  and
      Zheng, Ziwei  and
      Lu, Weiming  and
      Yan, Ming  and
      Huang, Fei  and
      Xiao, Jun  and
      Zhuang, Yueting",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.428/",
    pages = "9474--9496",
    ISBN = "979-8-89176-390-6",
    abstract = "GUI agents have demonstrated remarkable progress in automating complex user interface interactions. However, training such agents for long-horizon tasks remains challenging. Single-turn reinforcement learning conditions on expert histories during training but self-generated histories during deployment, causing distribution mismatch. Online multi-turn methods eliminate this gap via environment interaction but suffer from sparse rewards and prohibitive costs. We propose $\underline{\textbf{E}}$xperience-driven $\underline{\textbf{M}}$ulti-turn $\underline{\textbf{P}}$olicy $\underline{\textbf{O}}$ptimization ($\textbf{EMPO}$), which leverages expert trajectories as environment experiences for on-policy multi-turn training. The agent constructs self-generated history throughout rollouts; when actions match expert experiences, the trajectory provides valid state transitions, and a Patch Module recovers mismatched steps to maintain on-policy rollouts. EMPO further incorporates discounted future rewards and dual-level advantage estimation to capture long-horizon dependencies. We also propose $\textbf{AndroidControl-Real}$, an evaluation metric strongly correlated with real-world performance (R$^2$=0.934). With only 1K public trajectories as RL experiences, our method achieves substantial gains over the base model (e.g., +12.0{\%} on AndroidWorld and +23.8{\%} on AITW) and achieves competitive performance against strong baselines such as UI-TARS-7B and GPT-4o, demonstrating better generalization than prior single-turn RL approaches. Code available: \url{https://anonymous.4open.science/r/UI-S1-0DAF}."
}