@inproceedings{kong-etal-2026-mobileworld,
    title = "{M}obile{W}orld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and {MCP}-Augmented Environments",
    author = "Kong, Quyu  and
      Zhang, Xu  and
      Yang, Zhenyu  and
      Gao, Nolan  and
      Liu, Chen  and
      Tong, Panrong  and
      Cai, Chenglin  and
      Zhou, Hanzhang  and
      Zhang, Jianan  and
      Chen, Liangyu  and
      Liu, Zhidan  and
      Hoi, Steven  and
      Wang, Yue",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.278/",
    pages = "6142--6167",
    ISBN = "979-8-89176-390-6",
    abstract = "While AndroidWorld has become the dominant mobile-use benchmark due to its reproducible environment and deterministic evaluation, recent agents achieving over 90{\%} success rates indicate saturation and motivate the need for greater challenge. In addition, its environment lacks key application categories, such as e-commerce and enterprise communication, and does not reflect realistic mobile-use scenarios characterized by vague user instructions and hybrid tool usage. We introduce MobileWorld, a substantially more challenging benchmark with 201 tasks across 20 applications that reflects real-world usage through long-horizon, cross-application workflows requiring nearly twice as many steps (27.8 vs. 14.3) and featuring significantly more multi-app tasks (62.2{\%} vs. 9.5{\%}) than AndroidWorld. MobileWorld balances production-grade utility and reproducible evaluation using open-source alternatives to industry standards (e.g., Mattermost for Slack), enabling full observability through source code modification and direct database access. Beyond standard GUI manipulation, MobileWorld introduces novel task categories including agent-user interaction and Model Context Protocol (MCP)-augmented tasks for evaluating agents in user-aware, hybrid-tool scenarios. We develop a planner-executor framework with extended action spaces supporting user interactions and MCP calls. Results show a sharp performance drop from AndroidWorld, with the best agentic framework and end-to-end model achieving 51.7{\%} and 20.9{\%} success rates, respectively, highlighting substantial room for future research."
}