@inproceedings{sravanthi-etal-2025-perception,
    title = "From Perception to Reasoning: Enhancing Vision-Language Models for Mobile {UI} Understanding",
    author = "Sravanthi, Settaluri Lakshmi  and
      Mishra, Ankit  and
      Mondal, Debjyoti  and
      Panda, Subhadarshi  and
      Singh, Rituraj  and
      Bhattacharyya, Pushpak",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.1295/",
    pages = "25250--25269",
    ISBN = "979-8-89176-256-5",
    abstract = "Accurately grounding visual and textual elements within mobile user interfaces (UIs) remains a significant challenge for Vision-Language Models (VLMs). Visual grounding, a critical task in this domain, involves identifying the most relevant UI element or region based on a natural language query{---}a process that requires both precise perception and context-aware reasoning. In this work, we present - **MoUI**, a light-weight mobile UI understanding model trained on **MoIT**, an instruction-tuning dataset specifically tailored for mobile screen understanding and grounding, designed to bridge the gap between user intent and visual semantics. Complementing this dataset, we also present a human-annotated reasoning benchmark **MoIQ** that rigorously evaluates complex inference capabilities over mobile UIs. To harness these resources effectively, we propose a two-stage training approach that separately addresses perception and reasoning tasks, leading to stronger perception capabilities and improvement in reasoning abilities. Through extensive experiments, we demonstrate that our MoUI models achieve significant gains in accuracy across all perception tasks and {\_}state-of-the-art{\_} results on public reasoning benchmark **ComplexQA (78{\%}) and our MoIQ (49{\%})**. We will be open-sourcing our dataset, code, and models to foster further research and innovation in the field."
}