@inproceedings{hsiao-etal-2025-screenqa,
    title = "{S}creen{QA}: Large-Scale Question-Answer Pairs Over Mobile App Screenshots",
    author = "Hsiao, Yu-Chung  and
      Zubach, Fedir  and
      Baechler, Gilles  and
      Sunkara, Srinivas  and
      Carbune, Victor  and
      Lin, Jason  and
      Wang, Maria  and
      Zhu, Yun  and
      Chen, Jindong",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.477/",
    pages = "9427--9452",
    ISBN = "979-8-89176-189-6",
    abstract = "We introduce ScreenQA, a novel benchmarking dataset designed to advance screen content understanding through question answering. The existing screen datasets are focused either on low-level structural and component understanding, or on a much higher-level composite task such as navigation and task completion for autonomous agents. ScreenQA attempts to bridge this gap. By annotating 86k question-answer pairs over the RICO dataset, we aim to benchmark the screen reading comprehension capacity, thereby laying the foundation for vision-based automation over screenshots. Our annotations encompass full answers, short answer phrases, and corresponding UI contents with bounding boxes, enabling four subtasks to address various application scenarios. We evaluate the dataset{'}s efficacy using both open-weight and proprietary models in zero-shot, fine-tuned, and transfer learning settings. We further demonstrate positive transfer to web applications, highlighting its potential beyond mobile applications."
}