@inproceedings{zhou-etal-2025-glimpse,
    title = "{GLIMPSE}: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?",
    author = "Zhou, Yiyang  and
      Li, Linjie  and
      Qiu, Shi  and
      Yang, Zhengyuan  and
      Zhao, Yuyang  and
      Han, Siwei  and
      He, Yangfan  and
      Li, Kangqi  and
      Ji, Haonian  and
      Zhao, Zihao  and
      Tong, Haibo  and
      Wang, Lijuan  and
      Yao, Huaxiu",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1415/",
    pages = "27830--27844",
    ISBN = "979-8-89176-332-6",
    abstract = "Existing video benchmarks often resemble image-based benchmarks, with question types like ``What actions does the person perform throughout the video?'' or ``What color is the woman{'}s dress in the video?'' For these, models can often answer by scanning just a few key frames, without deep temporal reasoning. This limits our ability to assess whether large vision-language models (LVLMs) can truly think with videos rather than perform superficial frame-level analysis. To address this, we introduce , a benchmark specifically designed to evaluate whether LVLMs can genuinely think with videos. Unlike prior benchmarks, emphasizes comprehensive video understanding beyond static image cues. It consists of 3,269 videos and over 4,342 highly visual-centric questions across 11 categories, including Trajectory Analysis, Temporal Reasoning, and Forensics Detection. All questions are carefully crafted by human annotators and require watching the entire video and reasoning over full video context{---}this is what we mean by thinking with video. These questions cannot be answered by scanning selected frames or relying on text alone. In human evaluations, achieves 94.82{\%} accuracy, but current LVLMs face significant challenges. Even the best-performing model, GPT-o3, reaches only 66.43{\%}, highlighting that LVLMs still struggle to move beyond surface-level reasoning to truly think with videos. We publicly release our benchmark and code at https://github.com/aiming-lab/GLIMPSE."
}