@inproceedings{thawakar-etal-2025-llamav,
    title = "{L}lama{V}-o1: Rethinking Step-by-step Visual Reasoning in {LLM}s",
    author = "Thawakar, Omkar  and
      Dissanayake, Dinura  and
      More, Ketan Pravin  and
      Thawkar, Ritesh  and
      Heakl, Ahmed  and
      Ahsan, Noor  and
      Li, Yuhao  and
      Zumri, Ilmuz Zaman Mohammed  and
      Lahoud, Jean  and
      Anwer, Rao Muhammad  and
      Cholakkal, Hisham  and
      Laptev, Ivan  and
      Shah, Mubarak  and
      Khan, Fahad Shahbaz  and
      Khan, Salman",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.1247/",
    pages = "24290--24315",
    ISBN = "979-8-89176-256-5",
    abstract = "Step-by-step reasoning is crucial for solving complex visual tasks, yet existing approaches lack a comprehensive framework for evaluating this capability and do not emphasize step-wise problem-solving. To this end, we propose a comprehensive framework for advancing multi-step visual reasoning in large multimodal models (LMMs) through three key contributions. First, we introduce a Visual Reasoning Chain Benchmark, the most comprehensive benchmark for multi-step visual reasoning, covering eight diverse categories and over 4k reasoning steps. This enables rigorous evaluation of LMMs' ability to reason accurately and interpretably across multiple steps. Second, we propose a fine-grained reasoning metric that evaluates correctness and logical coherence at each step, providing deeper insights beyond traditional accuracy metrics. Third, we introduce LlamaV-o1, a state-of-the-art multimodal reasoning model trained using a multi-step curriculum learning approach. LlamaV-o1 is optimized for structured, step-by-step reasoning and significantly outperforms existing open-source models. It surpasses Llava-CoT with a 3.8{\%} absolute gain across six benchmarks, achieving an average score of 67.3 while being 5x faster during inference scaling. Our benchmark, model, and code is available at https://github.com/mbzuai-oryx/LlamaV-o1."
}