@inproceedings{jenkins-2026-thinking,
    title = "Thinking in Pictures: A Diagnostic Study of Visual vs. Textual Chain-of-Thought Reasoning in Vision-Language Models",
    author = "Jenkins, Ben",
    editor = "Yan, Qianqi  and
      Montariol, Syrielle  and
      Fan, Yue  and
      Gu, Jing  and
      Pan, Jiayi  and
      Li, Manling  and
      Kordjamshidi, Parisa  and
      Suhr, Alane  and
      Wang, Xin Eric",
    booktitle = "Proceedings of the 4th Workshop on Advances in Language and Vision Research ({ALVR})",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.alvr-main.1/",
    pages = "1--12",
    ISBN = "979-8-89176-398-2",
    abstract = "Chain-of-thought (CoT) reasoning has become a standard technique for eliciting complex reasoning in large language models, and recent work has extended it to vision-language models (VLMs). However, virtually all multimodal CoT methods generate intermediate reasoning steps in natural language, even for inherently visual problems such as spatial reasoning, geometric manipulation, and object tracking. We ask a fundamental question: when should a VLM reason in words, and when should it reason in pictures? We present VisCoT-Diag, a diagnostic benchmark of 1,200 instances across five visual reasoning categories, and compare four CoT paradigms across four VLMs. Our results reveal a striking modality gap: textual CoT degrades performance by up to 17.5{\%} on spatial transformation and 13.2{\%} on multi-object tracking, while visual CoT yields gains of up to 23.1{\%}. We identify three failure modes (spatial state collapse, transformation hallucination, tracking loss) and show that adaptive modality routing achieves 73.1{\%} accuracy versus 68.9{\%} for V-CoT-everywhere. We recommend practitioners use visual CoT for spatial tasks and textual CoT for compositional counting."
}