@inproceedings{niu-etal-2026-rotbench,
    title = "{R}ot{B}ench: Evaluating Multi-modal Large Language Models on Identifying Image Rotation",
    author = "Niu, Tianyi  and
      Cho, Jaemin  and
      Stengel-Eskin, Elias  and
      Bansal, Mohit",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.259/",
    pages = "5546--5569",
    ISBN = "979-8-89176-380-7",
    abstract = "We investigate to what extent Multimodal Large Language Models (MLLMs) can accurately identify the orientation of input images rotated 0{\textdegree}, 90{\textdegree}, 180{\textdegree}, and 270{\textdegree}. This task demands robust visual reasoning capabilities to detect rotational cues and contextualize spatial relationships within images, regardless of their orientation. To evaluate MLLMs on these abilities, we introduce RotBench, a 350-image manually-filtered benchmark comprising lifestyle, portrait, and landscape images. Despite the relatively simple nature of this task, we show that several state-of-the-art open and proprietary MLLMs, including GPT-5, o3, and Gemini-2.5-Pro, do not reliably identify rotation in input images. Providing models with auxiliary information{---}including captions, depth maps, and more{---}or using chain-of-thought prompting offers only small and inconsistent improvements. Our results indicate that most models are able to reliably identify right-side-up (0{\textdegree}) images, while certain models are able to identify upside-down (180{\textdegree}) images. None can reliably distinguish between 90{\textdegree} and 270{\textdegree} rotated images. Simultaneously showing the image rotated in different orientations leads to moderate performance gains for reasoning models, while a modified setup using voting improves the performance of weaker models. We further show that fine-tuning does not improve models' ability to distinguish 90{\textdegree} and 270{\textdegree} rotations, despite substantially improving the identification of 180{\textdegree} images. Together, these results reveal a significant gap between MLLMs' spatial reasoning capabilities and human perception in identifying rotation."
}