Find-the-Common: A Benchmark for Explaining Visual Patterns from Images

Yuting Shi, Naoya Inoue, Houjing Wei, Yufeng Zhao, Tao Jin

Abstract
Recent advances in Instruction-fine-tuned Vision and Language Models (IVLMs), such as GPT-4V and InstructBLIP, have prompted some studies have started an in-depth analysis of the reasoning capabilities of IVLMs. However, Inductive Visual Reasoning, a vital skill for text-image understanding, remains underexplored due to the absence of benchmarks. In this paper, we introduce Find-the-Common (FTC): a new vision and language task for Inductive Visual Reasoning. In this task, models are required to identify an answer that explains the common attributes across visual scenes. We create a new dataset for the FTC and assess the performance of several contemporary approaches including Image-Based Reasoning, Text-Based Reasoning, and Image-Text-Based Reasoning with various models. Extensive experiments show that even state-of-the-art models like GPT-4V can only archive with 48% accuracy on the FTC, for which, the FTC is a new challenge for the visual reasoning research community. Our dataset has been released and is available online: https://github.com/SSSSSeki/Find-the-common.
Anthology ID:
2024.lrec-main.642
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
7307–7313
Language:
URL:
https://aclanthology.org/2024.lrec-main.642
DOI:
Bibkey:
Cite (ACL):
Yuting Shi, Naoya Inoue, Houjing Wei, Yufeng Zhao, and Tao Jin. 2024. Find-the-Common: A Benchmark for Explaining Visual Patterns from Images. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7307–7313, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Find-the-Common: A Benchmark for Explaining Visual Patterns from Images (Shi et al., LREC-COLING 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.lrec-main.642.pdf
PDF Search