Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models

Canshi Wei


Abstract
Fine-grained image classification, especially in zero-/few-shot scenarios, poses a considerable challenge for vision-language models (VLMs) like CLIP, which often struggle to differentiate between semantically similar classes due to insufficient supervision for fine-grained tasks. On the other hand, Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities in tasks like Visual Question Answering (VQA) but remain underexplored in the context of fine-grained image classification. This paper presents CascadeVLM, a novel framework that harnesses the complementary strengths of both CLIP-like and LVLMs VLMs to tackle these challenges. Using granular knowledge effectively in LVLMs and integrating a cascading approach, CascadeVLM dynamically allocates samples using an entropy threshold, balancing computational efficiency with classification accuracy. Experiments on multiple fine-grained datasets, particularly the Stanford Cars dataset, show that CascadeVLM outperforms existing models, achieving 92% accuracy. Our results highlight the potential of combining VLM and LVLM for robust, efficient and interpretable fine-grained image classification, offering new insights into their synergy.
Anthology ID:
2024.findings-emnlp.102
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2024
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1857–1871
Language:
URL:
https://preview.aclanthology.org/icon-24-ingestion/2024.findings-emnlp.102/
DOI:
10.18653/v1/2024.findings-emnlp.102
Bibkey:
Cite (ACL):
Canshi Wei. 2024. Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1857–1871, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models (Wei, Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/icon-24-ingestion/2024.findings-emnlp.102.pdf