Retrieval-enriched zero-shot image classification in low-resource domains

Nicola Dall’Asen, Yiming Wang, Enrico Fini, Elisa Ricci

Abstract
Low-resource domains, characterized by scarce data and annotations, present significant challenges for language and visual understanding tasks, with the latter much under-explored in the literature. Recent advancements in Vision-Language Models (VLM) have shown promising results in high-resource domains but fall short in low-resource concepts that are under-represented (e.g. only a handful of images per category) in the pre-training set. We tackle the challenging task of zero-shot low-resource image classification from a novel perspective. By leveraging a retrieval-based strategy, we achieve this in a training-free fashion. Specifically, our method, named CoRE (Combination of Retrieval Enrichment), enriches the representation of both query images and class prototypes by retrieving relevant textual information from large web-crawled databases. This retrieval-based enrichment significantly boosts classification performance by incorporating the broader contextual information relevant to the specific class. We validate our method on a newly established benchmark covering diverse low-resource domains, including medical imaging, rare plants, and circuits. Our experiments demonstrate that CoRE outperforms existing state-of-the-art methods that rely on synthetic data generation and model fine-tuning.
Anthology ID:
2024.emnlp-main.1186
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
21287–21302
Language:
URL:
https://aclanthology.org/2024.emnlp-main.1186
DOI:
10.18653/v1/2024.emnlp-main.1186
Bibkey:
Cite (ACL):
Nicola Dall’Asen, Yiming Wang, Enrico Fini, and Elisa Ricci. 2024. Retrieval-enriched zero-shot image classification in low-resource domains. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21287–21302, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Retrieval-enriched zero-shot image classification in low-resource domains (Dall’Asen et al., EMNLP 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/dois-2013-emnlp/2024.emnlp-main.1186.pdf
PDF Search