@inproceedings{qiu-etal-2024-snapntell,
    title = "{S}nap{NT}ell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal {LLM}",
    author = "Qiu, Jielin  and
      Madotto, Andrea  and
      Lin, Zhaojiang  and
      Crook, Paul A.  and
      Xu, Yifan Ethan  and
      Damavandi, Babak  and
      Dong, Xin Luna  and
      Faloutsos, Christos  and
      Li, Lei  and
      Moon, Seungwhan",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.findings-emnlp.14/",
    doi = "10.18653/v1/2024.findings-emnlp.14",
    pages = "247--266",
    abstract = "Vision-extended LLMs have made significant strides in Visual Question Answering (VQA). Despite these advancements, VLLMs still encounter substantial difficulties in handling queries involving long-tail entities, with a tendency to produce erroneous or hallucinated responses. In this work, we introduce a novel evaluative benchmark named \textbf{SnapNTell}, specifically tailored for entity-centric VQA. This task aims to test the models' capabilities in identifying entities and providing detailed, entity-specific knowledge. We have developed the \textbf{SnapNTell Dataset}, distinct from traditional VQA datasets: (1) It encompasses a wide range of categorized entities, each represented by images and explicitly named in the answers; (2) It features QA pairs that require extensive knowledge for accurate responses. The dataset is organized into 22 major categories, containing 7,568 unique entities in total. For each entity, we curated 10 illustrative images and crafted 10 knowledge-intensive QA pairs. To address this novel task, we devised a scalable, efficient, and transparent retrieval-augmented multimodal LLM. Our approach markedly outperforms existing methods on the SnapNTell dataset, achieving a 66.5{\%} improvement in the BELURT score."
}