@inproceedings{jia-etal-2023-kafa,
    title = "{KAFA}: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models",
    author = "Jia, Zhiwei  and
      Narayana, Pradyumna  and
      Akula, Arjun  and
      Pruthi, Garima  and
      Su, Hao  and
      Basu, Sugato  and
      Jampani, Varun",
    editor = "Sitaram, Sunayana  and
      Beigman Klebanov, Beata  and
      Williams, Jason D",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest_wac_2008/2023.acl-industry.74/",
    doi = "10.18653/v1/2023.acl-industry.74",
    pages = "772--785",
    abstract = "Image ad understanding is a crucial task with wide real-world applications. Although highly challenging with the involvement of diverse atypical scenes, real-world entities, and reasoning over scene-texts, how to interpret image ads is relatively under-explored, especially in the era of foundational vision-language models (VLMs) featuring impressive generalizability and adaptability. In this paper, we perform the first empirical study of image ad understanding through the lens of pre-trained VLMs. We benchmark and reveal practical challenges in adapting these VLMs to image ad understanding. We propose a simple feature adaptation strategy to effectively fuse multimodal information for image ads and further empower it with knowledge of real-world entities. We hope our study draws more attention to image ad understanding which is broadly relevant to the advertising industry."
}