@inproceedings{basu-etal-2024-distilling,
    title = "Distilling Knowledge from Text-to-Image Generative Models Improves Visio-Linguistic Reasoning in {CLIP}",
    author = "Basu, Samyadeep  and
      Hu, Shell Xu  and
      Sanjabi, Maziar  and
      Massiceti, Daniela  and
      Feizi, Soheil",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Add-Cong-Liu-Florida-Atlantic-University-author-id/2024.emnlp-main.351/",
    doi = "10.18653/v1/2024.emnlp-main.351",
    pages = "6105--6113",
    abstract = "Image-text contrastive models like CLIP have wide applications in zero-shot classification, image-text retrieval, and transfer learning. However, they often struggle on compositional visio-linguistic tasks (e.g., attribute-binding or object-relationships) where their performance is no better than random chance. To address this, we introduce SDS-CLIP, a lightweight and sample-efficient distillation method to enhance CLIP`s compositional visio-linguistic reasoning. Our approach fine-tunes CLIP using a distillation objective borrowed from large text-to-image generative models like Stable-Diffusion, which are known for their strong visio-linguistic reasoning abilities. On the challenging Winoground benchmark, SDS-CLIP improves the visio-linguistic performance of various CLIP models by up to 7{\%}, while on the ARO dataset, it boosts performance by up to 3{\%}. This work underscores the potential of well-designed distillation objectives from generative models to enhance contrastive image-text models with improved visio-linguistic reasoning capabilities."
}