@inproceedings{zhang-etal-2025-pre,
    title = "Pre-training {CLIP} against Data Poisoning with Optimal Transport-based Matching and Alignment",
    author = "Zhang, Tong  and
      Gao, Kuofeng  and
      Bai, Jiawang  and
      Zhang, Leo Yu  and
      Yin, Xin  and
      Wang, Zonghui  and
      Ji, Shouling  and
      Chen, Wenzhi",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.497/",
    pages = "9836--9849",
    ISBN = "979-8-89176-332-6",
    abstract = "Recent studies have shown that Contrastive Language-Image Pre-training (CLIP) models are threatened by targeted data poisoning and backdoor attacks due to massive training image-caption pairs crawled from the Internet. Previous defense methods correct poisoned image-caption pairs by matching a new caption for each image. However, the matching process solely relies on the global representations of images and captions, overlooking fine-grained features of visual and textual features. It may introduce incorrect image-caption pairs and detriment the CLIP pre-training. To address their limitations, we propose an Optimal Transport-based framework to reconstruct the image-caption pairs, named OTCCLIP. We involve a new optimal transport-based distance measure between fine-grained visual and textual feature sets and re-assign new captions based on the proposed optimal transport distance. Additionally, to further reduce the negative impact of mismatched pairs, we encourage the inter- and intra-modality fine-grained alignment by employing optimal transport-based objective functions. Our experiments demonstrate that OTCCLIP can successfully decrease the attack success rates of poisoning attacks to 0{\%} in most cases. Also, compared to previous methods, OTCCLIPsignificantly improves CLIP{'}s zero-shot and linear probing performance trained on poisoned datasets."
}