@inproceedings{chen-etal-2023-difference,
    title = "On the Difference of {BERT}-style and {CLIP}-style Text Encoders",
    author = "Chen, Zhihong  and
      Chen, Guiming  and
      Diao, Shizhe  and
      Wan, Xiang  and
      Wang, Benyou",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2023.findings-acl.866/",
    doi = "10.18653/v1/2023.findings-acl.866",
    pages = "13710--13721",
    abstract = "Masked language modeling (MLM) has been one of the most popular pretraining recipes in natural language processing, \textit{e.g.}, BERT, one of the representative models. Recently, contrastive language-image pretraining (CLIP) has also attracted attention, especially its vision models that achieve excellent performance on a broad range of vision tasks. However, few studies are dedicated to studying the text encoders learned by CLIP. In this paper, we analyze the difference between \textit{BERT-style} and \textit{CLIP-style} text encoders from three experiments: (i) general text understanding, (ii) vision-centric text understanding, and (iii) text-to-image generation. Experimental analyses show that although CLIP-style text encoders underperform BERT-style ones for general text understanding tasks, they are equipped with a unique ability, \textit{i.e.}, \textit{synesthesia}, for the cross-modal association, which is more similar to the senses of humans."
}