Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, Pascale Fung

Abstract
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with 7× fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.
Anthology ID:
2022.findings-acl.187
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2383–2395
Language:
URL:
https://aclanthology.org/2022.findings-acl.187
DOI:
10.18653/v1/2022.findings-acl.187
Bibkey:
Cite (ACL):
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2383–2395, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (Dai et al., Findings 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/remove-xml-comments/2022.findings-acl.187.pdf
Data
COCO, GLUE, OK-VQA, nocaps
PDF Search