LightVLP: A Lightweight Vision-Language Pre-training via Gated Interactive Masked AutoEncoders
Xingwu Sun, Zhen Yang, Ruobing Xie, Fengzong Lian, Zhanhui Kang, Chengzhong Xu
Abstract
This paper studies vision-language (V&L) pre-training for deep cross-modal representations. Recently, pre-trained V&L models have shown great success in V&L tasks. However, most existing models apply multi-modal encoders to encode the image and text, at the cost of high training complexity because of the input sequence length. In addition, they suffer from noisy training corpora caused by V&L mismatching. In this work, we propose a lightweight vision-language pre-training (LightVLP) for efficient and effective V&L pre-training. First, we design a new V&L framework with two autoencoders. Each autoencoder involves an encoder, which only takes in unmasked tokens (removes masked ones), as well as a lightweight decoder that reconstructs the masked tokens. Besides, we mask and remove large portions of input tokens to accelerate the training. Moreover, we propose a gated interaction mechanism to cope with noise in aligned image-text pairs. As for a matched image-text pair, the model tends to apply cross-modal representations for reconstructions. By contrast, for an unmatched pair, the model conducts reconstructions mainly using uni-modal representations. Benefiting from the above-mentioned designs, our base model shows competitive results compared to ALBEF while saving 44% FLOPs. Further, we compare our large model with ALBEF under the setting of similar FLOPs on six datasets and show the superiority of LightVLP. In particular, our model achieves 2.2% R@1 gains on COCO Text Retrieval and 1.1% on refCOCO+.- Anthology ID:
- 2024.lrec-main.918
- Volume:
- Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
- Venues:
- LREC | COLING
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 10499–10510
- Language:
- URL:
- https://aclanthology.org/2024.lrec-main.918
- DOI:
- Cite (ACL):
- Xingwu Sun, Zhen Yang, Ruobing Xie, Fengzong Lian, Zhanhui Kang, and Chengzhong Xu. 2024. LightVLP: A Lightweight Vision-Language Pre-training via Gated Interactive Masked AutoEncoders. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10499–10510, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- LightVLP: A Lightweight Vision-Language Pre-training via Gated Interactive Masked AutoEncoders (Sun et al., LREC-COLING 2024)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2024.lrec-main.918.pdf