@inproceedings{sun-etal-2023-fusion,
    title = "Fusion or Defusion? Flexible Vision-and-Language Pre-Training",
    author = "Sun, Rongyi  and
      Li, Ziran  and
      Ding, Yifeng  and
      Wang, Qifan  and
      Wang, Jingang  and
      Zheng, Haitao  and
      Wu, Wei  and
      Xian, Yunsen",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2023.findings-acl.316/",
    doi = "10.18653/v1/2023.findings-acl.316",
    pages = "5105--5119",
    abstract = "Existing approaches in the vision-and-language pre-training (VLP) paradigm mainly deploy either fusion-based encoders or dual-encoders, failing to achieve both effectiveness and efficiency in downstream multimodal tasks. In this paper, we build a flexible VLP model by incorporating cross-modal fusions into a dual-encoder architecture, where the introduced fusion modules can be easily decoupled from the dual encoder so as to switch the model to a fusion-free one. To better absorb cross-modal features from the fusion modules, we design a cross-modal knowledge transfer strategy along with other comprehensive pre-training tasks to guide the training process, which can further strengthen both the fusion-based and fusion-free representation learning. Extensive experiments conducted on various downstream vision-language tasks show that our proposed model is well-equipped with effectiveness as well as efficiency, demonstrating a superior performance compared with other strong VLP models."
}