@inproceedings{zhu-etal-2022-knowledge-transfer,
    title = "Knowledge Transfer with Visual Prompt in multi-modal Dialogue Understanding and Generation",
    author = "Zhu, Minjun  and
      Weng, Yixuan  and
      Li, Bin  and
      He, Shizhu  and
      Liu, Kang  and
      Zhao, Jun",
    editor = "Dernoncourt, Franck  and
      Nguyen, Thien Huu  and
      Lai, Viet Dac  and
      Veyseh, Amir Pouran Ben  and
      Bui, Trung H.  and
      Yoon, David Seunghyun",
    booktitle = "Proceedings of the First Workshop On Transcript Understanding",
    month = oct,
    year = "2022",
    address = "Gyeongju, South Korea",
    publisher = "International Conference on Computational Linguistics",
    url = "https://aclanthology.org/2022.tu-1.2",
    pages = "8--19",
    abstract = "Visual Dialogue (VD) task has recently received increasing attention in AI research. Visual Dialog aims to generate multi-round, interactive responses based on the dialog history and image content. Existing textual dialogue models cannot fully understand visual information, resulting in a lack of scene features when communicating with humans continuously. Therefore, how to efficiently fuse multimodal data features remains to be a challenge. In this work, we propose a knowledge transfer method with visual prompt (VPTG) fusing multi-modal data, which is a flexible module that can utilize the text-only seq2seq model to handle visual dialogue tasks. The VPTG conducts text-image co-learning and multi-modal information fusion with visual prompts and visual knowledge distillation. Specifically, we construct visual prompts from visual representations and then induce sequence-to-sequence(seq2seq) models to fuse visual information and textual contexts by visual-text patterns. And we also realize visual knowledge transfer through distillation between two different models{'} text representations, so that the seq2seq model can actively learn visual semantic representations. Extensive experiments on the multi-modal dialogue understanding and generation (MDUG) datasets show the proposed VPTG outperforms other single-modal methods, which demonstrate the effectiveness of visual prompt and visual knowledge transfer.",
}