Abstract
Video-grounded dialogue systems aim to integrate video understanding and dialogue understanding to generate responses that are relevant to both the dialogue and video context. Most existing approaches employ deep learning models and have achieved remarkable performance, given the relatively small datasets available. However, the results are partially accomplished by exploiting biases in the datasets rather than developing multimodal reasoning, resulting in limited generalization. In this paper, we propose a novel approach of Compositional Counterfactual Contrastive Learning (C3) to develop contrastive training between factual and counterfactual samples in video-grounded dialogues. Specifically, we design factual/counterfactual samples based on the temporal steps in videos and tokens in dialogues and propose contrastive loss functions that exploit object-level or action-level variance. Different from prior approaches, we focus on contrastive hidden state representations among compositional output tokens to optimize the representation space in a generation setting. We achieved promising performance gains on the Audio-Visual Scene-Aware Dialogues (AVSD) benchmark and showed the benefits of our approach in grounding video and dialogue context.- Anthology ID:
- 2023.sigdial-1.51
- Volume:
- Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue
- Month:
- September
- Year:
- 2023
- Address:
- Prague, Czechia
- Editors:
- Svetlana Stoyanchev, Shafiq Joty, David Schlangen, Ondrej Dusek, Casey Kennington, Malihe Alikhani
- Venue:
- SIGDIAL
- SIG:
- SIGDIAL
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 548–561
- Language:
- URL:
- https://aclanthology.org/2023.sigdial-1.51
- DOI:
- 10.18653/v1/2023.sigdial-1.51
- Cite (ACL):
- Hung Le, Nancy Chen, and Steven C.H. Hoi. 2023. C3: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 548–561, Prague, Czechia. Association for Computational Linguistics.
- Cite (Informal):
- C3: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues (Le et al., SIGDIAL 2023)
- PDF:
- https://preview.aclanthology.org/landing_page/2023.sigdial-1.51.pdf