Abstract
Visual Dialog is a multi-modal task that requires a model to participate in a multi-turn human dialog grounded on an image, and generate correct, human-like responses. In this paper, we propose a novel Adversarial Multi-modal Feature Encoding (AMFE) framework for effective and robust auxiliary training of visual dialog systems. AMFE can force the language-encoding part of a model to generate hidden states in a distribution closely related to the distribution of real-world images, resulting in language features containing general knowledge from both modalities by nature, which can help generate both more correct and more general responses with reasonably low time cost. Experimental results show that AMFE can steadily bring performance gains to different models on different scales of data. Our method outperforms both the supervised learning baselines and other fine-tuning methods, achieving state-of-the-art results on most metrics of VisDial v0.5/v0.9 generative tasks.- Anthology ID:
- N19-1266
- Volume:
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
- Month:
- June
- Year:
- 2019
- Address:
- Minneapolis, Minnesota
- Editors:
- Jill Burstein, Christy Doran, Thamar Solorio
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2588–2598
- Language:
- URL:
- https://aclanthology.org/N19-1266
- DOI:
- 10.18653/v1/N19-1266
- Cite (ACL):
- Yiqun Yao, Jiaming Xu, and Bo Xu. 2019. The World in My Mind: Visual Dialog with Adversarial Multi-modal Feature Encoding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2588–2598, Minneapolis, Minnesota. Association for Computational Linguistics.
- Cite (Informal):
- The World in My Mind: Visual Dialog with Adversarial Multi-modal Feature Encoding (Yao et al., NAACL 2019)
- PDF:
- https://preview.aclanthology.org/ingest-2024-clasp/N19-1266.pdf
- Data
- MS COCO, VisDial