Check It Again:Progressive Visual Question Answering via Visual Entailment

Qingyi Si, Zheng Lin, Ming yu Zheng, Peng Fu, Weiping Wang


Abstract
While sophisticated neural-based models have achieved remarkable success in Visual Question Answering (VQA), these models tend to answer questions only according to superficial correlations between question and answer. Several recent approaches have been developed to address this language priors problem. However, most of them predict the correct answer according to one best output without checking the authenticity of answers. Besides, they only explore the interaction between image and question, ignoring the semantics of candidate answers. In this paper, we propose a select-and-rerank (SAR) progressive framework based on Visual Entailment. Specifically, we first select the candidate answers relevant to the question or the image, then we rerank the candidate answers by a visual entailment task, which verifies whether the image semantically entails the synthetic statement of the question and each candidate answer. Experimental results show the effectiveness of our proposed framework, which establishes a new state-of-the-art accuracy on VQA-CP v2 with a 7.55% improvement.
Anthology ID:
2021.acl-long.317
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Editors:
Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4101–4110
Language:
URL:
https://aclanthology.org/2021.acl-long.317
DOI:
10.18653/v1/2021.acl-long.317
Bibkey:
Cite (ACL):
Qingyi Si, Zheng Lin, Ming yu Zheng, Peng Fu, and Weiping Wang. 2021. Check It Again:Progressive Visual Question Answering via Visual Entailment. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4101–4110, Online. Association for Computational Linguistics.
Cite (Informal):
Check It Again:Progressive Visual Question Answering via Visual Entailment (Si et al., ACL-IJCNLP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-1/2021.acl-long.317.pdf
Video:
 https://preview.aclanthology.org/nschneid-patch-1/2021.acl-long.317.mp4
Code
 PhoebusSi/SAR
Data
Visual Question Answering