Abstract
Event-Level Video Question Answering (EVQA) requires complex reasoning across video events to obtain the visual information needed to provide optimal answers. However, despite significant progress in model performance, few studies have focused on using the explicit semantic connections between the question and visual information especially at the event level. There is need for using such semantic connections to facilitate complex reasoning across video frames. Therefore, we propose a semantic-aware dynamic retrospective-prospective reasoning approach for video-based question answering. Specifically, we explicitly use the Semantic Role Labeling (SRL) structure of the question in the dynamic reasoning process where we decide to move to the next frame based on which part of the SRL structure (agent, verb, patient, etc.) of the question is being focused on. We conduct experiments on a benchmark EVQA dataset - TrafficQA. Results show that our proposed approach achieves superior performance compared to previous state-of-the-art models. Our code is publicly available at https://github.com/lyuchenyang/Semantic-aware-VideoQA.- Anthology ID:
- 2023.acl-srw.7
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Vishakh Padmakumar, Gisela Vallejo, Yao Fu
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 50–56
- Language:
- URL:
- https://aclanthology.org/2023.acl-srw.7
- DOI:
- 10.18653/v1/2023.acl-srw.7
- Cite (ACL):
- Chenyang Lyu, Tianbo Ji, Yvette Graham, and Jennifer Foster. 2023. Semantic-Aware Dynamic Retrospective-Prospective Reasoning for Event-Level Video Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 50–56, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Semantic-Aware Dynamic Retrospective-Prospective Reasoning for Event-Level Video Question Answering (Lyu et al., ACL 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.acl-srw.7.pdf