Scene-robust Natural Language Video Localization via Learning Domain-invariant Representations

Zehan Wang, Yang Zhao, Haifeng Huang, Yan Xia, Zhou Zhao

Abstract
Natural language video localization(NLVL) task involves the semantic matching of a text query with a moment from an untrimmed video. Previous methods primarily focus on improving performance with the assumption of independently identical data distribution while ignoring the out-of-distribution data. Therefore, these approaches often fail when handling the videos and queries in novel scenes, which is inevitable in real-world scenarios. In this paper, we, for the first time, formulate the scene-robust NLVL problem and propose a novel generalizable NLVL framework utilizing data in multiple available scenes to learn a robust model. Specifically, our model learns a group of generalizable domain-invariant representations by alignment and decomposition. First, we propose a comprehensive intra- and inter-sample distance metric for complex multi-modal feature space, and an asymmetric multi-modal alignment loss for different information densities of text and vision. Further, to alleviate the conflict between domain-invariant features for generalization and domain-specific information for reasoning, we introduce domain-specific and domain-agnostic predictors to decompose and refine the learned features by dynamically adjusting the weights of samples. Based on the original video tags, we conduct extensive experiments on three NLVL datasets with different-grained scene shifts to show the effectiveness of our proposed methods.
Anthology ID:
2023.findings-acl.11
Volume:
Findings of the Association for Computational Linguistics: ACL 2023
Month:
July
Year:
2023
Address:
Toronto, Canada
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
144–160
Language:
URL:
https://aclanthology.org/2023.findings-acl.11
DOI:
10.18653/v1/2023.findings-acl.11
Bibkey:
Cite (ACL):
Zehan Wang, Yang Zhao, Haifeng Huang, Yan Xia, and Zhou Zhao. 2023. Scene-robust Natural Language Video Localization via Learning Domain-invariant Representations. In Findings of the Association for Computational Linguistics: ACL 2023, pages 144–160, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
Scene-robust Natural Language Video Localization via Learning Domain-invariant Representations (Wang et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/remove-xml-comments/2023.findings-acl.11.pdf
PDF Search