Contrastive Visual and Language Learning for Visual Relationship Detection

Thanh Tran, Maelic Neau, Paulo Santos, David Powers

Abstract
Visual Relationship Detection aims to understand real-world objects’ interactions by grounding visual concepts to compositional visual relation triples, written in the form of (subject, predicate, object). Previous works have explored the use of contrastive learning to implicitly predict the predicates from the relevant image regions. However, these models often directly leverage in-distribution spatial and language co-occurrences biases during training, preventing the models from generalizing to out-of-distribution compositions. In this work, we examine whether contrastive vision and language models pre-trained on large-scale external image and text dataset can assist the detection of compositional visual relationships. To this end, we propose a semi-supervised contrastive fine-tuning approach for the visual relationship detection task. The results show that fine-tuned models that were pre-trained on larger datasets do not yield better performance when performing visual relationship detection, and larger models can yield lower performance when compared with their smaller counterparts.
Anthology ID:
2022.alta-1.23
Volume:
Proceedings of the 20th Annual Workshop of the Australasian Language Technology Association
Month:
December
Year:
2022
Address:
Adelaide, Australia
Editors:
Pradeesh Parameswaran, Jennifer Biggs, David Powers
Venue:
ALTA
SIG:
Publisher:
Australasian Language Technology Association
Note:
Pages:
170–177
Language:
URL:
https://aclanthology.org/2022.alta-1.23
DOI:
Bibkey:
Cite (ACL):
Thanh Tran, Maelic Neau, Paulo Santos, and David Powers. 2022. Contrastive Visual and Language Learning for Visual Relationship Detection. In Proceedings of the 20th Annual Workshop of the Australasian Language Technology Association, pages 170–177, Adelaide, Australia. Australasian Language Technology Association.
Cite (Informal):
Contrastive Visual and Language Learning for Visual Relationship Detection (Tran et al., ALTA 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2022.alta-1.23.pdf
PDF Search