Cross-modal Transfer Between Vision and Language for Protest Detection
Ria Raj, Kajsa Andreasson, Tobias Norlund, Richard Johansson, Aron Lagerberg
Abstract
Most of today’s systems for socio-political event detection are text-based, while an increasing amount of information published on the web is multi-modal. We seek to bridge this gap by proposing a method that utilizes existing annotated unimodal data to perform event detection in another data modality, zero-shot. Specifically, we focus on protest detection in text and images, and show that a pretrained vision-and-language alignment model (CLIP) can be leveraged towards this end. In particular, our results suggest that annotated protest text data can act supplementarily for detecting protests in images, but significant transfer is demonstrated in the opposite direction as well.- Anthology ID:
- 2022.case-1.8
- Volume:
- Proceedings of the 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Venue:
- CASE
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 56–60
- Language:
- URL:
- https://aclanthology.org/2022.case-1.8
- DOI:
- Cite (ACL):
- Ria Raj, Kajsa Andreasson, Tobias Norlund, Richard Johansson, and Aron Lagerberg. 2022. Cross-modal Transfer Between Vision and Language for Protest Detection. In Proceedings of the 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE), pages 56–60, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- Cross-modal Transfer Between Vision and Language for Protest Detection (Raj et al., CASE 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.case-1.8.pdf