Exploring Contrastive Learning for Multimodal Detection of Misogynistic Memes

Charic Farinango Cuervo, Natalie Parde


Abstract
Misogynistic memes are rampant on social media, and often convey their messages using multimodal signals (e.g., images paired with derogatory text or captions). However, to date very few multimodal systems have been leveraged for the detection of misogynistic memes. Recently, researchers have turned to contrastive learning, and most notably OpenAI’s CLIP model, is an innovative solution to a variety of multimodal tasks. In this work, we experiment with contrastive learning to address the detection of misogynistic memes within the context of SemEval 2022 Task 5. Although our model does not achieve top results, these experiments provide important exploratory findings for this task. We conduct a detailed error analysis, revealing promising clues and offering a foundation for follow-up work.
Anthology ID:
2022.semeval-1.109
Volume:
Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)
Month:
July
Year:
2022
Address:
Seattle, United States
Venue:
SemEval
SIGs:
SIGLEX | SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
785–792
Language:
URL:
https://aclanthology.org/2022.semeval-1.109
DOI:
10.18653/v1/2022.semeval-1.109
Bibkey:
Cite (ACL):
Charic Farinango Cuervo and Natalie Parde. 2022. Exploring Contrastive Learning for Multimodal Detection of Misogynistic Memes. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 785–792, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Exploring Contrastive Learning for Multimodal Detection of Misogynistic Memes (Farinango Cuervo & Parde, SemEval 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/author-url/2022.semeval-1.109.pdf
Video:
 https://preview.aclanthology.org/author-url/2022.semeval-1.109.mp4
Code
 moein-shariatnia/OpenAI-CLIP +  additional community code
Data
Hateful Memes