Abstract
We propose our solution to the multimodal semantic role labeling task from the CONSTRAINT’22 workshop. The task aims at classifying entities in memes into classes such as “hero” and “villain”. We use several pre-trained multi-modal models to jointly encode the text and image of the memes, and implement three systems to classify the role of the entities. We propose dynamic sampling strategies to tackle the issue of class imbalance. Finally, we perform qualitative analysis on the representations of the entities.- Anthology ID:
- 2022.constraint-1.7
- Volume:
- Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Tanmoy Chakraborty, Md. Shad Akhtar, Kai Shu, H. Russell Bernard, Maria Liakata, Preslav Nakov, Aseem Srivastava
- Venue:
- CONSTRAINT
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 55–65
- Language:
- URL:
- https://aclanthology.org/2022.constraint-1.7
- DOI:
- 10.18653/v1/2022.constraint-1.7
- Cite (ACL):
- Syrielle Montariol, Étienne Simon, Arij Riabi, and Djamé Seddah. 2022. Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance. In Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations, pages 55–65, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance (Montariol et al., CONSTRAINT 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2022.constraint-1.7.pdf