YNU-HPCC at SemEval-2022 Task 5: Multi-Modal and Multi-label Emotion Classification Based on LXMERT

Chao Han, Jin Wang, Xuejie Zhang

Abstract
This paper describes our system used in the SemEval-2022 Task5 Multimedia Automatic Misogyny Identification (MAMI). This task is to use the provided text-image pairs to classify emotions. In this paper, We propose a multi-label emotion classification model based on pre-trained LXMERT. We use Faster-RCNN to extract visual representation and utilize LXMERT’s cross-attention for multi-modal alignment. Then we use the Bilinear-interaction layer to fuse these features. Our experimental results surpass the F1 score of baseline. For Sub-task A, our F1 score is 0.662 and Sub-task B’s F1 score is 0.633. The code of this study is available on GitHub.
Anthology ID:
2022.semeval-1.104
Volume:
Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, Shyam Ratan
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
748–755
Language:
URL:
https://aclanthology.org/2022.semeval-1.104
DOI:
10.18653/v1/2022.semeval-1.104
Bibkey:
Cite (ACL):
Chao Han, Jin Wang, and Xuejie Zhang. 2022. YNU-HPCC at SemEval-2022 Task 5: Multi-Modal and Multi-label Emotion Classification Based on LXMERT. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 748–755, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
YNU-HPCC at SemEval-2022 Task 5: Multi-Modal and Multi-label Emotion Classification Based on LXMERT (Han et al., SemEval 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2022.semeval-1.104.pdf
PDF Search