Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning

Sijie Mai, Ya Sun, Haifeng Hu


Abstract
In the field of multimodal sentiment analysis (MSA), a few studies have leveraged the inherent modality correlation information stored in samples for self-supervised learning. However, they feed the training pairs in a random order without consideration of difficulty. Without human annotation, the generated training pairs of self-supervised learning often contain noise. If noisy or hard pairs are used for training at the easy stage, the model might be stuck in bad local optimum. In this paper, we inject curriculum learning into weakly supervised multimodal correlation learning. The weakly supervised correlation learning leverages the label information to generate scores for negative pairs to learn a more discriminative embedding space, where negative pairs are defined as two unimodal embeddings from different samples. To assist the correlation learning, we feed the training pairs to the model according to difficulty by the proposed curriculum learning, which consists of elaborately designed scoring and feeding functions. The scoring function computes the difficulty of pairs using pre-trained and current correlation predictors, where the pairs with large losses are defined as hard pairs. Notably, the hardest pairs are discarded in our algorithm, which are assumed as noisy pairs. Moreover, the feeding function takes the difference of correlation losses as feedback to determine the feeding actions (‘stay’, ‘step back’, or ‘step forward’). The proposed method reaches state-of-the-art performance on MSA.
Anthology ID:
2022.emnlp-main.209
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3191–3203
Language:
URL:
https://aclanthology.org/2022.emnlp-main.209
DOI:
Bibkey:
Cite (ACL):
Sijie Mai, Ya Sun, and Haifeng Hu. 2022. Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3191–3203, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning (Mai et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-ingestion/2022.emnlp-main.209.pdf