SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis

Minping Chen, Xia Li

Abstract
Multimodal sentiment analysis aims to predict sentiment of language text with the help of other modalities, such as vision and acoustic features. Previous studies focused on learning the joint representation of multiple modalities, ignoring some useful knowledge contained in language modal. In this paper, we try to incorporate sentimental words knowledge into the fusion network to guide the learning of joint representation of multimodal features. Our method consists of two components: shallow fusion part and aggregation part. For the shallow fusion part, we use crossmodal coattention mechanism to obtain bidirectional context information of each two modals to get the fused shallow representations. For the aggregation part, we design a multitask of sentimental words classification to help and guide the deep fusion of the three modalities and obtain the final sentimental words aware fusion representation. We carry out several experiments on CMU-MOSI, CMU-MOSEI and YouTube datasets. The experimental results show that introducing sentimental words prediction as a multitask can really improve the fusion representation of multiple modalities.
Anthology ID:
2020.coling-main.93
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
1067–1077
Language:
URL:
https://aclanthology.org/2020.coling-main.93
DOI:
10.18653/v1/2020.coling-main.93
Bibkey:
Cite (ACL):
Minping Chen and Xia Li. 2020. SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1067–1077, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis (Chen & Li, COLING 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl24-info/2020.coling-main.93.pdf
Code
 gdufsnlp/swafn
Data
CMU-MOSEI
PDF Search Code