Abstract
Multimodal fusion has been proved to improve emotion recognition performance in previous works. However, in real-world applications, we often encounter the problem of missing modality, and which modalities will be missing is uncertain. It makes the fixed multimodal fusion fail in such cases. In this work, we propose a unified model, Missing Modality Imagination Network (MMIN), to deal with the uncertain missing modality problem. MMIN learns robust joint multimodal representations, which can predict the representation of any missing modality given available modalities under different missing modality conditions. Comprehensive experiments on two benchmark datasets demonstrate that the unified MMIN model significantly improves emotion recognition performance under both uncertain missing-modality testing conditions and full-modality ideal testing condition. The code will be available at https://github.com/AIM3-RUC/MMIN.- Anthology ID:
- 2021.acl-long.203
- Volume:
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Month:
- August
- Year:
- 2021
- Address:
- Online
- Editors:
- Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
- Venues:
- ACL | IJCNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2608–2618
- Language:
- URL:
- https://aclanthology.org/2021.acl-long.203
- DOI:
- 10.18653/v1/2021.acl-long.203
- Cite (ACL):
- Jinming Zhao, Ruichen Li, and Qin Jin. 2021. Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2608–2618, Online. Association for Computational Linguistics.
- Cite (Informal):
- Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (Zhao et al., ACL-IJCNLP 2021)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2021.acl-long.203.pdf
- Code
- aim3-ruc/mmin
- Data
- IEMOCAP, MSP-IMPROV