## COGMEN; Official Pytorch Implementation


**CO**ntextualized **G**NN based **M**ultimodal **E**motion recognitio**N**
![Teaser image](./COGMEN_Architecture.png)
**Picture:** *COGMEN Model Architecture*

This repository contains the official Pytorch implementation of the following paper:
> **COGMEN: COntextualized GNN based Multimodal Emotion recognitioN**<br>

> Anonymous ACL submission<br>
>
> **Abstract:** *Emotions are an inherent part of human interactions, and consequently, it is imperative to develop AI systems that understand and recognize human emotions. During a conversation involving various people, the emotions of a person are influenced by the other speaker's utterances and their own emotional state over utterances. In this paper, we propose Contextualized Graph Neural Network (GNN) based Multimodal Emotion Recognition system (COGMEN) that leverages local information (inter/intra dependency between speakers) and global information (context). The proposed model uses GNN based architecture to model the complex dependencies (local and global information) in a conversation. Our model produces SOTA results on IEMOCAP and MOSEI datasets, and further detailed experiments show the importance of modelling information at both levels.*

## Requirements

- We use PyG (PyTorch Geometric) for the GNN component in our architecture. [RGCNConv](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.RGCNConv) and [TransformerConv](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.TransformerConv)

- We use comet for logging all our experiments and its Bayesian optimizer for hyperparameter tuning. 

- For textual features we use [SBERT](https://www.sbert.net/).
### Installations
- [Install PyTorch Geometric](https://pytorch-geometric.readthedocs.io/en/latest/notes/installation.html)

- [Install Comet.ml](https://www.comet.ml/docs/python-sdk/advanced/)
- [Install SBERT](https://www.sbert.net/)


## Preparing datasets for training

        python preprocess.py --dataset="iemocap_4"

## Training networks 

        python train.py --dataset="iemocap_4" --modalities="atv" --seqcontext_nlayer=4 --wp=10 --wf=10 --gnn_nheads=7 --from_begin --class_weight --epochs=55 --drop_rate=0.1

## Results

Table below shows **COGMEN** results on the IEMOCAP dataset for all the modality combinations. 

| Modalities    | IEMOCAP-4way (F1 Score (\%))      | IEMOCAP-6way (F1 Score (\%))   
| :-----       | :---              | :-----   
| a            | 63.58             |  47.579                  
| t            | 81.55             |  66.00                   
| v            | 43.85             |  37.589                  
| at           | 81.59             |  65.42                   
| av           | 64.48             |  52.205                  
| tv           | 81.52             |  62.199                  
**atv** | **84.50**    |  **67.63**           




## Conclusion



> We presented a novel approach of using GNNs for multimodal emotion recognition and propose **COGMEN**: **CO**ntextualized **G**NN based **M**ultimodal **E**motion recognitio**N**. We test **COGMEN** on two widely known multimodal emotion recognition datasets **IEMOCAP** and **MOSEI**. **COGMEN** outperforms the existing state of the art methods in multimodal emotion recognition by a significant margin (**7.7\% increase IEMOCAP 4-Way**). By comprehensive analysis and ablation studies over **COGMEN**, we show the importance of different modules. **COGMEN** fuses information effectively from multiple modalities to improve the performance on emotion prediction tasks. We perform a detailed error analysis and observe that the misclassifications are mainly  between the similar classes and emotion shift cases. We plan to address this in future work, where the focus will be to incorporate a component for capturing the emotional shifts for fine-grained emotion prediction. 

## Acknowledgments
The structure of our code is inspired by [pytorch-DialogueGCN-mianzhang](https://github.com/mianzhang/dialogue_gcn).
