Skip to content

Commit

Permalink
Update the results on the SIMSv2 dataset
Browse files Browse the repository at this point in the history
  • Loading branch information
@cherishPre
cherishPre committed Dec 27, 2023
1 parent de4c48f commit 698bae0
Show file tree
Hide file tree
Showing 10 changed files with 278 additions and 19 deletions.
22 changes: 11 additions & 11 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -132,26 +132,26 @@ MMSA uses feature files that are organized as follows:

## 3. Supported MSA Models

| Type | Model Name | From | Published |
| Type | Model Name | From | Published |
| :---------: | :-----------------------------------------------------: | :------------------------------------------------------------------------------------: | :---------------: |
| Single-Task | [TFN](src/MMSA/models/singleTask/TFN.py) | [Tensor-Fusion-Network](https://github.com/A2Zadeh/TensorFusionNetwork) | EMNLP 2017 |
| Single-Task | [TFN](src/MMSA/models/singleTask/TFN.py) | [Tensor-Fusion-Network](https://github.com/A2Zadeh/TensorFusionNetwork) | EMNLP 2017 |
| Single-Task | [EF_LSTM](src/MMSA/models/singleTask/EF_LSTM.py) | [MultimodalDNN](https://github.com/rhoposit/MultimodalDNN) | ACL 2018 Workshop |
| Single-Task | [LF_DNN](src/MMSA/models/singleTask/LF_DNN.py) | [MultimodalDNN](https://github.com/rhoposit/MultimodalDNN) | ACL 2018 Workshop |
| Single-Task | [LMF](src/MMSA/models/singleTask/LMF.py) | [Low-rank-Multimodal-Fusion](https://github.com/Justin1904/Low-rank-Multimodal-Fusion) | ACL 2018 |
| Single-Task | [MFN](src/MMSA/models/singleTask/MFN.py) | [Memory-Fusion-Network](https://github.com/pliang279/MFN) | AAAI 2018 |
| Single-Task | [MFN](src/MMSA/models/singleTask/MFN.py) | [Memory-Fusion-Network](https://github.com/pliang279/MFN) | AAAI 2018 |
| Single-Task | [Graph-MFN](src/MMSA/models/singleTask/Graph_MFN.py) | [Graph-Memory-Fusion-Network](https://github.com/A2Zadeh/CMU-MultimodalSDK/) | ACL 2018 |
| Single-Task | [MulT](src/MMSA/models/singleTask/MulT.py)(without CTC) | [Multimodal-Transformer](https://github.com/yaohungt/Multimodal-Transformer) | ACL 2019 |
| Single-Task | [MFM](src/MMSA/models/singleTask/MFM.py) | [MFM](https://github.com/pliang279/factorized/) | ICRL 2019 |
| Single-Task | [MFM](src/MMSA/models/singleTask/MFM.py) | [MFM](https://github.com/pliang279/factorized/) | ICRL 2019 |
| Multi-Task | [MLF_DNN](src/MMSA/models/multiTask/MLF_DNN.py) | [MMSA](https://github.com/thuiar/MMSA) | ACL 2020 |
| Multi-Task | [MTFN](src/MMSA/models/multiTask/MTFN.py) | [MMSA](https://github.com/thuiar/MMSA) | ACL 2020 |
| Multi-Task | [MLMF](src/MMSA/models/multiTask/MLMF.py) | [MMSA](https://github.com/thuiar/MMSA) | ACL 2020 |
| Multi-Task | [SELF_MM](src/MMSA/models/multiTask/SELF_MM.py) | [Self-MM](https://github.com/thuiar/Self-MM) | AAAI 2021 |
| Multi-Task | [TETFN](src/MMSA/models/multiTask/TETFN.py) | TETFN | PR 2023 |
| Multi-Task | [SELF_MM](src/MMSA/models/multiTask/SELF_MM.py) | [Self-MM](https://github.com/thuiar/Self-MM) | AAAI 2021 |
| Single-Task | [BERT-MAG](src/MMSA/models/singleTask/BERT_MAG.py) | [MAG-BERT](https://github.com/WasifurRahman/BERT_multimodal_transformer) | ACL 2020 |
| Single-Task | [MISA](src/MMSA/models/singleTask/MISA.py) | [MISA](https://github.com/declare-lab/MISA) | ACMMM 2020 | |
| Single-Task | [MMIM](src/MMSA/models/singleTask/MMIM.py) | [MMIM](https://github.com/declare-lab/Multimodal-Infomax) | EMNLP 2021 |
| Single-Task | BBFN (Work in Progress) | [BBFN](https://github.com/declare-lab/BBFN) | ICMI 2021 |
| Single-Task | [CENET](src/MMSA/models/singleTask/CENET.py) | [CENET](https://github.com/Say2L/CENet) | TMM 2022 |
| Single-Task | [MISA](src/MMSA/models/singleTask/MISA.py) | [MISA](https://github.com/declare-lab/MISA) | ACMMM 2020 |
| Single-Task | [MMIM](src/MMSA/models/singleTask/MMIM.py) | [MMIM](https://github.com/declare-lab/Multimodal-Infomax) | EMNLP 2021 |
| Single-Task | BBFN (Work in Progress) | [BBFN](https://github.com/declare-lab/BBFN) | ICMI 2021 |
| Single-Task | [CENET](src/MMSA/models/singleTask/CENET.py) | [CENET](https://github.com/Say2L/CENet) | TMM 2022 |
| Multi-Task | [TETFN](src/MMSA/models/multiTask/TETFN.py) | TETFN | PR 2023 |


## 4. Results
Expand Down Expand Up @@ -186,4 +186,4 @@ Please cite our paper if you find our work useful for your research:
pages={10790--10797},
year={2021}
}
```
```
21 changes: 21 additions & 0 deletions results/result-stat.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,28 @@
| tetfn |81.18 |63.24 |41.79 |80.24 |42.00 |57.65 | Unaligned |
| cenet |77.90 |62.58 |33.92 |77.53 |47.09 |53.95 | Unaligned |

- SIMSv2

| Model | Mult_acc_2 | Mult_acc_3 | Mult_acc_5 | F1_score | MAE | Corr | Data Setting |
| :-------: | :--------: | :--------: | :--------: | :------: | :---: | :---: | :----------: |
| ef_lstm | 78.94 | 72.38 | 49.26 | 79.03 | 33.74 | 65.88 | Unaligned |
| lf_dnn | 79.36 | 72.59 | 52.76 | 79.46 | 30.14 | 71.20 | Unaligned |
| tfn | 80.14 | 72.21 | 52.55 | 80.14 | 30.31 | 70.73 | Unaligned |
| lmf | 74.18 | 64.90 | 47.79 | 73.88 | 36.72 | 55.69 | Unaligned |
| mfn | 81.14 | 73.66 | 54.53 | 81.19 | 29.54 | 72.66 | Unaligned |
| graph_mfn | 73.35 | 67.18 | 45.78 | 72.60 | 37.87 | 57.43 | Unaligned |
| mult | 80.68 | 73.19 | 54.81 | 80.73 | 29.05 | 73.78 | Unaligned |
| mlf_dnn | 76.59 | 68.47 | 49.67 | 76.62 | 33.52 | 63.95 | Unaligned |
| mtfn | 80.43 | 73.71 | 56.05 | 80.50 | 29.26 | 72.03 | Unaligned |
| mlmf | 77.23 | 69.98 | 51.22 | 77.3 | 32.22 | 67.03 | Unaligned |
| self_mm | 79.69 | 72.61 | 52.77 | 79.76 | 31.06 | 69.52 | Unaligned |
| tetfn | 79.73 | 73.65 | 54.47 | 79.81 | 31.03 | 69.50 | Unaligned |
| cenet | 79.56 | 73.10 | 53.04 | 79.63 | 30.96 | 69.94 | Unaligned |



## Classification

> Data setting is the same as `Regression`
- MOSI
Expand Down
2 changes: 1 addition & 1 deletion src/MMSA/__main__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ def parse_args():
choices=['lf_dnn', 'ef_lstm', 'tfn', 'mctn','lmf', 'mfn', 'graph_mfn', 'mult', 'bert_mag',
'misa', 'mfm', 'mlf_dnn', 'mtfn', 'mlmf', 'self_mm', 'mmim','tfr_net','tetfn','cenet'])
parser.add_argument('-d', '--dataset', type=str, default='sims',
choices=['sims', 'mosi', 'mosei'], help='Name of dataset')
choices=['sims', 'mosi', 'mosei', 'simsv2'], help='Name of dataset')
parser.add_argument('-c', '--config', type=str, default='',
help='Path to config file. If not specified, default config file will be used.')
parser.add_argument('-t', '--tune', action='store_true',
Expand Down
18 changes: 18 additions & 0 deletions src/MMSA/config/citations.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,18 @@
"paper_url": "https://arxiv.org/pdf/2109.00412.pdf",
"citation": "",
"description": "MultiModal-InfoMax"
},
"TETFN": {
"title": "TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis",
"paper_url": "https://www.sciencedirect.com/science/article/pii/S0031320322007385",
"citation": "",
"description": ""
},
"CENET": {
"title": "Cross-modal Enhancement Network for Multimodal Sentiment Analysis",
"paper_url": "https://ieeexplore.ieee.org/abstract/document/9797846",
"citation": "",
"description": ""
}
},
"datasets": {
Expand All @@ -109,6 +121,12 @@
"paper_url": "https://aclanthology.org/P18-1208.pdf",
"citation": "",
"description": "The CMU-MOSEI Datset."
},
"SIMSv2": {
"title": "Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module",
"paper_url": "https://dl.acm.org/doi/pdf/10.1145/3536221.3556630",
"citation": "",
"description": "CH-SIMSv2: Chinese Multimodal Sentiment Analysis Dataset."
}
}
}
Loading

0 comments on commit 698bae0

Please sign in to comment.