A Comprehensive Interaction in Multiscale Multichannel EEG Signals for Emotion Recognition
Abstract
:1. Introduction
- We propose to jointly consider the spatial relationship and temporal scale for EEG-based emotion recognition with a novel cascade framework.
- With multiscale EEG patch signals, we introduce the SAG to learn the relationship of multiple channels and propose the CET, which is superior to other fusion strategies, to better integrate multiscale representations.
- We conduct extensive experiments on the issues of spatial relationship and temporal scale and set up new state-of-the-art performance on two popular EEG emotion datasets.
2. Related Work
2.1. Valence–Arousal Model
2.2. EEG Emotion Recognition
2.3. The Usage of Multichannel and Multiscale Information
3. Methodology
3.1. Preliminaries
3.2. Overview of Cascade Scale-Aware Adaptive Graph and Cross-Transformer Method
3.3. Scale-Aware Adaptive GCN
3.4. Cross-EEG Transformer
Algorithm 1 Training procedure of SAG-CET |
Input:raw signal data D Output:Network Initialize:,
|
4. Experiments
4.1. Datasets and Experiment Setting
4.2. Comparisons to State of the Art
4.3. Experiment on DEAP
4.4. Experiment on DREAMER
4.5. Results Analysis
4.6. Ablation Study
- By comparing the results of experiments conducted solely using long-stream, short-stream, or dual-stream input on two datasets, we can observe that dual-stream input performs better than long-stream or short-stream input, which fully explains the effective promotion with the application of the cross-attention block.
- By comparing the results of experiments conducted on the SAG block, removing the SAG component from the SAG-CET reduces the performance. The CET with SAG outperforms the CET without SAG, which indicates that the SAG is effective in constructing the spatial correlations of multiple channels. Notably, our analysis showed that the best performance was achieved by combining all the aforementioned factors across all the experiments.
4.7. Limitations and Future Work
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Zheng, L.; Ma, Y.; Li, M.; Xiao, Y.; Feng, W.; Wu, X. Time-frequency decomposition-based weighted ensemble learning for motor imagery EEG classification. In Proceedings of the 2021 IEEE International Conference on Real-time Computing and Robotics (RCAR), Qinghai, China, 15–19 July 2021; IEEE: Piscateville, NJ, USA, 2021; pp. 620–625. [Google Scholar]
- Tang, X.; Zhang, J.; Qi, Y.; Liu, K.; Li, R.; Wang, H. A Spatial Filter Temporal Graph Convolutional Network for decoding motor imagery EEG signals. Expert Syst. Appl. 2024, 238, 121915. [Google Scholar] [CrossRef]
- Wang, Y.; Wu, Q.; Wang, S.; Fang, X.; Ruan, Q. MI-EEG: Generalized model based on mutual information for EEG emotion recognition without adversarial training. Expert Syst. Appl. 2024, 244, 122777. [Google Scholar] [CrossRef]
- Mukherjee, P.; Roy, A.H. EEG sensor driven assistive device for elbow and finger rehabilitation using deep learning. Expert Syst. Appl. 2023, 244, 122954. [Google Scholar] [CrossRef]
- Yu, C.; Wang, M. Survey of emotion recognition methods using EEG information. Cogn. Robot. 2022, 2, 132–146. [Google Scholar] [CrossRef]
- Li, X.; Zhang, Y.; Tiwari, P.; Song, D.; Hu, B.; Yang, M.; Zhao, Z.; Kumar, N.; Marttinen, P. EEG based emotion recognition: A tutorial and review. ACM Comput. Surv. 2022, 55, 1–57. [Google Scholar] [CrossRef]
- Ezzameli, K.; Mahersia, H. Emotion recognition from unimodal to multimodal analysis: A review. Inf. Fusion 2023, 99, 101847. [Google Scholar] [CrossRef]
- Cacioppo, J.T. Feelings and emotions: Roles for electrophysiological markers. Biol. Psychol. 2004, 67, 235–243. [Google Scholar] [CrossRef]
- Liu, Y.; Sourina, O.; Nguyen, M.K. Real-time EEG-based emotion recognition and its applications. In Transactions on Computational Science XII: Special Issue on Cyberworlds; Springer: Berlin/Heidelberg, Germany, 2011; pp. 256–277. [Google Scholar]
- Jie, X.; Cao, R.; Li, L. Emotion recognition based on the sample entropy of EEG. Bio-Med. Mater. Eng. 2014, 24, 1185–1192. [Google Scholar] [CrossRef]
- Liu, Y.; Sourina, O. Real-Time Fractal-Based Valence Level Recognition from EEG. In Transactions on Computational Science XVIII; Gavrilova, M.L., Tan, C.J.K., Kuijper, A., Eds.; Springer: Berlin/Heidelberg, Germany, 2013; pp. 101–120. [Google Scholar]
- Shi, L.C.; Jiao, Y.Y.; Lu, B.L. Differential entropy feature for EEG-based vigilance estimation. In Proceedings of the 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, Japan, 3–7 July 2013; IEEE: Piscateville, NJ, USA, 2013; pp. 6627–6630. [Google Scholar]
- Frantzidis, C.A.; Bratsas, C.; Papadelis, C.L.; Konstantinidis, E.I.; Pappas, C.; Bamidis, P.D. Toward emotion aware computing: An integrated approach using multichannel neurophysiological recordings and affective visual stimuli. IEEE Trans. Inf. Technol. Biomed. 2010, 14, 589–597. [Google Scholar] [CrossRef] [PubMed]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2020, arXiv:2010.11929. [Google Scholar]
- Wang, Y.; Skerry-Ryan, R.J.; Stanton, D.; Wu, Y.; Saurous, R.A. Tacotron: Towards End-to-End Speech Synthesis. arXiv 2017, arXiv:1703.10135. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention is All You Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 4–9 December 2017; NIPS’17. pp. 6000–6010. [Google Scholar]
- Lin, L.; Li, P.; Wang, Q.; Bai, B.; Cui, R.; Yu, Z.; Gao, D.; Zhang, Y. An EEG-based cross-subject interpretable CNN for game player expertise level classification. Expert Syst. Appl. 2024, 237, 121658. [Google Scholar] [CrossRef]
- Choo, S.; Park, H.; Kim, S.; Park, D.; Jung, J.Y.; Lee, S.; Nam, C.S. Effectiveness of multi-task deep learning framework for EEG-based emotion and context recognition. Expert Syst. Appl. 2023, 227, 120348. [Google Scholar] [CrossRef]
- Srinivasan, S.; Johnson, S.D. A Novel Approach to Schizophrenia Detection: Optimized Preprocessing and Deep Learning Analysis of Multichannel EEG Data. Expert Syst. Appl. 2023, 246, 122937. [Google Scholar] [CrossRef]
- Zhang, D.; Yao, L.; Chen, K.; Monaghan, J. A convolutional recurrent attention model for subject-independent EEG signal analysis. IEEE Signal Process. Lett. 2019, 26, 715–719. [Google Scholar] [CrossRef]
- Li, Y.; Zheng, W.; Wang, L.; Zong, Y.; Cui, Z. From regional to global brain: A novel hierarchical spatial-temporal neural network model for EEG emotion recognition. IEEE Trans. Affect. Comput. 2019, 13, 568–578. [Google Scholar] [CrossRef]
- Du, X.; Ma, C.; Zhang, G.; Li, J.; Lai, Y.K.; Zhao, G.; Deng, X.; Liu, Y.J.; Wang, H. An efficient LSTM network for emotion recognition from multichannel EEG signals. IEEE Trans. Affect. Comput. 2020, 13, 1528–1540. [Google Scholar] [CrossRef]
- Bagchi, S.; Bathula, D.R. EEG-ConvTransformer for single-trial EEG-based visual stimulus classification. Pattern Recognit. 2022, 129, 108757. [Google Scholar] [CrossRef]
- Shawky, E.; El-Khoribi, R.; Shoman, M.A.I.; Wahby, M.A. EEG-Based Emotion Recognition using 3D Convolutional Neural Networks. Int. J. Adv. Comput. Sci. Appl. 2018, 9, 329. [Google Scholar]
- Zhang, Z.; Meng, Q.; Jin, L.; Wang, H.; Hou, H. A novel EEG-based graph convolution network for depression detection: Incorporating secondary subject partitioning and attention mechanism. Expert Syst. Appl. 2024, 239, 122356. [Google Scholar] [CrossRef]
- Song, T.; Liu, S.; Zheng, W.; Zong, Y.; Cui, Z. Instance-adaptive graph for EEG emotion recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA, 7–12 February 2020; Volume 34, pp. 2701–2708. [Google Scholar]
- Zhang, G.; Yu, M.; Liu, Y.J.; Zhao, G.; Zhang, D.; Zheng, W. SparseDGCNN: Recognizing Emotion From Multichannel EEG Signals. IEEE Trans. Affect. Comput. 2023, 14, 537–548. [Google Scholar] [CrossRef]
- Jiang, W.B.; Yan, X.; Zheng, W.L.; Lu, B.L. Elastic Graph Transformer Networks for EEG-Based Emotion Recognition. In Proceedings of the ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 4–10 June 2023; IEEE: Piscateville, NJ, USA, 2023; pp. 1–5. [Google Scholar]
- Song, T.; Liu, S.; Zheng, W.; Zong, Y.; Cui, Z.; Li, Y.; Zhou, X. Variational Instance-Adaptive Graph for EEG Emotion Recognition. IEEE Trans. Affect. Comput. 2023, 14, 343–356. [Google Scholar] [CrossRef]
- Costa, M.; Goldberger, A.L.; Peng, C.K. Multiscale entropy analysis of biological signals. Phys. Rev. E 2005, 71, 021906. [Google Scholar] [CrossRef] [PubMed]
- Song, Z.; Deng, B.; Wei, X.; Cai, L.; Yu, H.; Wang, J.; Wang, R.; Chen, Y. Scale-specific effects: A report on multiscale analysis of acupunctured EEG in entropy and power. Phys. A Stat. Mech. Its Appl. 2018, 492, S0378437117311901. [Google Scholar] [CrossRef]
- Wang, Z. Emotion Recognition Based on Multi-scale Convolutional Neural Network. In Proceedings of the Data Mining and Big Data; Tan, Y., Shi, Y., Eds.; Springer Nature: Singapore, 2022; pp. 152–164. [Google Scholar]
- Hu, J.; Wang, C.; Jia, Q.; Bu, Q.; Sutcliffe, R.; Feng, J. ScalingNet: Extracting features from raw EEG data for emotion recognitionPy. Neurocomputing 2021, 463, 177–184. [Google Scholar]
- Wang, Z.M.; Zhang, J.W.; He, Y.; Zhang, J. EEG emotion recognition using multichannel weighted multiscale permutation entropy. Appl. Intell. 2022, 52, 12064–12076. [Google Scholar] [CrossRef]
- Russell, J.A. Core affect and the psychological construction of emotion. Psychol. Rev. 2003, 110, 145. [Google Scholar] [CrossRef]
- Li, M.; Lu, B.L. Emotion classification based on gamma-band EEG. In Proceedings of the 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, USA, 3–6 September 2009; IEEE: Piscateville, NJ, USA, 2009; pp. 1223–1226. [Google Scholar]
- Bazgir, O.; Mohammadi, Z.; Habibi, S.A.H. Emotion recognition with machine learning using EEG signals. In Proceedings of the 2018 25th national and 3rd international iranian conference on biomedical engineering (ICBME), Qom, Iran, 29–30 November 2018; IEEE: Piscateville, NJ, USA, 2018; pp. 1–5. [Google Scholar]
- Khare, S.K.; Bajaj, V. An evolutionary optimized variational mode decomposition for emotion recognition. IEEE Sens. J. 2020, 21, 2035–2042. [Google Scholar] [CrossRef]
- Zhong, S.h.; Fares, A.; Jiang, J. An attentional-LSTM for improved classification of brain activities evoked by images. In Proceedings of the 27th ACM international conference on multimedia, Nice, France, 21–25 October 2019; pp. 1295–1303. [Google Scholar]
- Li, X.; Song, D.; Zhang, P.; Yu, G.; Hou, Y.; Hu, B. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. In Proceedings of the 2016 IEEE international conference on bioinformatics and biomedicine (BIBM), Shenzhen, China, 15–18 December 2016; IEEE: Piscateville, NJ, USA, 2016; pp. 352–359. [Google Scholar]
- Liao, J.; Zhong, Q.; Zhu, Y.; Cai, D. Multimodal physiological signal emotion recognition based on convolutional recurrent neural network. In Proceedings of the IOP Conference Series: Materials Science and Engineering, Chennai, India, 16–17 September 2020; IOP Publishing: Bristol, UK, 2020; Volume 782, p. 032005. [Google Scholar]
- Tao, W.; Li, C.; Song, R.; Cheng, J.; Liu, Y.; Wan, F.; Chen, X. EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans. Affect. Comput. 2020, 14, 382–393. [Google Scholar] [CrossRef]
- Jia, Z.; Lin, Y.; Wang, J.; Feng, Z.; Xie, X.; Chen, C. HetEmotionNet: Two-stream heterogeneous graph recurrent neural network for multi-modal emotion recognition. In Proceedings of the 29th ACM International Conference on Multimedia, online. 15 July 2021; pp. 1047–1056. [Google Scholar]
- Jomaa, M.E.S.H.; Van Bogaert, P.; Jrad, N.; Kadish, N.E.; Japaridze, N.; Siniatchkin, M.; Colominas, M.A.; Humeau-Heurtier, A. Multivariate improved weighted multiscale permutation entropy and its application on EEG data. Biomed. Signal Process. Control 2019, 52, 420–428. [Google Scholar] [CrossRef]
- Su, Y.; Zhang, Z.; Li, X.; Zhang, B.; Ma, H. The multiscale 3D convolutional network for emotion recognition based on electroencephalogram. Front. Neurosci. 2022, 16, 872311. [Google Scholar] [CrossRef] [PubMed]
- Defferrard, M.; Bresson, X.; Vandergheynst, P. Convolutional neural networks on graphs with fast localized spectral filtering. Adv. Neural Inf. Process. Syst. 2016, 29, 3844–3852. [Google Scholar]
- Wright, J.; Liley, D. Dynamics of the brain at global and microscopic scales: Neural networks and the EEG. Behav. Brain Sci. 1996, 19, 285–295. [Google Scholar] [CrossRef]
- Nunez, P.L. Toward a quantitative description of large-scale neocortical dynamic function and EEG. Behav. Brain Sci. 2000, 23, 371–398. [Google Scholar] [CrossRef] [PubMed]
- Hramov, A.E.; Koronovskii, A.A.; Makarov, V.A.; Pavlov, A.N.; Sitnikova, E. Wavelets in Neuroscience; Springer: Berlin/Heidelberg, Germany, 2015. [Google Scholar]
- Koelstra, S.; Muhl, C.; Soleymani, M.; Lee, J.S.; Yazdani, A.; Ebrahimi, T.; Pun, T.; Nijholt, A.; Patras, I. Deap: A database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 2011, 3, 18–31. [Google Scholar] [CrossRef]
- Katsigiannis, S.; Ramzan, N. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J. Biomed. Health Inform. 2017, 22, 98–107. [Google Scholar] [CrossRef] [PubMed]
- Yang, Y.; Wu, Q.; Qiu, M.; Wang, Y.; Chen, X. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil, 8–13 July 2018; IEEE: Piscateville, NJ, USA, 2018; pp. 1–7. [Google Scholar]
- Wang, X.W.; Nie, D.; Lu, B.L. Emotional state classification from EEG data using machine learning approach. Neurocomputing 2014, 129, 94–106. [Google Scholar] [CrossRef]
- Liu, J.; Zhang, L.; Wu, H.; Zhao, H. Transformers for EEG emotion recognition. arXiv 2021, arXiv:2110.06553. [Google Scholar]
- Ma, J.; Tang, H.; Zheng, W.L.; Lu, B.L. Emotion recognition using multimodal residual LSTM network. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 21–25 October 2019; pp. 176–183. [Google Scholar]
- Song, T.; Zheng, W.; Song, P.; Cui, Z. EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks. IEEE Trans. Affect. Comput. 2020, 11, 532–541. [Google Scholar] [CrossRef]
- Jia, Z.; Lin, Y.; Cai, X.; Chen, H.; Gou, H.; Wang, J. Sst-emotionnet: Spatial-spectral-temporal based attention 3d dense network for eeg emotion recognition. In Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA, 12–16 October 2020; pp. 2909–2917. [Google Scholar]
Dataset | Subjects | Label | Experiments | Signals (Channel) |
---|---|---|---|---|
DEAP | 32 | Valence, arousal, dominance | 40/subject | EEG (32), EMG (2), EOG (2), GSR (1), BVP (1), temperature (1), respiration (1) |
DREAMER | 23 | Valence, arousal, dominance, liking | 18/subject | EEG (14), ECG (2) |
Model | Feature | Modality | Backbone |
---|---|---|---|
CRAM [20] | Raw signal | EEG | CNN + LSTM + attention |
SSTemotionNet [57] | Raw signal + DE | EEG | CNN + attention |
ACRNN [42] | Raw signal | EEG | CNN + LSTM + attention |
DGCNN [56] | PSD | EEG | GCN |
IAG [26] | PSD | EEG + PPS | GCN + LSTM |
V-IAG [29] | PSD | EEG + PPS | GCN + LSTM |
HetEmotionNet [43] | Raw signal + DE | EEG + PPS | GCN + GRU |
MMResLSTM [55] | Raw signal | EEG + PPS | LSTM + ResNet |
EeT [54] | Raw signal | EEG | Transformer |
TNAS [6] | Raw signal | EEG | Transformer |
SAG-CET (ours) | Raw signal | EEG | GCN + transformer |
Model | Valence (%) | Arousal (%) |
---|---|---|
CRAM [20] | 87.09 ± 7.49 | 84.46 ± 9.27 |
DGCNN [56] | 90.44 ± 3.01 | 91.70 ± 3.46 |
MMResLSTM [55] | 92.30 ± 1.55 | 92.87 ± 2.11 |
EeT [54] | 93.34 ± 2.12 | 92.86 ± 2.35 |
ACRNN [42] | 93.72 ± 3.21 | 93.38 ± 3.73 |
SST-EmotionNet [57] | 95.54 ± 2.54 | 95.97 ± 2.86 |
HetEmotionNet [43] | 97.66 ± 1.54 | 97.30 ± 1.65 |
TNAS [6] | 98.66 ± 0.94 | 98.68 ± 0.98 |
SAG-CET (ours) | 98.89 ± 0.84 | 98.92 ± 0.81 |
Model | Valence (%) | Arousal (%) |
---|---|---|
DGCNN [56] | 86.23 ± 3.25 | 84.54 ± 3.16 |
IAG [26] | 90.75 ± 2.27 | 91.03 ± 1.65 |
CRAM [20] | 92.27 ± 2.95 | 93.03 ± 1.87 |
V-IAG [29] | 92.82 ± 2.16 | 93.09 ± 1.44 |
TNAS [6] | 96.95 ± 3.35 | 96.41 ± 3.61 |
ACRNN [42] | 97.39 ± 1.37 | 97.98 ± 1.92 |
SAG-CET (ours) | 99.08 ± 0.51 | 99.21 ± 0.72 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Guo, Y.; Zhang, B.; Fan, X.; Shen, X.; Peng, X. A Comprehensive Interaction in Multiscale Multichannel EEG Signals for Emotion Recognition. Mathematics 2024, 12, 1180. https://doi.org/10.3390/math12081180
Guo Y, Zhang B, Fan X, Shen X, Peng X. A Comprehensive Interaction in Multiscale Multichannel EEG Signals for Emotion Recognition. Mathematics. 2024; 12(8):1180. https://doi.org/10.3390/math12081180
Chicago/Turabian StyleGuo, Yiquan, Bowen Zhang, Xiaomao Fan, Xiaole Shen, and Xiaojiang Peng. 2024. "A Comprehensive Interaction in Multiscale Multichannel EEG Signals for Emotion Recognition" Mathematics 12, no. 8: 1180. https://doi.org/10.3390/math12081180