MST-DGCN: A Multi-Scale Spatio-Temporal and Dynamic Graph Convolution Fusion Network for Electroencephalogram Recognition of Motor Imagery
Abstract
:1. Introduction
- We proposed the MST-DGCN model for MI-EEG classification, which incorporates a multi-scale spatio-temporal module based on a CNN and a dynamic graph convolution module based on a GNN. This model extracts effective features relevant to the current task from multiple temporal scales and spatial structures.
- We conducted extensive experiments on real EEG datasets (Dataset IIa from the BCI Competition IV) and compared the results with multiple traditional classical methods and state-of-the-art approaches. Additionally, we designed ablation experiments to validate the contributions and effectiveness of each component of the model.
2. Methods
2.1. Notions and Definitions
2.2. Multi-Scale Spatio-Temporal and Dynamic Graph Convolution Fusion Network (MST-DGCN)
2.2.1. Multi-Scale Spatio-Temporal Module
2.2.2. Dynamic Graph Convolution Module
2.2.3. Classification Module
3. Experiments and Results
3.1. Datasets and Evaluation Metrics
3.2. Data Augmentation and Experimental Setup
3.2.1. Data Augmentation
- Fourier Transform Surrogate (FTS): For the original signal X, the primary operation of the FTS is to calculate the Fourier coefficients for all the channels and then add random noise to their phases.
- where F represents the Fourier transform operator, f is a frequency, Δφ is a frequency-specific random perturbation, and after adding the perturbation, the inverse Fourier transform is applied to obtain the newly augmented signal. This augmentation primarily encourages the model to learn the power spectral density (PSD) information of the signal. In this study, Δφ is set to 9/10 π.
- SmoothTimeMask (STM): For the original signal X with a total time length of t, the STM mainly operates on a specific channel. It randomly samples a time point t_cut from the time range [0, t − Δt], and then it replaces the length of Δt with 0. This augmentation primarily encourages the model to be less affected by transient signal jumps that are irrelevant to the task, thus facilitating the learning of task-relevant patterns.
3.2.2. Preprocessing and Experimental Setup
3.3. Experimental Results
3.3.1. MI Recognition Results Based on MST-DGCN with Data Augmentation
3.3.2. Ablation Experiments of MST-DGCN
- where the MT module represents the multi-scale temporal module, the SA module represents the spatial attention module, and the DGC module represents the dynamic graph convolution module. From Table 2, it can be observed that the removal of any individual module has a relatively minor impact on the performance. The most significant change occurred when the MT module was removed, resulting in a decrease in the accuracy (Acc) of 5.57%. Removing the SA module and DGC module individually led to decreases in the accuracy of 4.09% and 2.93%, respectively. This demonstrates the crucial role of the MT module in effectively integrating global information. The combination of the MT and SA modules achieved the second best performance, indicating that our SA module can effectively refine the global information obtained by the MT module, highlighting the critical roles of the MT and SA modules in information fusion and extraction. It is worth noting that any network solely utilizing the DGC module did not perform well. We speculate that the DGC module may be more suitable for combination with other modules, relying on contextual information.
3.3.3. The Impact of Model Parameters Used in MST-DGCN
4. Discussion
4.1. Analysis of Data Augmentation Methods
4.2. Analysis of Model Complexity and Visualization
4.3. Comparison with Other Studies
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Wolpaw, J.R.; Birbaumer, N.; McFarland, D.J.; Pfurtscheller, G.; Vaughan, T.M. Brain–computer interfaces for communication and control. Clin. Neurophysiol. 2002, 113, 767–791. [Google Scholar] [CrossRef] [PubMed]
- Lebedev, M.A.; Nicolelis, M.A. Brain–machine interfaces: Past, present and future. Trends Neurosci. 2006, 29, 536–546. [Google Scholar] [CrossRef] [PubMed]
- Pfurtscheller, G.; Neuper, C.; Muller, G.; Obermaier, B.; Krausz, G.; Schlogl, A.; Scherer, R.; Graimann, B.; Keinrath, C.; Skliris, D. Graz-BCI: State of the art and clinical applications. IEEE Trans. Neural Syst. Rehabil. Eng. 2003, 11, 1–4. [Google Scholar] [CrossRef] [PubMed]
- Kübler, A.; Birbaumer, N. Brain–computer interfaces and communication in paralysis: Extinction of goal directed thinking in completely paralysed patients? Clin. Neurophysiol. 2008, 119, 2658–2666. [Google Scholar] [CrossRef] [PubMed]
- Allison, B.Z.; Wolpaw, E.W.; Wolpaw, J.R. Brain–computer interface systems: Progress and prospects. Expert Rev. Med. Devices 2007, 4, 463–474. [Google Scholar] [CrossRef] [PubMed]
- Carlson, T.; Millan, J.d.R. Brain-controlled wheelchairs: A robotic architecture. IEEE Robot. Autom. Mag. 2013, 20, 65–73. [Google Scholar] [CrossRef]
- Bundy, D.T.; Souders, L.; Baranyai, K.; Leonard, L.; Schalk, G.; Coker, R.; Moran, D.W.; Huskey, T.; Leuthardt, E.C. Contralesional brain–computer interface control of a powered exoskeleton for motor recovery in chronic stroke survivors. Stroke 2017, 48, 1908–1915. [Google Scholar] [CrossRef] [PubMed]
- Moldoveanu, A.; Ferche, O.; Moldoveanu, F.; Lupu, R.; Cinteză, D.; Irimia, D.; Toader, C. The travee system for a multimodal neuromotor rehabilitation. IEEE Access 2019, 7, 8151–8171. [Google Scholar] [CrossRef]
- Staffa, M.; Giordano, M.; Ficuciello, F. A WiSARD network approach for a BCI-based robotic prosthetic control. Int. J. Soc. Robot. 2020, 12, 749–764. [Google Scholar] [CrossRef]
- Mane, R.; Chouhan, T.; Guan, C. BCI for stroke rehabilitation: Motor and beyond. J. Neural Eng. 2020, 17, 041001. [Google Scholar] [CrossRef]
- Sebastián-Romagosa, M.; Cho, W.; Ortner, R.; Murovec, N.; Von Oertzen, T.; Kamada, K.; Allison, B.Z.; Guger, C. Brain computer interface treatment for motor rehabilitation of upper extremity of stroke patients—A feasibility study. Front. Neurosci. 2020, 14, 591435. [Google Scholar] [CrossRef] [PubMed]
- Grosse-Wentrup, M.; Buss, M. Multiclass common spatial patterns and information theoretic feature extraction. IEEE Trans. Biomed. Eng. 2008, 55, 1991–2000. [Google Scholar] [CrossRef]
- Ang, K.K.; Chin, Z.Y.; Zhang, H.; Guan, C. Filter bank common spatial pattern (FBCSP) in brain-computer interface. In Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China, 1–8 June 2008; pp. 2390–2397. [Google Scholar]
- Kang, H.; Nam, Y.; Choi, S. Composite common spatial pattern for subject-to-subject transfer. IEEE Signal Process. Lett. 2009, 16, 683–686. [Google Scholar] [CrossRef]
- Lotte, F.; Congedo, M.; Lécuyer, A.; Lamarche, F.; Arnaldi, B. A review of classification algorithms for EEG-based brain–computer interfaces. J. Neural Eng. 2007, 4, R1. [Google Scholar] [CrossRef] [PubMed]
- Schirrmeister, R.T.; Springenberg, J.T.; Fiederer, L.D.J.; Glasstetter, M.; Eggensperger, K.; Tangermann, M.; Hutter, F.; Burgard, W.; Ball, T. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum. Brain Mapp. 2017, 38, 5391–5420. [Google Scholar] [CrossRef] [PubMed]
- Lawhern, V.J.; Solon, A.J.; Waytowich, N.R.; Gordon, S.M.; Hung, C.P.; Lance, B.J. EEGNet: A compact convolutional neural network for EEG-based brain–computer interfaces. J. Neural Eng. 2018, 15, 056013. [Google Scholar] [CrossRef] [PubMed]
- Riyad, M.; Khalil, M.; Adib, A. Incep-EEGNet: A convnet for motor imagery decoding. In Proceedings of the Image and Signal Processing: 9th International Conference, ICISP 2020, Marrakesh, Morocco, 4–6 June 2020; pp. 103–111. [Google Scholar]
- Riyad, M.; Khalil, M.; Adib, A. MI-EEGNET: A novel convolutional neural network for motor imagery classification. J. Neurosci. Methods 2021, 353, 109037. [Google Scholar] [CrossRef] [PubMed]
- Woo, S.; Park, J.; Lee, J.-Y.; Kweon, I.S. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 3–19. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Galassi, A.; Lippi, M.; Torroni, P. Attention in natural language processing. IEEE Trans. Neural Netw. Learn. Syst. 2020, 32, 4291–4308. [Google Scholar] [CrossRef] [PubMed]
- Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv 2018, arXiv:1810.04805. [Google Scholar]
- Krizhevsky, A.; Hinton, G. Learning Multiple Layers of Features from Tiny Images. 2009. Available online: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf (accessed on 17 May 2023).
- Xiao, H.; Rasul, K.; Vollgraf, R. Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv 2017, arXiv:1708.07747. [Google Scholar]
- Li, Y.; Guo, L.; Liu, Y.; Liu, J.; Meng, F. A temporal-spectral-based squeeze-and-excitation feature fusion network for motor imagery EEG decoding. IEEE Trans. Neural Syst. Rehabil. Eng. 2021, 29, 1534–1545. [Google Scholar] [CrossRef] [PubMed]
- Li, H.; Zhang, D.; Xie, J. MI-DABAN: A dual-attention-based adversarial network for motor imagery classification. Comput. Biol. Med. 2023, 152, 106420. [Google Scholar] [CrossRef] [PubMed]
- Takahashi, K.; Sun, Z.; Solé-Casals, J.; Cichocki, A.; Phan, A.H.; Zhao, Q.; Zhao, H.-H.; Deng, S.; Micheletto, R. Data augmentation for Convolutional LSTM based brain computer interface system. Appl. Soft Comput. 2022, 122, 108811. [Google Scholar] [CrossRef]
- Rommel, C.; Paillard, J.; Moreau, T.; Gramfort, A. Data augmentation for learning predictive models on EEG: A systematic comparison. J. Neural Eng. 2022, 19, 066020. [Google Scholar] [CrossRef] [PubMed]
- Klepl, D.; He, F.; Wu, M.; Blackburn, D.J.; Sarrigiannis, P.G. Cross-frequency multilayer network analysis with bispectrum-based functional connectivity: A study of Alzheimer’s disease. Neuroscience 2023, 521, 77–88. [Google Scholar] [CrossRef] [PubMed]
- Cao, R.; Hao, Y.; Wang, X.; Gao, Y.; Shi, H.; Huo, S.; Wang, B.; Guo, H.; Xiang, J. EEG functional connectivity underlying emotional valance and arousal using minimum spanning trees. Front. Neurosci. 2020, 14, 355. [Google Scholar] [CrossRef] [PubMed]
- Adebisi, A.T.; Veluvolu, K.C. Brain network analysis for the discrimination of dementia disorders using electrophysiology signals: A systematic review. Front. Aging Neurosci. 2023, 15, 1039496. [Google Scholar] [CrossRef]
- Klepl, D.; Wu, M.; He, F. Graph neural network-based eeg classification: A survey. IEEE Trans. Neural Syst. Rehabil. Eng. 2024, 32, 493–503. [Google Scholar] [CrossRef] [PubMed]
- Xu, L.; Zhang, H.; Hui, M.; Long, Z.; Jin, Z.; Liu, Y.; Yao, L. Motor execution and motor imagery: A comparison of functional connectivity patterns based on graph theory. Neuroscience 2014, 261, 184–194. [Google Scholar] [CrossRef]
- Kılıç, B.; Aydın, S. Classification of contrasting discrete emotional states indicated by EEG based graph theoretical network measures. Neuroinformatics 2022, 20, 863–877. [Google Scholar] [CrossRef]
- Veličković, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio, P.; Bengio, Y. Graph attention networks. arXiv 2017, arXiv:1710.10903. [Google Scholar]
- Seo, Y.; Defferrard, M.; Vandergheynst, P.; Bresson, X. Structured sequence modeling with graph convolutional recurrent networks. In Proceedings of the Neural Information Processing: 25th International Conference, ICONIP 2018, Siem Reap, Cambodia, 13–16 December 2018; pp. 362–373. [Google Scholar]
- Shen, L.; Sun, M.; Li, Q.; Li, B.; Pan, Z.; Lei, J. Multiscale temporal self-attention and dynamical graph convolution hybrid network for EEG-based stereogram recognition. IEEE Trans. Neural Syst. Rehabil. Eng. 2022, 30, 1191–1202. [Google Scholar] [CrossRef]
- Wu, H.; Niu, Y.; Li, F.; Li, Y.; Fu, B.; Shi, G.; Dong, M. A Parallel Multiscale Filter Bank Convolutional Neural Networks for Motor Imagery EEG Classification. Front. Neurosci. 2019, 13, 1275. [Google Scholar] [CrossRef]
- Bo, D.; Wang, X.; Liu, Y.; Fang, Y.; Li, Y.; Shi, C. A survey on spectral graph neural networks. arXiv 2023, arXiv:2302.05631. [Google Scholar]
- Defferrard, M.; Bresson, X.; Vandergheynst, P. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the Advances in Neural Information Processing Systems 29 (NIPS 2016), Barcelona, Spain, 5–10 December 2016. [Google Scholar]
- Brunner, C.; Leeb, R.; Müller-Putz, G.; Schlögl, A.; Pfurtscheller, G. BCI Competition 2008–Graz Data Set A; Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces), Graz University of Technology: Graz, Austria, 2008; Volume Voume 16, pp. 1–6. [Google Scholar]
- Dornhege, G. Toward Brain-Computer Interfacing; MIT Press: Cambridge, MA, USA, 2007. [Google Scholar]
- Altaheri, H.; Muhammad, G.; Alsulaiman, M. Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification. IEEE Trans. Ind. Inform. 2023, 19, 2249–2258. [Google Scholar] [CrossRef]
- Thanigaivelu, P.S.; Sridhar, S.; Sulthana, S.F. OISVM: Optimal Incremental Support Vector Machine-based EEG Classification for Brain-computer Interface Model. Cogn. Comput. 2023, 15, 888–903. [Google Scholar] [CrossRef]
- Xie, X.; Zou, X.; Yu, T.; Tang, R.; Hou, Y.; Qi, F. Multiple graph fusion based on Riemannian geometry for motor imagery classification. Appl. Intell. 2022, 52, 9067–9079. [Google Scholar] [CrossRef]
- Ni, T.; He, C.; Gu, X. Semi-supervised classifier with projection graph embedding for motor imagery electroencephalogram recognition. Multimed. Tools Appl. 2024, 83, 14189–14209. [Google Scholar] [CrossRef]
Methods | Subjects’ Acc (%) | Kappa | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
A01 | A02 | A03 | A04 | A05 | A06 | A07 | A08 | A09 | Avg | ||
MST-DGCN | 85.71 ± 0.22 | 55.07 ± 1.73 | 92.36 ± 0.90 | 67.35 ± 0.87 | 65.90 ± 1.62 | 55.87 ± 3.12 | 88.54 ± 2.14 | 79.28 ± 0.93 | 84.31 ± 0.40 | 74.93 ± 0.21 | 0.6657 |
STM + MST-DGCN | 84.10 ± 2.08 | 59.19 ± 5.00 | 92.47 ± 0.74 | 68.51 ± 1.50 | 68.86 ± 1.98 | 58.51 ± 1.74 | 88.91 ± 4.40 | 80.88 ± 0.52 | 84.78 ± 0.86 | 76.25 ± 1.53 | 0.6833 |
FTS + MST-DGCN | 86.40 ± 1.19 | 64.81 ± 5.95 | 94.89 ± 0.35 | 69.18 ± 3.28 | 69.73 ± 0.33 | 60.80 ± 1.26 | 89.22 ± 1.70 | 81.69 ± 0.56 | 84.26 ± 0.59 | 77.89 ± 0.75 | 0.7052 |
Modules | Acc (%) | ||
---|---|---|---|
MT | SA | DGC | |
✓ | 71.08% | ||
✓ | 68.73% | ||
✓ | 66.24% | ||
✓ | ✓ | 73.80% | |
✓ | ✓ | 72.32% | |
✓ | ✓ | 74.96% | |
✓ | ✓ | ✓ | 77.89% |
Used Scales | Acc (%) | |
---|---|---|
Small Scales (26, 1) and (16, 1) | Large Scales (64, 1) and (40, 1) | |
✓ | 73.20% | |
✓ | 75.44% | |
✓ | ✓ | 77.89% |
Layers | Acc |
---|---|
1–2 | All results < 70% |
3 | 71.98% |
4 | 75.59% |
5 | 77.89% |
6 | 72.34% |
7 | 69.76% |
Acc | |
---|---|
0.1 | 73.83% |
0.2 | 75.31% |
0.3 | 77.89% |
0.4 | 76.56% |
0.5 | 75.92% |
Methods | Parameter (Million) | Inference Time (ms) | Decoding Time (ms) | Acc (%) |
---|---|---|---|---|
ConvNet | 0.15 | 0.85 | 6.87 | 72.53 |
EEGNet | 0.03 | 0.32 | 4.28 | 74.61 |
TS-SEFFNet | 1.34 | 3.29 | 18.48 | 74.71 |
MST-DGCN | 0.16 | 1.27 | 11.35 | 77.89 |
Methods | Year | Acc | Kappa |
---|---|---|---|
FBCSP [13] | 2008 | 67.75% | 0.5700 |
CCSP [14] | 2009 | 66.51% | 0.5535 |
ConvNet [16] | 2017 | 72.53% | 0.6337 |
EEGNet [17] | 2018 | 74.61% | 0.6615 |
Incep-EEGNet [18] | 2020 | 74.07% | 0.6543 |
TS-SEFFNet [26] | 2021 | 74.71% | 0.6628 |
MRGF [46] | 2022 | 70.11% | 0.6015 |
MI-DABAN [27] | 2023 | 76.16% | 0.6821 |
SCPGE [47] | 2023 | 68.64% | 0.5817 |
Our Method | 2024 | 77.89% | 0.7052 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, Y.; Liu, P.; Li, D. MST-DGCN: A Multi-Scale Spatio-Temporal and Dynamic Graph Convolution Fusion Network for Electroencephalogram Recognition of Motor Imagery. Electronics 2024, 13, 2174. https://doi.org/10.3390/electronics13112174
Chen Y, Liu P, Li D. MST-DGCN: A Multi-Scale Spatio-Temporal and Dynamic Graph Convolution Fusion Network for Electroencephalogram Recognition of Motor Imagery. Electronics. 2024; 13(11):2174. https://doi.org/10.3390/electronics13112174
Chicago/Turabian StyleChen, Yuanling, Peisen Liu, and Duan Li. 2024. "MST-DGCN: A Multi-Scale Spatio-Temporal and Dynamic Graph Convolution Fusion Network for Electroencephalogram Recognition of Motor Imagery" Electronics 13, no. 11: 2174. https://doi.org/10.3390/electronics13112174