Motor Imaging EEG Signal Recognition of ResNet18 Network Based on Deformable Convolution
Abstract
:1. Introduction
2. Deformable Convolutional Network
3. DCN-ResNet18 Network
3.1. ResNet18 Network
3.2. Improved ResNet18 Network Model Building
4. Experimental Simulation and Analysis
4.1. Experimental Test Dataset
- BCIC IV dataset 2b
- BCIC IV dataset 2a
- BCIC III dataset 3a
4.2. Data Preprocessing
4.3. Simulation Verification and Analysis of Results
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Jiang, G.; Zhao, C. A Review of EEG-based Brain-Computer Interface Development. Comput. Meas. Control 2022, 30, 1–8. [Google Scholar]
- Wang, H.; Hu, J.; Wang, Y. A review of EEG signal processing methods. Comput. Age 2018, 13–15+19. [Google Scholar] [CrossRef]
- Wang, D.; Tao, Q.; Zhang, X.; Wu, B.; Fang, J.; Lu, Z. Four Types of Expression-Assisted EEG Signal Recognition Methods Using Improved Cospatial Mode Algorithm. J. Xi’an Jiaotong Univ. 2022, 1–9. Available online: http://kns.cnki.net/kcms/detail/61.1069.T.20220822.1552.002.html (accessed on 30 October 2022).
- Singh, A.; Hussain, A.A.; Lal, S.; Guesgen, H.W. A comprehensive review on critical issues and possible solutions of motor imagery based electroencephalography brain-computer interface. Sensors 2021, 21, 2173. [Google Scholar] [CrossRef] [PubMed]
- Raza, H.; Chowdhury, A.; Bhattacharyya, S.; Samothrakis, S. Single-trial EEG classification with EEGNet and neural structured learning for improving BCI performance. In Proceedings of the IEEE International Joint Conference on Neural Networks, Glasgow, UK, 19–24 July 2020; pp. 1–8. [Google Scholar]
- Yong, Y.; Zhang, H.; Cheng, Q.; Sun, G.; Yang, J. Hybrid brain-computer interface and its research progress. Comput. Meas. Control 2020, 28, 9–13. [Google Scholar]
- Zhu, J. Multi-Perspective Clustering Model for Epilepsy EEG Signals; Jiangnan University: Wuxi, China, 2021. [Google Scholar]
- Arias-Vergara, T.; Klumpp, P.; Vasquez-Correa, J.C.; Nöth, E.; Orozco-Arroyave, J.R.; Schuster, M. Multi-channel spectrograms for speech processing applications using deep learning methods. Pattern Anal. Appl. 2021, 24, 423–431. [Google Scholar] [CrossRef]
- Lopac, N.; Hržić, F.; Vuksanović, I.P.; Lerga, J. Detection of Non-Stationary GW Signals in High Noise From Cohen’s Class of Time–Frequency Representations Using Deep Learning. IEEE Access 2021, 10, 2408–2428. [Google Scholar] [CrossRef]
- Khare, S.K.; Bajaj, V. Time–frequency representation and convolutional neural network-based emotion recognition. IEEE Trans. Neural Netw. Learn. Syst. 2020, 32, 2901–2909. [Google Scholar] [CrossRef]
- Xu, B.; Zhang, L.; Song, A.; Wu, C.; Li, W.; Zhang, D.; Xu, G.; Li, H.; Zeng, H. Wavelet transform time-frequency image and convolutional network-based motor imagery EEG classification. IEEE Access 2018, 7, 6084–6093. [Google Scholar] [CrossRef]
- Liu, B.; Cai, M.; Bo, Y.; Zhang, X. A Feature Extraction and Classification Algorithm of Motor Imaging EEG Signal Based on PSO-CSP-SVM. J. Cent. South Univ. 2020, 51, 2855–2866. [Google Scholar]
- Ji, J.; She, Q.; Zhang, Q.; Meng, M. Classification method of motor imaginative EEG signals based on optimal regional cospatial mode. Chin. J. Sens. Technol. 2020, 33, 34–39. [Google Scholar]
- Shan, H.; Zhu, S. Brain-computer interface channel selection based on Relief-SBS. Chin. J. Biomed. Eng. 2016, 33, 350–356. [Google Scholar]
- Feng, J.; Jin, J.; Daly, I.; Zhou, J.; Niu, Y.; Wang, X.; Cichocki, A. An optimized channel selection method based on multifrequency CSP-rank for motor imagery-based BCI system. Comput. Intell. Neurosci. 2019, 2019, 8068357. [Google Scholar] [CrossRef] [Green Version]
- Jin, J.; Miao, Y.; Daly, I.; Zuo, C.; Hu, D.; Cichocki, A. Correlation-based channel selection and regularized feature optimization for MI-based BCI. Neural Netw. 2019, 118, 262–270. [Google Scholar] [CrossRef] [PubMed]
- Varsehi, H.; Firoozabadi, S.M.P. An EEG channel selection method for motor imagery based brain–computer interface and neurofeedback using Granger causality. Neural Netw. 2021, 133, 193–206. [Google Scholar] [CrossRef] [PubMed]
- Dai, J.; Qi, H.; Xiong, Y.; Li, Y.; Zhang, G.; Hu, H.; Wei, Y. Deformable convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 764–773. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778. [Google Scholar]
- Tangermann, M.; Müller, K.-R.; Aertsen, A.; Birbaumer, N.; Braun, C.; Brunner, C.; Leeb, R.; Mehring, C.; Miller, K.J.; Müller-Putz, G.; et al. Review of the BCI competition IV. Front. Neurosci. 2012, 6, 55. [Google Scholar] [CrossRef]
- Brunner, C.; Leeb, R.; Müller-Putz, G.; Schlögl, A.; Pfurtscheller, G. BCI Competition 2008–Graz Data Set A. In Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces); Graz University of Technology: Graz, Austria, 2008; pp. 1–6. [Google Scholar]
- Schlögl, A.; Pfurtscheller, G. Dataset IIIa: 4-Class EEG Data. BCI Compet III. 2005. Available online: https://www.bbci.de/competition/iii/ (accessed on 4 October 2022).
- Koles, Z.J.; Lazar, M.S.; Zhou, S.Z. Spatial patterns underlying population differences in the background EEG. Brain Topogr. 1990, 2, 275–284. [Google Scholar] [CrossRef]
- Shovon, T.H.; Al Nazi, Z.; Dash, S.; Hossain, M.F. September. Classification of motor imagery EEG signals with multi-input convolutional neural network by augmenting STFT. In Proceedings of the 2019 5th International Conference on Advances in Electrical Engineering (ICAEE), Dhaka, Bangladesh, 26–28 September 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 398–403. [Google Scholar]
- Wang, J.; Feng, Z.; Ren, X.; Lu, N.; Luo, J.; Sun, L. Feature subset and time segment selection for the classification of EEG data based motor imagery. Biomed. Signal Process. Control 2020, 61, 102026. [Google Scholar] [CrossRef]
- Tang, X.; Li, W.; Li, X.; Ma, W.; Dang, X. Motor imagery EEG recognition based on conditional optimization empirical mode decomposition and multi-scale convolutional neural network. Expert Syst. Appl. 2020, 149, 113285. [Google Scholar] [CrossRef]
- Chu, Y.; Zhao, X.; Zou, Y.; Xu, W.; Han, J.; Zhao, Y. A decoding scheme for incomplete motor imagery EEG with deep belief 18 network. Front. Neurosci. 2018, 12, 680. [Google Scholar] [CrossRef]
- Li, D.; Wang, J.; Xu, J.; Fang, X. Densely feature fusion based on convolutional neural networks for motor imagery EEG classification. IEEE Access 2019, 7, 132720–132730. [Google Scholar] [CrossRef]
- Lawhern, V.J.; Solon, A.J.; Waytowich, N.R.; Gordon, S.M.; Hung, C.P.; Lance, B.J. EEGNet: A compact convolutional neural network for EEG-based brain–computer interfaces. J. Neural Eng. 2018, 15, 056013. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhang, R.; Zong, Q.; Dou, L.; Zhao, X. A novel hybrid deep learning scheme for four-class motor imagery classification. J. Neural Eng. 2019, 16, 066004. [Google Scholar] [CrossRef] [PubMed]
- Gaur, P.; Pachori, R.B.; Wang, H.; Prasad, G. A multi-class EEG-based BCI classification using multivariate empirical mode decomposition based filtering and Riemannian geometry. Expert Syst. Appl. 2018, 95, 201–211. [Google Scholar] [CrossRef]
- Wu, H.; Niu, Y.; Li, F.; Li, Y.; Fu, B.; Shi, G.; Dong, M. A parallel multiscale filter bank convolutional neural networks for motor imagery EEG classification. Front. Neurosci. 2019, 13, 1275. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Liu, X.; Shen, Y.; Liu, J.; Yang, J.; Xiong, P.; Lin, F. Parallel spatial–temporal self-attention CNN-based motor imagery classification for BCI. Front. Neurosci. 2020, 14, 587520. [Google Scholar] [CrossRef] [PubMed]
- Song, Y.; Wang, D.; Yue, K.; Zheng, N.; Shen, Z.J.M. EEG-based motor imagery classification with deep multi-task learning. In Proceedings of the 2019 International Joint Conference on Neural Networks, Budapest, Hungary, 14–19 July 2019; pp. 1–8. [Google Scholar]
- Amin, S.U.; Alsulaiman, M.; Muhammad, G.; Mekhtiche, M.A.; Hossain, M.S. Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion. Future Gener. Comput. Syst. 2019, 101, 542–554. [Google Scholar] [CrossRef]
- Ingolfsson, T.M.; Hersche, M.; Wang, X.; Kobayashi, N.; Cavigelli, L.; Benini, L. EEG-TCNet: An accurate temporal convolutional network for embedded motor-imagery brain–machine interfaces. In Proceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Toronto, ON, Canada, 11–14 October 2020; pp. 2958–2965. [Google Scholar]
- Zhu, L.; Zhu, J.; Ding, W.; Yang, J.; Hu, Q.; Ying, N.; Xu, P.; Zhang, J. Feature extraction algorithm of motor imaging EEG signals based on kernel method and manifold learning. J. Sens. Technol. 2022, 35, 504–510. [Google Scholar]
- Zhao, X.; Zhang, H.; Zhu, G.; You, F.; Kuang, S.; Sun, L. A multi-branch 3D convolutional neural network for EEG-based motor imagery classification. IEEE Trans. Neural Syst. Rehabil. Eng. 2019, 27, 2164–2177. [Google Scholar] [CrossRef]
ResNet18 | DCN-ResNet18 | ||||
---|---|---|---|---|---|
Layer-Name | Output-Size | Module-Size | Layer-Name | Output-Size | Module-Size |
conv1 | 112 × 112 | 7 × 7, 64, stride 2 | conv1 | 112 × 112 | 7 × 7, 64, stride 2 |
conv2_x | 56 × 56 | 3 × 3 max pool, stride 2 | conv2_x | 56 × 56 | 3 × 3 max pool, stride 2 |
conv3_x | 28 × 28 | Conv3_x | 28 × 2814 × 14 | ||
conv4_x | 14 × 14 | Conv4_x | 28 × 2814 × 14 | ||
conv5_x | 7 × 7 | conv5_x | 7 × 7 | ||
1 × 1 | average pool fully connected layer softmax | 1 × 1 | average pool fully connected layer softmax |
Residual Structure 3 | Residual Structure 2 | ||
---|---|---|---|
n = 3 | n = 5 | n = 7 | |
n = 3 | 90.3% | 85.2% | 83.4% |
n = 5 | 86.4% | 85.6% | 84.8% |
n = 7 | 85.1% | 84.5% | 83.1% |
Residual Structure 3 | Residual Structure 2 | ||
---|---|---|---|
n = 3 | n = 5 | n = 7 | |
n = 3 | 86.50% | 83.16% | 80.41% |
n = 5 | 83.67% | 82.73% | 78.66% |
n = 7 | 81.45% | 80.19% | 77.27% |
Residual Structure 3 | Residual Structure 2 | ||
---|---|---|---|
n = 3 | n = 5 | n = 7 | |
n = 3 | 88.08% | 86.46% | 83.95% |
n = 5 | 86.72% | 85.28% | 83.12% |
n = 7 | 84.76% | 83.88% | 81.16% |
Method | Different Subject Recognition Rates/% | Average Recognition Rate/% | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
B01 | B02 | B03 | B04 | B05 | B06 | B07 | B08 | B09 | ||
ResNet18 | 81.1 | 67.3 | 65.6 | 97.1 | 93.5 | 87.4 | 82.9 | 92.4 | 87.4 | 83.8 |
DCN-ResNet18(2,3) | 88.7 | 79.1 | 77.9 | 98.7 | 95.9 | 92.6 | 89.7 | 95.8 | 94.6 | 90.3 |
DCN-ResNet18(3,4) | 86.8 | 77.2 | 75.1 | 98.1 | 94.8 | 91.7 | 88.5 | 94.5 | 93.4 | 88.9 |
DCN-ResNet18(1,4) | 85.9 | 75.4 | 72.9 | 97.5 | 93.9 | 90.1 | 87.3 | 92.8 | 91.4 | 87.4 |
DCN-ResNet18(1,2) | 82.6 | 71.8 | 69.9 | 97.3 | 93.6 | 88.7 | 83.7 | 92.6 | 90.3 | 85.6 |
Method | Different Subject Recognition Rates/% | Average Recognition Rate/% | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
B01 | B02 | B03 | B04 | B05 | B06 | B07 | B08 | B09 | ||
Chin(BCI IV 1st place in the competition) | 70.0 | 61.0 | 61.0 | 98.0 | 93.0 | 81.0 | 78.0 | 93.0 | 87.0 | 80.2 |
Gan(BCI IV 2st place in the competition) | 71.0 | 61.0 | 57.0 | 97.0 | 86.0 | 81.0 | 81.0 | 92.0 | 89.0 | 79.4 |
Coyle(BCI IV 3st place in the competition) | 60.0 | 56.0 | 56.0 | 89.0 | 79.0 | 75.0 | 69.0 | 93.0 | 81.0 | 73.1 |
KLD [25] | 73.25 | 63.27 | 60.43 | 97.72 | 91.94 | 80.48 | 85.78 | 93.48 | 85.31 | 81.3 |
CEMD-MSCNN [26] | 80.56 | 65.44 | 65.97 | 99.32 | 89.19 | 86.11 | 81.25 | 88.82 | 86.81 | 82.61 |
DBN [27] | 66.56 | 62.50 | 60.00 | 96.87 | 82.02 | 77.44 | 76.56 | 88.75 | 86.06 | 77.42 |
DCN-ResNet18 | 88.7 | 79.1 | 77.9 | 98.7 | 95.9 | 92.6 | 89.7 | 95.8 | 94.6 | 90.3 |
Accuracy (%) | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
Subject | A01 | A02 | A03 | A04 | A05 | A06 | A07 | A08 | A09 | Mean |
DFFN [28] | 85.40 | 69.30 | 90.29 | 71.07 | 65.41 | 69.45 | 88.18 | 86.46 | 93.54 | 79.90 |
EEGNET [29] | 83.33 | 63.80 | 88.76 | 62.41 | 58.72 | 58.51 | 84.81 | 82.12 | 78.30 | 73.42 |
CNN-LSTM [30] | 87.41 | 77.39 | 90.73 | 82.77 | 72.89 | 82.51 | 89.58 | 85.17 | 88.91 | 84.15 |
SS-MEMDBF [31] | 91.49 | 60.56 | 94.16 | 76.72 | 58.52 | 68.52 | 78.67 | 97.01 | 93.85 | 79.94 |
MSFBCNN [32] | 81.60 | 64.15 | 86.98 | 68.14 | 71.27 | 63.37 | 90.54 | 77.87 | 70.31 | 74.91 |
STSACNN [33] | 82.99 | 56.25 | 93.06 | 84.03 | 68.75 | 58.34 | 88.20 | 88.20 | 86.81 | 78.51 |
DMTLCNN [34] | 83.50 | 49.00 | 92.70 | 74.90 | 71.30 | 63.70 | 80.08 | 80.00 | 81.70 | 75.21 |
MCCNN [35] | 90.21 | 63.40 | 89.35 | 71.16 | 62.82 | 47.66 | 90.86 | 83.72 | 82.32 | 75.72 |
EEG-TCNet [36] | 85.77 | 65.02 | 94.51 | 64.91 | 75.36 | 61.40 | 87.36 | 83.76 | 78.03 | 77.35 |
DCN-ResNet18 | 91.83 | 80.15 | 91.46 | 82.79 | 76.60 | 82.76 | 90.77 | 89.94 | 92.28 | 86.50 |
Accuracy/% | ||||
---|---|---|---|---|
Subject | K3b | K6b | L1b | Mean |
EEGNET [29] | 96.25 | 70.52 | 80.21 | 82.33 |
MSFBCNN [32] | 96.39 | 78.54 | 82.29 | 85.74 |
Kernel-B2DDLPP [37] | 88.33 | 71.67 | 68.33 | 76.11 |
Multi-branch-3D [38] | 94.81 | 75.28 | 80.78 | 83.63 |
DCN-ResNet18 | 94.78 | 83.65 | 85.83 | 88.08 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Du, X.; Li, K.; Lv, Y.; Qiu, S. Motor Imaging EEG Signal Recognition of ResNet18 Network Based on Deformable Convolution. Electronics 2022, 11, 3674. https://doi.org/10.3390/electronics11223674
Du X, Li K, Lv Y, Qiu S. Motor Imaging EEG Signal Recognition of ResNet18 Network Based on Deformable Convolution. Electronics. 2022; 11(22):3674. https://doi.org/10.3390/electronics11223674
Chicago/Turabian StyleDu, Xiuli, Kai Li, Yana Lv, and Shaoming Qiu. 2022. "Motor Imaging EEG Signal Recognition of ResNet18 Network Based on Deformable Convolution" Electronics 11, no. 22: 3674. https://doi.org/10.3390/electronics11223674