3. Materials and Methods
The EEG signal processing and DNN-based WM load prediction model comprises five stages.
Figure 2 illustrates the steps in processing the EEG signal to predict WM load. During the N-back and block-tapping task sessions, subjects were instructed to wear an EEG cap connected to a 16-channel OpenBCI EEG data collection board, as shown in
Figure 3a. The N-back is the most versatile cognitive assessment, which requires information retention, continual updating, and interference resolution. As a result of its comprehensive nature, the N-back task has found extensive application in WM evaluation in human subjects [
29]. It keeps the participant’s WM system continuously engaged at its limit, thereby stimulating an increase in WM function, hence its widespread use for WM load prediction [
30]. In addition to N-back, we created the block-tapping task in which the subject was required to remember the sequence of appearance of blocks in a spatial grid. These two visuospatial WM tasks were a comprehensive set of tasks used in this work for WM load prediction. The EEG cap was adjusted to fit the subject’s head and ensure proper electrode placement. The task consisted of tapping a block in response to visual cues displayed on a computer screen. During the task, EEG signals were recorded and saved for offline analysis.
Figure 3b shows a subject wearing the EEG cap during the task, and
Figure 3c shows the subject performing the block-tapping task.
The raw EEG data were bandpass-filtered, and ICA was used to select the most significant electrode channels. By applying ICA to the EEG data, we efficiently reduced the number of electrodes while preserving signal quality. The methodology involves formulating the problem as X = AS, whitening the mixed EEG signals (X), estimating the unmixing matrix (W), obtaining independent components (S_est = W X_w), selecting a relevant subset of ICs, and reconstructing reduced EEG signals (X_red = A_red S_red).
From the reduced-electrode EEG data, we extracted IMFs to identify relevant features for predicting WM load.
Figure 4 shows an example of the raw EEG data (in red) and the corresponding IMFs (in green). The IMFs were obtained using EEMD, a time-frequency technique that decomposes nonstationary signals into a finite number of IMFs and a residual component. The IMFs represent the underlying oscillatory modes in the EEG signal and provides valuable information about the dynamic changes in brain activity related to cognitive processes. The EEMD methodology includes the following steps:
- i.
Add white noise, n(t), to the original signal, x(t), forming x_n(t) = x(t) + n(t)
- ii.
Perform EMD on x_n(t) to obtain intrinsic mode functions (IMFs), EMD(x_n(t)) = {c_1(t), c_2(t), …, c_N(t)}.
- iii.
Repeat steps 1 and 2 for M realizations of white noise, n_i(t), and compute ensemble mean for each IMF, C_j(t) = (1/M) ∑ C_i^j(t).
- iv.
Calculate residual, r(t) = x(t) − ∑ C_j(t), and determine stopping criteria.
In this study, a DNN architecture was developed to predict WM load using EEG signals.
Figure 5 shows the architecture of the DNN model used in this study. The model consisted of six hidden layers, a Relu activation input layer, and a Sigmoid activation output layer. The model was trained using the backpropagation algorithm with the Adam optimizer and a categorical cross-entropy loss function. The model’s performance was evaluated using Overall Accuracy (OA), specificity, sensitivity, F1 score, and Kappa metrics. In this study, the IMFs were used as features for predicting WM load using a DNN model.
A total of 18 subjects participated in the experiment. Each subject filled out the participant consent form before the commencement of the session. Each subject’s response on the block tapping and N-back WM tasks were recorded to calculate low and high WM load for the three trials, while the 16-channel EEG data were also recorded. The EEG data were preprocessed as per the workflow in
Figure 2. The IMF features were divided into training, testing, and validation sets before training the DNN. For each trial, a confusion matrix was computed to discern between two classes: Low WM load and high WM load. The format of the confusion matrix is shown in
Figure 6. The performance metrics were calculated using the formulas given in
Table 1, where TH is True High WM load, TL is True Low WM load, FH is False High WM load, and FL is False Low WM load.
The Kappa score tests the inter-reliability of the results, i.e., how much of the accuracy is obtained by chance. Po is the proportion of observed agreement, and Pe is the proportion of agreement expected by chance. The specificity score measures the proportion of true negative samples among all samples that have been predicted as negative by the DNN. Sensitivity is the proportion of true positive samples among all actual positive samples in the dataset.
Author Contributions
Conceptualization, S.S. and V.M.; methodology, S.S., A.R. and V.M.; software, V.M. and A.R.; validation, S.S. and V.M.; formal analysis, S.S. and V.M.; investigation, S.S. and V.M.; resources, S.S. and V.M.; data curation, V.M.; writing—original draft preparation, S.S., A.R. and V.M.; writing—review and editing, S.S., A.R. and V.M.; visualization, S.S., A.R. and V.M.; supervision, V.M.; project administration, V.M.; funding acquisition, V.M. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
The study was conducted in accordance with the Institutional Review Board of University of Puerto Rico at Mayaguez (protocol code 2022100012, 1 November 2022) for studies involving humans.
Informed Consent Statement
Informed consent was obtained from all subjects involved in the study.
Data Availability Statement
EEG data collected in this project is available on request.
Acknowledgments
The authors would like to thank the undergraduate students from the Electrical and Computer Engineering Department at the University of Puerto Rico at Mayaguez for their assistance in data collection and the recruited subjects for their participation in the experiments.
Conflicts of Interest
The authors declare no conflict of interest.
References
- Bosnes, O.; Almkvist, O.; Bosnes, I.; Stordal, E. Subjective working memory predicts objective memory in cognitively normal aging: A HUNT study. BMC Psychol. 2020, 8, 77. [Google Scholar] [CrossRef]
- Baddeley, A. The episodic buffer: A new component of working memory? Trends Cogn. Sci. 2000, 4, 417–423. [Google Scholar] [CrossRef] [PubMed]
- Briganti, G.; Le Moine, O. Artificial Intelligence in Medicine: Today and Tomorrow. Front. Med. 2020, 7, 27. [Google Scholar] [CrossRef] [PubMed]
- Cavedoni, S.; Chirico, A.; Pedroli, E.; Cipresso, P.; Riva, G. Digital Biomarkers for the Early Detection of Mild Cognitive Impairment: Artificial Intelligence Meets Virtual Reality. Front. Hum. Neurosci. 2020, 14, 245. [Google Scholar] [CrossRef] [PubMed]
- Jiang, Y.; Li, J.; Schmitt, F.A.; Jicha, G.A.; Munro, N.B.; Zhao, X.; Smith, C.D.; Kryscio, R.J.; Abner, E.L. Memory-Related Frontal Brainwaves Predict Transition to Mild Cognitive Impairment in Healthy Older Individuals Five Years before Diagnosis. J. Alzheimers. Dis. 2021, 79, 531–541. [Google Scholar] [CrossRef] [PubMed]
- Aurtenetxe, S.; García-Pacios, J.; del Río, D.; López, M.E.; Pineda-Pardo, J.A.; Marcos, A.; Losada, M.L.D.; López-Frutos, J.M.; Maestú, F. Interference impacts working memory in mild cognitive impairment. Front. Neurosci. 2016, 10, 443. [Google Scholar] [CrossRef]
- Engedal, K.; Barca, M.L.; Høgh, P.; Bo Andersen, B.; Dombernowsky, N.W.; Naik, M.; Gudmundsson, T.E.; Øksengaard, A.R.; Wahlund, L.O.; Snaedal, J. The Power of EEG to Predict Conversion from Mild Cognitive Impairment and Subjective Cognitive Decline to Dementia. Dement. Geriatr. Cogn. Disord. 2020, 49, 38–47. [Google Scholar] [CrossRef]
- Zarjam, P.; Epps, J.; Chen, F.; Lovell, N.H. Classification of working memory load using wavelet complexity features of EEG signals. In Proceedings of the Neural Information Processing: 19th International Conference, ICONIP 2012, Doha, Qatar, 12–15 November 2012; Volume 7664 LNCS, pp. 692–699. [Google Scholar]
- Wu, Z.; Huang, N.E. Ensemble empirical mode decomposition: A noise-assisted data analysis method. Adv. Adapt. Data Anal. 2009, 1, 1–41. [Google Scholar] [CrossRef]
- Romney, A.; Manian, V. Optimizing Seizure Prediction from Reduced Scalp EEG Channels Based on Spectral Features and MAML. Accept. IEEE Access 2021, 9, 164348–164357. [Google Scholar]
- Trinh, T.T.; Tsai, C.F.; Hsiao, Y.T.; Lee, C.Y.; Wu, C.T.; Liu, Y.H. Identifying Individuals With Mild Cognitive Impairment Using Working Memory-Induced Intra-Subject Variability of Resting-State EEGs. Front. Comput. Neurosci. 2021, 15, 700467. [Google Scholar] [CrossRef]
- Li, J.; Broster, L.S.; Jicha, G.A.; Munro, N.B.; Schmitt, F.A.; Abner, E.; Kryscio, R.; Smith, C.D.; Jiang, Y. A cognitive electrophysiological signature differentiates amnestic mild cognitive impairment from normal aging. Alzheimer’s Res. Ther. 2017, 9, 3. [Google Scholar] [CrossRef] [PubMed]
- Mamani, G.Q.; Fraga, F.J.; Tavares, G.; Johns, E.; Phillips, N.D. EEG-based biomarkers on working memory tasks for early diagnosis of Alzheimer’s disease and mild cognitive impairment. In Proceedings of the 2017 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT), Bethesda, MD, USA, 6–8 November 2017; Volume 2017, pp. 237–240. [Google Scholar] [CrossRef]
- Timothy, L.T.; Krishna, B.M.; Nair, U. Cross recurrence quantification analysis of mild cognitive impairment EEG under working memory condition. In Proceedings of the 2017 IEEE Life Sciences Conference (LSC), Sydney, Australia, 13–15 December 2017; Volume 2018, pp. 145–148. [Google Scholar] [CrossRef]
- Du, H.; Qi, H.; Wang, X.; Ramanath, R.; Snyder, W.E. Band selection using independent component analysis for hyperspectral image processing. In Proceedings of the 32nd Applied Imagery Pattern Recognition Workshop, Washington, DC, USA, 15–17 October 2003; Volume 2003, pp. 93–98. [Google Scholar] [CrossRef]
- Rejer, I.; Gorski, P. Benefits of ICA in the Case of a Few Channel EEG. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 25–29 August 2015; Volume 2015, pp. 7434–7437. [Google Scholar] [CrossRef]
- Fuchs, A.H. Ebbinghaus’s contributions to psychology after 1885. Am. J. Psychol. 1997, 110, 621–633. [Google Scholar] [CrossRef]
- Cowan, N. Chapter 20 What are the differences between long-term, short-term, and working memory? Prog. Brain Res. 2008, 169, 323–338. [Google Scholar] [CrossRef]
- Hawthorne, B.S.; Vella-brodrick, D.A.; Hattie, J. Well-Being as a Cognitive Load Reducing Agent: A Review of the Literature. In Frontiers in Education; Frontiers Media SA: Lausanne, Switzerland, 2019; Volume 4, p. 121. [Google Scholar] [CrossRef]
- Simon, S.S.; Tusch, E.S.; Holcomb, P.J.; Daffner, K.R. Increasing working memory load reduces processing of cross-modal task-irrelevant stimuli even after controlling for task difficulty and executive capacity. Front. Hum. Neurosci. 2016, 10, 13. [Google Scholar] [CrossRef] [PubMed]
- Sweller, J. Cognitive Load Theory. Psychol. Learn. Motiv. Adv. Res. Theory 2011, 55, 37–76. [Google Scholar] [CrossRef]
- Paas, F.; van Merriënboer, J.J.G. Cognitive-Load Theory: Methods to Manage Working Memory Load in the Learning of Complex Tasks. Curr. Dir. Psychol. Sci. 2020, 29, 394–398. [Google Scholar] [CrossRef]
- Puszta, A.; Pertich, Á.; Giricz, Z.; Nyujtó, D.; Bodosi, B.; Eördegh, G.; Nagy, A. Predicting Stimulus Modality and Working Memory Load During Visual- and Audiovisual-Acquired Equivalence Learning. Front. Hum. Neurosci. 2020, 14, 410. [Google Scholar] [CrossRef]
- Collins, A.G.E.; Ciullo, B.; Frank, M.J.; Badre, D. Working Memory Load Strengthens Reward Prediction Errors. J. Neurosci. 2017, 37, 4332. [Google Scholar] [CrossRef]
- Eryilmaz, H.; Dowling, K.F.; Hughes, D.E.; Rodriguez-Thompson, A.; Tanner, A.; Huntington, C.; Coon, W.G.; Roffman, J.L. Working memory load-dependent changes in cortical network connectivity estimated by machine learning. Neuroimage 2020, 217, 116895. [Google Scholar] [CrossRef]
- Zhang, Y.; Wang, C.; Wu, F.; Huang, K.; Yang, L.; Ji, L. Prediction of working memory ability based on EEG by functional data analysis. J. Neurosci. Methods 2020, 333, 108552. [Google Scholar] [CrossRef]
- Ben-Artzi, I.; Luria, R.; Shahar, N. Working memory capacity estimates moderate value learning for outcome-irrelevant features. Sci. Rep. 2022, 12, 19677. [Google Scholar] [CrossRef] [PubMed]
- Mirjalili, M.; Zomorrodi, R.; Daskalakis, Z.J.; Hill, S.; Rajji, T.K. Individualized real-time prediction of working memory performance by classifying electroencephalography signals. Int. J. Imaging Syst. Technol. 2022, 32, 575–589. [Google Scholar] [CrossRef]
- Carina Coulacoglou, D.H.S. Executive Function, Theory of Mind, and Adaptive Behavior. Psychom. Psychol. Assess. 2017, 91–130. [Google Scholar]
- Moustafa, A.A.; Gray, S.; Morris, A.; Matar, G.; Garg, A.; Helal, A.A.; Ayoub AE, A.; Hamza, E.A. Working-memory impairment in schizophrenia and schizotypal personality disorder. In Cognitive and Behavioral Dysfunction in Schizophrenia; Elsevier Academic Press: Amsterdam, The Netherlands, 2021. [Google Scholar]
- Ygierewicz, J.; Janik, R.A.; Podolak, I.T.; Drozd, A.; Malinowska, U.; Poziomska, M.; Wojciechowski, J.; Ogniewski, P.; Niedbalski, P.; Terczynska, I.; et al. Decoding working memory-related information from repeated psychophysiological EEG experiments using convolutional and contrastive neural networks. J. Neural Eng. 2022, 19, 046053. [Google Scholar] [CrossRef] [PubMed]
- Kwak, Y.; Song, W.J.; Kim, S.E. Classification of Working Memory Performance from EEG with Deep Artificial Neural Networks. In Proceedings of the 2019 7th International Winter Conference on Brain-Computer Interface (BCI), Gangwon, South Korea, 18–20 February 2019; pp. 6–8. [Google Scholar] [CrossRef]
- Wu, Y.; Qian, H.; Yang, X.; Chu, H.; Yan, C.; Gong, X. Classification of EEG signals during Working-Memory Maintenance based on Phase Space Reconstruction of Empirical Mode Decomposition. In Proceedings of the 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Chengdu, China, 17–19 October 2020; pp. 675–680. [Google Scholar] [CrossRef]
- Changoluisa, V.; Poch, C.; Campo, P.; Rodriguez, F.B. Predicting Working Memory Performance Based on Specific Individual EEG Spatiotemporal Features. bioRxiv 2022, 2022.05.06.490941. Available online: https://www.biorxiv.org/content/10.1101/2022.05.06.490941v1.abstract (accessed on 10 May 2023).
- Yoo, G.; Kim, H.; Hong, S. Prediction of Cognitive Load from Electroencephalography Signals Using Long Short-Term Memory Network. Bioengineering 2023, 10, 361. [Google Scholar] [CrossRef] [PubMed]
Figure 1.
Model for information processing in the brain.
Figure 2.
The proposed model for predicting WM load, outlining the various stages for achieving accurate working memory load predictions.
Figure 3.
(a) A 16-channel OpenBCI EEG data collection board, (b) EEG cap worn by subject, and (c) subject performing block tapping task.
Figure 4.
Raw EEG data (red) and Intrinsic Mode Functions (IMFs) (green).
Figure 5.
Deep Neural Network architecture.
Figure 6.
Confusion Matrix Format.
Figure 7.
Performance Metrics for the ICA+EEMD+DNN working memory load prediction method.
Figure 8.
Average power spectral density of intrinsic model functions per subject for resting state.
Figure 9.
Coefficient of variance of intrinsic model functions per subject for resting state.
Figure 10.
Average power spectral density of intrinsic model functions per subject for low WM load.
Figure 11.
Coefficient of variance of intrinsic mode functions per subject for low WM load.
Figure 12.
Average power spectral density of intrinsic model functions per subject for high WM load.
Figure 13.
Coefficient of variance of intrinsic model functions per subject for high WM load.
Figure 14.
Selected electrodes and corresponding brain region for each subject.
Figure 15.
Brain topological map for rest state and working memory task in a subject between 20 and 40 years.
Figure 16.
Brain topological map for rest state and working memory task in a subject with MCI.
Table 1.
Performance measures.
Performance Metrics | Formula |
---|
Sensitivity | |
Specificity | |
F1-Score | |
Overall Accuracy (OA) | |
Kappa (K) | |
Table 2.
Subject-specific WM load prediction performance of the ICA + EEMD + DNN method.
Group Tested | Subject | OA (%) | Kappa (%) | Sensitivity (%) | Specificity (%) | F1-Score (%) |
---|
| S01 | 100 | 100 | 97.24 | 93.87 | 100.00 |
| S02 | 100 | 100 | 100 | 100 | 100.00 |
20 to 40 years | S03 | 98.39 | 96.78 | 97.48 | 99.32 | 97.58 |
| S04 | 99.86 | 99.72 | 99.75 | 99.97 | 99.79 |
| S05 | 93.73 | 87.14 | 96.02 | 92.14 | 90.31 |
| S06 | 98.82 | 97.65 | 98.28 | 99.34 | 98.23 |
| S07 | 100 | 100 | 100 | 100 | 100.00 |
| S08 | 91.88 | 83.5 | 87.24 | 95.71 | 87.49 |
40 to 60 years | S09 | 99 | 98.01 | 98.74 | 99.29 | 98.50 |
| S10 | 90.52 | 80.85 | 91.1 | 91.67 | 85.41 |
| S11 | 99.81 | 99.61 | 100 | 99.61 | 99.71 |
| S12 | 97.75 | 95.49 | 97.71 | 97.79 | 96.61 |
>60 years | S13 | 98.03 | 96.01 | 97.75 | 98.37 | 97.01 |
| S14 | 93.61 | 87.12 | 90.92 | 97.13 | 90.25 |
| S15 | 98.36 | 96.67 | 98.04 | 98.23 | 97.51 |
Subjects with MCI | S16 | 99.66 | 87.14 | 99.53 | 99.76 | 99.64 |
| S17 | 99.68 | 99.32 | 99.43 | 99.83 | 99.63 |
| S18 | 96.23 | 92.21 | 97.88 | 95.74 | 96.8 |
Table 3.
Statistics on WM load prediction performance of the ICA+EEMD+DNN method for each tested group.
| 20 to 40 Years | 40 to 60 Years | Above 60 Years | Subjects with MCI |
---|
| 100.00 | 100.00 | 99.81 | 93.61 |
| 100.00 | 91.88 | 97.75 | 98.36 |
| 98.39 | 99.00 | 98.03 | 99.66 |
| 99.86 | 90.52 | | 99.68 |
| 93.73 | | | 96.23 |
| 98.82 | | | |
Average | 98.47 | 95.35 | 98.53 | 97.51 |
Std. dev. | 2.42 | 4.84 | 1.12 | 2.59 |
Table 4.
Results of analysis of variance (ANOVA) conducted on the overall prediction accuracy for each tested group.
Group Tested | p-Value | F-Critical Value | F-Value |
---|
20–40 years | 1.26 × 10−11 | 4.96 | 1134.56 |
40–60 years | 1.04 × 10−05 | 5.99 | 181.40 |
above 60 years | 0.01929 | 7.71 | 14.35 |
Subjects with MCI | 5.08 × 10−05 | 5.32 | 61.37 |
Table 5.
Comparison of proposed method with state of the art (SOA) on WM load prediction.
Year | Publication | Conference/Journal | Algorithm | Patient Specific Channels | Number of EEG Electrodes | Overall Accuracy |
---|
2012 | P. Zarjam, J. Epps, F. Chen, and N. H. Lovell [31] | 19th Intl. conf. neural information processing | Wavelet features + Artificial neural network | No | 17 | 83.94% |
2016 | A. Abrantes, E. Comitz, P. Mosaly, and L. Mazur [26] | Advances in intelligent systems and computing | Lasso regression and SVM | No | - | 69.7% |
2019 | Y. Kwak, W.J. Song, and S.E. Kim [32] | 7th Intl. conference on BCI | Power ratio feature + DNN | No | 30 | 61% |
2020 | Y. Wu, H. Qian et al. [33] | 13th Intl. congress image and signal processing, biomedical engineering | EMD features + SVM, kNN, RF classifiers | No | 128 | 73.6% |
2020 | Y. Zhang et al. [26] | Journal of neuroscience methods | Functional linear regression | No | 32 | 73% |
2022 | J. Zygierewicż et al. [34] | Journal of neural engineering | Convolutional neural network | No | 19 | 66.05% |
2020 | A. Puszta et al. [23] | Frontiers in Human neuroscience | Theta and alpha phase connectivity for WM load prediction | No | 64 | 75% |
2022 | V. Changoluisa et al. [34] | BioRxiv | Spatiotemporal features | yes | 128 | - |
2023 | G. Yoo et al. [35] | Bioengineering | Cognitive load prediction using LSTM | No | 2 to 10 | 87.1% |
Table 6.
Results of analysis of variance (ANOVA) conducted on the relationship between average power spectral density of intrinsic mode functions for low and high WM load in normal subjects and in subjects with MCI.
AVG PSD of IMFs between Low WM and High WM Load | p-Value | F-Critical Value | F-Value |
---|
Normal subjects | 0.0469 | 4.03 | 4.14 |
Subjects with MCI | 0.0145 | 5.32 | 9.65 |
| Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).