Classification of Known and Unknown Study Items in a Memory Task Using Single-Trial Event-Related Potentials and Convolutional Neural Networks
Abstract
:1. Introduction
2. Materials and Methods
2.1. Participants
2.2. Experimental Design
2.3. EEG Signal Recording
2.4. EEG Signal Preprocessing
2.5. ERP Processing
2.6. CNN Architecture Description
2.6.1. EEGNet
2.6.2. Deep Convolutional Neural Network (DeepConvNet)
2.6.3. Shallow Convolutional Neural Network (ShallowConvNet)
2.7. Pretraining of the Models
2.8. Model Assessment
3. Results
3.1. Pretraining Results
Model | Overall Accuracy | ROC AUC | Sensitivity | Specificity |
---|---|---|---|---|
EEGNET | 0.650 | 0.650 | 0.625 | 0.657 |
DeepConvNet | 0.550 | 0.57 | 0.107 | 0.943 |
ShallowConvNet | 0.610 | 0.57 | 0.530 | 0.606 |
3.2. Cross-Validation Results
4. Discussion
4.1. Comparison with Previous Studies
4.2. Limitations of the Study
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Schacter, D. Implicit memory: History and current status. J. Exp. Psychol. Learn. Mem. Cogn. 1987, 13, 501–518. [Google Scholar] [CrossRef]
- Cherry, K. Implicit Memory vs. Explicit Memory How the Different Types of Long-Term Memory Work. Available online: https://www.verywellmind.com/implicit-and-explicit-memory-2795346 (accessed on 24 May 2024).
- Yonelinas, A.P. The nature of recollection and familiarity: A review of 30 years of research. J. Mem. Lang. 2002, 46, 441–517. [Google Scholar] [CrossRef]
- Paller, K.A.; Voss, J.L.; Boehm, S.G. Validating neural correlates of familiarity. Trends Cogn. Sci. 2007, 11, 243–250. [Google Scholar] [CrossRef] [PubMed]
- Wagner, A.D.; Koutstaal, W.; Schacter, D.L. When encoding yields remembering: Insights from event-related neuroimaging. Philosophical Transactions of the Royal Society of London. Ser. B Biol. Sci. 1999, 354, 1307–1324. [Google Scholar] [CrossRef] [PubMed]
- Friedman, D.; Johnson, R. Event-related potential (ERP) studies of memory encoding and retrieval: A selective review. Microsc. Res. Tech. 2000, 51, 6–28. [Google Scholar] [CrossRef]
- Curran, T. Effects of attention and confidence on the hypothesized ERP correlates of recollection and familiarity. Neuropsychologia 2004, 42, 1088–1106. [Google Scholar] [CrossRef]
- Diana, R.A.; van den Boom, W.; Yonelinas, A.P.; Ranganath, C. ERP correlates of source memory: Unitized source information increases familiarity-based retrieval. Brain Res. 2011, 1367, 278–286. [Google Scholar] [CrossRef]
- Strózak, P.; Abedzadeh, D.; Curran, T. Separating the FN400 and N400 potentials across recognition memory experiments. Brain Res. 2016, 1635, 41–60. [Google Scholar] [CrossRef]
- Leynes, P.; Upadhyay, T. Context dissociations of the FN400 and N400 are evidence for recognition based on relative or absolute familiarity. Brain Cogn. 2022, 162, 105903. [Google Scholar] [CrossRef]
- Addante, R.J.; Ranganath, C.; Yonelinas, A.P. Examining ERP correlates of recognition memory: Evidence of accurate source recognition without recollection. NeuroImage 2012, 62, 439–450. [Google Scholar] [CrossRef]
- Dimsdale-Zucker, H.R.; Maciejewska, K.; Kim, K.; Yonelinas, A.P.; Ranganath, C. Individual differences in behavioral and electrophysiological signatures of familiarity- and recollection-based recognition memory. Neuropsychologia 2022, 173, 108287. [Google Scholar] [CrossRef] [PubMed]
- Fukuda, K.; Woodman, G.F. Predicting and improving recognition memory using multiple electrophysiological signals in real time. Psychol. Sci. 2015, 26, 1026–1037. [Google Scholar] [CrossRef]
- Khurana, V.; Kumar, P.; Saini, R.; Roy, P.P. EEG based word familiarity using features and frequency bands combination. Cogn. Syst. Res. 2018, 49, 33–48. [Google Scholar] [CrossRef]
- Luck, S.J. An Introduction to the Event-Related Potential Technique, 2nd ed.; The MIT Press: Cambridge, MA, USA, 2014. [Google Scholar]
- Luck, S.J. Virtual ERP Boot Camp: Introduction to ERPs. Available online: https://courses.erpinfo.org/courses/take/Intro-to-ERPs/texts/14727771-chapter-1-overview (accessed on 15 August 2023).
- Wilding, E.; Ranganath, C. Electrophysiological Correlates of Episodic Memory Processes. In The Oxford Handbook of Event-Related Potential Components; Kappenman, E., Luck, S.J., Eds.; Oxford Handbooks; Oxford University Press: New York, NY, USA, 2012. [Google Scholar]
- Bandt, C.; Weymar, M.; Samaga, D.; Hamm, A.O. A simple classification tool for single-trial analysis of ERP components. Psychophysiology 2009, 46, 747–757. [Google Scholar] [CrossRef]
- Stober, S.; Cameron, D.J.; Grahn, J.A. Using Convolutional Neural Networks to Recognize Rhythm Stimuli from Electroencephalography Recordings. In Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS’14), Montreal, QC, Canada, 8 December 2014. [Google Scholar]
- Oh, S.L.; Vicnesh, J.; Ciaccio, E.J.; Yuvaraj, R.; Acharya, U.R. Deep convolutional neural network model for automated diagnosis of Schizophrenia using EEG signals. Appl. Sci. 2019, 9, 2870. [Google Scholar] [CrossRef]
- Borra, D.; Magosso, E. Deep learning-based EEG analysis: Investigating P3 ERP components. J. Integr. Neurosci. 2021, 20, 791–811. [Google Scholar] [CrossRef] [PubMed]
- Komolovaitė, D.; Maskeliūnas, R.; Damaševičius, R. Deep convolutional neural network-based visual stimuli classification using electroencephalography signals of healthy and Alzheimer’s disease subjects. Life 2022, 12, 374. [Google Scholar] [CrossRef]
- Craik, A.; He, Y.; Contreras-Vidal, J.L. Deep learning for electroencephalogram (EEG) classification tasks: A review. J. Neural. Eng. 2019, 16, 031001. [Google Scholar] [CrossRef]
- Depuydt, E.; Criel, Y.; de Letter, M.; van Mierlo, P. Single-trial ERP quantification using neural networks. Brain Topogr. 2023, 36, 767–790. [Google Scholar] [CrossRef]
- Alimardani, M.; Kaba, M. Deep Learning for Neuromarketing: Classification of User Preference using EEG Signals. In Proceedings of the 12th Augmented Human International Conference (AH2021), Geneva, Switzerland, 27 May 2021. [Google Scholar]
- Atilla, F.; Alimardani, M. EEG-based Classification of Drivers Attention using Convolutional Neural Network. In Proceedings of the 2021 IEEE International Conference on Human-Machine Systems (ICHMS 2021), Magdeburg, Germany, 8 September 2021. [Google Scholar]
- jsPsych. Available online: https://www.jspsych.org/7.3/ (accessed on 24 February 2023).
- de Leeuw, J.R. jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behav. Res. Methods 2015, 47, 1–12. [Google Scholar] [CrossRef]
- Massol, S.; Caron, C.; Franck, N.; Demily, C.; Chainay, H. Emotional modulation of episodic memory in school-age children and adults: An event-related potential study. Brain Sci. 2021, 11, 1598. [Google Scholar] [CrossRef]
- Guo, A.; Yang, W.; Yang, X.; Lin, J.; Li, Z.; Ren, Y.; Yang, J.; Wu, J. Audiovisual n-back training alters the neural processes of working memory and audiovisual integration: Evidence of changes in ERPs. Brain Sci. 2023, 13, 992. [Google Scholar] [CrossRef] [PubMed]
- Delorme, A.; Makeig, S. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 2004, 134, 9–21. [Google Scholar] [CrossRef] [PubMed]
- Calbi, M.; Siri, F.; Heimann, K.; Barratt, D.; Gallese, V.; Kolesnikov, A.; Umiltà, M.A. How context influences the interpretation of facial expressions: A source localization high-density EEG study on the “Kuleshov effect.”. Sci. Rep. 2019, 9, 2107. [Google Scholar] [CrossRef]
- Herzmann, G.; Curran, T. Experts’ memory: An ERP study of perceptual expertise effects on encoding and recognition. Mem. Cogn. 2011, 39, 412–432. [Google Scholar] [CrossRef] [PubMed]
- Bailey, K.; Chapman, P. When can we choose to forget? An ERP study into item-method directed forgetting of emotional words. Brain Cogn. 2012, 78, 133–147. [Google Scholar] [CrossRef]
- Lopez-Calderon, J.; Luck, S.J. ERPLAB: An open-source toolbox for the analysis of event-related potentials. Front. Hum. Neurosci. 2014, 8, 213. [Google Scholar] [CrossRef]
- Luck, S.J.; Stewart, A.X.; Simmons, A.M.; Rhemtulla, M. Standardized measurement error: A universal metric of data quality for averaged event-related potentials. Psychophysiology 2021, 58, e13793. [Google Scholar] [CrossRef]
- Schirrmeister, R.T.; Springenberg, J.T.; Fiederer, L.D.J.; Glasstetter, M.; Eggensperger, K.; Tangermann, M.; Hutter, F.; Burgard, W.; Ball, T. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum. Brain Mapp. 2017, 38, 5391–5420. [Google Scholar] [CrossRef]
- Lawhern, V.J.; Solon, A.J.; Waytowich, N.R.; Gordon, S.M.; Hung, C.P.; Lance, B.J. EEGNet: A compact convolutional neural network for EEG-based brain-computer interfaces. J. Neural Eng. 2018, 15, 056013. [Google Scholar] [CrossRef]
- Lawhern, V. Army Research Laboratory (ARL) EEGModels. Available online: https://github.com/vlawhern/arl-eegmodels (accessed on 24 February 2024).
- Keras Documentation. Layer Activation Functions. Available online: https://keras.io/api/layers/activations/ (accessed on 24 February 2024).
- Rakhmatulin, I.; Dao, M.S.; Nassibi, A.; Mandic, D. Exploring convolutional neural network architectures for EEG feature extraction. Sensors 2024, 24, 76. [Google Scholar] [CrossRef]
- Zhao, H.; Yang, Y.; Karlsson, P.; McEwan, A. Can recurrent neural network enhanced EEGNet improve the accuracy of ERP classification task? An Exploration and a Discussion. Health Technol. 2020, 10, 979–995. [Google Scholar] [CrossRef]
- Zhang, H.; Wang, Z.; Yu, Y.; Yin, H.; Chen, C.; Wang, H. An improved EEGNet for single-trial EEG classification in rapid serial visual presentation task. Brain Sci. Adv. 2022, 8, 111–126. [Google Scholar] [CrossRef]
- Xia, M.; Zhao, X.; Deng, R.; Lu, Z.; Cao, J. EEGNet Classification of sleep EEG for individual specialization based on data augmentation. Cogn. Neurodyn. 2024. [CrossRef]
- Fu, R.; Wang, Z.; Wang, S.; Xu, X.; Chen, J.; Wen, G. EEGNet-MSD: A sparse convolutional neural network for efficient EEG-based intent decoding. IEEE Sens. J. 2023, 23, 19684–19691. [Google Scholar] [CrossRef]
- Altan, G.; Yayık, A.; Kutlu, Y. Deep learning with ConvNet predicts imagery tasks through EEG. Neural Process. Lett. 2021, 53, 2917–2932. [Google Scholar] [CrossRef]
- Jo, S.Y.; Jeong, J.W. Prediction of Visual Memorability with EEG Signals using Deep Neural Networks. In Proceedings of the International Winter Workshop on Brain-Computer Interface (BCI), Gangwon, Republic of Korea, 26 February 2020. [Google Scholar]
Hyperparameter | Values | ||
---|---|---|---|
EEGNet | DeepConvNet | ShallowConvNet | |
Kernel length | 64 | 64 | 64 |
F1 | 8 | N/A | N/A |
D | 2 | N/A | N/A |
F2 | 16 | N/A | N/A |
Optimizer | ADAM | ADAM | ADAM |
Loss Function | Binary Crossentropy | Binary Crossentropy | Binary Crossentropy |
Learning rate | 1 × 10−4 | 1 × 10−4 | 1 × 10−4 |
Batch Size | 64 | 16 | 32 |
Regularization of L1 | 1 × 10−5 | N/A | 1 × 10−5 |
Regularization of L2 | 1 × 10−5 | N/A | 1 × 10−5 |
Dropout Rate | 0.2 | 0.2 | 0.2 |
Epochs | 500 | 500 | 500 |
(a) EEGNet | (b) DeepConvNet | (c) ShallowConvNet | ||||||
---|---|---|---|---|---|---|---|---|
Layer (Type) | Output Shape | Param # | Layer (Type) | Output Shape | Param # | Layer (Type) | Output Shape | Param # |
input_3 (InputLayer) | [(None, 20, 128, 1)] | 0 | input_4 (InputLayer) | [(None, 20, 128, 1)] | 0 | input_1 (InputLayer) | [(None, 20, 128, 1)] | 0 |
conv2d_15 (Conv2D) | (None, 20, 124, 25) | 150 | ||||||
conv2d_2 (Conv2D) | (None, 20, 128, 8) | 512 | conv2d_16 (Conv2D) | (None, 1, 124, 25) | 12,525 | |||
batch_normalization_12 (BatchNormalization) | (None, 1, 124, 25) | 100 | conv2d (Conv2D) | (None, 20, 116, 40) | 560 | |||
batch_normalization_6 (BatchNormalization) | (None, 20, 128, 8) | 32 | activation_15 (Activation) | (None, 1, 124, 25) | 0 | |||
max_pooling2d_12 (MaxPooling2D) | (None, 1, 62, 25) | 0 | ||||||
depthwise_conv2d_2 (DepthwiseConv2D) | (None, 1, 128, 16) | 320 | dropout_12 (Dropout) | (None, 1, 62, 25) | 0 | conv2d_1 (Conv2D) | (None, 1, 116, 40) | 32,000 |
conv2d_17 (Conv2D) | (None, 1, 58, 50) | 6300 | ||||||
batch_normalization_7 (BatchNormalization) | (None, 1, 128, 16) | 64 | batch_normalization_13 (BatchNormalization) | (None, 1, 58, 50) | 200 | |||
activation_16 (Activation) | (None, 1, 58, 50) | 0 | batch_normalization (Batch Normalization) | (None, 1, 116, 40) | 160 | |||
activation_4 (Activation) | (None, 1, 128, 16) | 0 | max_pooling2d_13 (MaxPooling2D) | (None, 1, 29, 50) | 0 | |||
dropout_13 (Dropout) | (None, 1, 29, 50) | 0 | ||||||
average_pooling2d_4 (AveragePooling2D) | (None, 1, 32, 16) | 0 | conv2d_18 (Conv2D) | (None, 1, 25, 100) | 25,100 | activation (Activation) | (None, 1, 116, 40) | 0 |
batch_normalization_14 (BatchNormalization) | (None, 1, 25, 100) | 400 | ||||||
spatial_dropout2d_4 (SpatialDropout2D) | (None, 1, 32, 16) | 0 | activation_17 (Activation) | (None, 1, 25, 100) | 0 | |||
max_pooling2d_14 (MaxPooling2D) | (None, 1, 12, 100) | 0 | average_pooling2d (AveragePooling2D) | (None, 1, 12, 40) | 0 | |||
separable_conv2d_2 (SeparableConv2D) | (None, 1, 32, 16) | 512 | dropout_14 (Dropout) | (None, 1, 12, 100) | 0 | |||
conv2d_19 (Conv2D) | (None, 1, 8, 200) | 100,200 | ||||||
batch_normalization_8 (BatchNormalization) | (None, 1, 32, 16) | 64 | batch_normalization_15 (BatchNormalization) | (None, 1, 8, 200) | 800 | activation_1 (Activation) | (None, 1, 12, 40) | 0 |
activation_5 (Activation) | (None, 1, 32, 16) | 0 | activation_18 (Activation) | (None, 1, 8, 200) | 0 | |||
average_pooling2d_5 (AveragePooling2D) | (None, 1, 4, 16) | 0 | max_pooling2d_15 (MaxPooling2D) | (None, 1, 4, 200) | 0 | |||
spatial_dropout2d_5 (SpatialDropout2D) | (None, 1, 4, 16) | 0 | dropout_15 (Dropout) | (None, 1, 4, 200) | 0 | dropout (Dropout) | (None, 1, 12, 40) | 0 |
flatten (Flatten) | (None, 64) | 0 | flatten_3 (Flatten) | (None, 800) | 0 | flatten (Flatten) | (None, 480) | 0 |
dense (Dense) | (None, 2) | 130 | dense_3 (Dense) | (None, 2) | 1602 | dense (Dense) | (None, 2) | 962 |
sigmoid (Activation) | (None, 2) | 0 | activation_19 (Activation) | (None, 2) | 0 | activation_2 (Activation) | (None, 2) | 0 |
Total params: 1634 (6.38 KB) | Total params: 147,377 (575.69 KB) | Total params: 33,682 (131.57 KB) | ||||||
Trainable params: 1554 (6.07 KB) | Trainable params: 146,627 (572.76 KB) | Trainable params: 33,602 (131.26 KB) | ||||||
Non-trainable params: 80 (320.00 Byte) | Non-trainable params: 750 (2.93 KB) | Non-trainable params: 80 (320.00 Byte) |
Model | Fold | Overall Accuracy | ROC-AUC | Sensitivity | Specificity |
---|---|---|---|---|---|
EEGNET | 0 | 0.689 | 0.624 | 0.732 | 0.394 |
1 | 0.674 | 0.589 | 0.750 | 0.309 | |
2 | 0.799 | 0.539 | 0.857 | 0.120 | |
3 | 0.709 | 0.615 | 0.696 | 0.451 | |
4 | 0.716 | 0.537 | 0.929 | 0.046 | |
5 | 0.739 | 0.522 | 0.857 | 0.126 | |
6 | 0.701 | 0.552 | 0.875 | 0.126 | |
7 | 0.649 | 0.562 | 0.875 | 0.189 | |
8 | 0.627 | 0.524 | 0.893 | 0.097 | |
9 | 0.701 | 0.594 | 0.750 | 0.343 | |
Mean | 0.700 | 0.566 | 0.821 | 0.220 | |
SD | 0.047 | 0.037 | 0.081 | 0.142 | |
DeepConvNet | 0 | 0.978 | 0.562 | 0.054 | 0.971 |
1 | 0.993 | 0.583 | 0.286 | 0.869 | |
2 | 0.970 | 0.575 | 0.143 | 0.954 | |
3 | 0.985 | 0.577 | 0.107 | 0.966 | |
4 | 0.985 | 0.543 | 0.054 | 0.966 | |
5 | 0.993 | 0.583 | 0.143 | 0.937 | |
6 | 0.978 | 0.562 | 0.125 | 0.960 | |
7 | 0.985 | 0.559 | 0.250 | 0.869 | |
8 | 0.970 | 0.537 | 0.000 | 0.977 | |
9 | 0.978 | 0.586 | 0.018 | 0.971 | |
Mean | 0.981 | 0.567 | 0.118 | 0.944 | |
SD | 0.008 | 0.017 | 0.094 | 0.041 | |
ShallowConvNet | 0 | 0.815 | 0.598 | 0.732 | 0.429 |
1 | 0.956 | 0.612 | 0.839 | 0.354 | |
2 | 0.978 | 0.612 | 0.839 | 0.320 | |
3 | 0.955 | 0.610 | 0.821 | 0.343 | |
4 | 0.963 | 0.615 | 0.839 | 0.320 | |
5 | 0.985 | 0.619 | 0.875 | 0.223 | |
6 | 0.993 | 0.602 | 0.804 | 0.314 | |
7 | 0.978 | 0.610 | 0.857 | 0.257 | |
8 | 0.993 | 0.606 | 0.875 | 0.251 | |
9 | 0.978 | 0.609 | 0.857 | 0.234 | |
Mean | 0.815 | 0.598 | 0.732 | 0.429 | |
SD | 0.956 | 0.612 | 0.839 | 0.354 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Delgado-Munoz, J.; Matsunaka, R.; Hiraki, K. Classification of Known and Unknown Study Items in a Memory Task Using Single-Trial Event-Related Potentials and Convolutional Neural Networks. Brain Sci. 2024, 14, 860. https://doi.org/10.3390/brainsci14090860
Delgado-Munoz J, Matsunaka R, Hiraki K. Classification of Known and Unknown Study Items in a Memory Task Using Single-Trial Event-Related Potentials and Convolutional Neural Networks. Brain Sciences. 2024; 14(9):860. https://doi.org/10.3390/brainsci14090860
Chicago/Turabian StyleDelgado-Munoz, Jorge, Reiko Matsunaka, and Kazuo Hiraki. 2024. "Classification of Known and Unknown Study Items in a Memory Task Using Single-Trial Event-Related Potentials and Convolutional Neural Networks" Brain Sciences 14, no. 9: 860. https://doi.org/10.3390/brainsci14090860