Development of RelaxQuest: A Serious EEG-Controlled Game Designed to Promote Relaxation and Self-Regulation with a Potential Focus on ADHD Intervention
Abstract
:1. Introduction
2. Materials and Methods
2.1. Signal Processing Applied to the Classification of Relaxation State
- Capture of signals from channels P7, O1, O2, and P8.
- EEG signal summation.
- Baseline correction.
- Signal segmentation.
- Noise reduction using wavelets.
- Filtering signals with a Butterworth filter (8–13 Hz).
- Application of the Fast Fourier Transform (FFT).
- Numerical integration using the trapezoidal rule.
- Thresholding.
- represents the sum of all signals at the sample.
- denotes the value of the -th signal at the sample, for .
- is the total number of signals.
- is the total number of samples.
- is the size of each segment (window).
- is the shift between windows, which can be equal to (non-overlapping windows) or less (overlapping windows).
- is the complex value representing the frequency component of the transformed signal.
- is the value of the signal in the time domain at sample .
- is the total number of samples in the signal.
- is the index of the frequency component.
- is the imaginary unit.
- represents a specific segment of the original signal.
- is the result of the FFT applied to segment , which takes values corresponding to the different segments, specifically , where represents the total number of segments analyzed.
- is the sum that approximates the area under the curve in the interval using a series of discrete points .
- is the step size or distance between two consecutive points on the x-axis, represented as , where is the lower limit of the interval, is the upper limit of the interval, and is the number of subintervals.
- Level 0 (active state):
- Level 1 (relaxed state 1):
- Level 2 (relaxed state 2):
2.2. Signal Processing Applied to Blink Classification
- Capture of signals from channels Fp1 and Fp2.
- EEG signal summation
- Baseline correction.
- Signal segmentation.
- Filtering signals with a Butterworth filter (4–20 Hz)
- Application of the Fast Fourier Transform (FFT).
- Numerical integration using the trapezoidal rule.
- Thresholding.
- Null state (no blinking):
- Active state (blinking):
2.3. Training Phase
2.4. Execution of the Virtual Game RelaxQuest
2.5. Interactive Physical Development
2.6. Subjects
3. Results
3.1. Signal Processing
3.2. Alpha Wave Detection Efficiency
3.3. Voluntary Blink Detection Efficiency
3.4. RelaxQuest Evaluation
4. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Zheng, Y.; Li, R.; Li, S.; Zhang, Y.; Yang, S.; Ning, H. A Review on Serious Games for ADHD. arXiv 2021, arXiv:2105.02970v1. [Google Scholar]
- Dipeolu, A.O. College Students with Adhd: Prescriptive Concepts for Best Practices in Career Development. J. Career Dev. 2011, 38, 408–427. [Google Scholar] [CrossRef]
- Sújar, A.; Martín-Moratinos, M.; Rodrigo-Yanguas, M.; Bella-Fernández, M.; González-Tardón, C.; Delgado-Gómez, D.; Blasco-Fontecilla, H. Developing Serious Video Games to Treat Attention Deficit Hyperactivity Disorder: Tutorial Guide. JMIR Serious Games 2022, 10, e33884. [Google Scholar] [CrossRef] [PubMed]
- De Luca, V.; Schena, A.; Covino, A.; Di Bitonto, P.; Potenza, A.; Barba, M.C.; D’Errico, G.; De Paolis, L.T. Serious Games for the Treatment of Children with ADHD: The BRAVO Project. Inf. Syst. Front. 2024, 1, 1–23. [Google Scholar] [CrossRef]
- Alqithami, S. Modeling an AR Serious Game to Increase Attention of ADHD Patients. In Proceedings of the 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC), Madrid, Spain, 13–17 July 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 1379–1384. [Google Scholar]
- Babusiak, B.; Hostovecky, M.; Smondrk, M.; Huraj, L. Spectral Analysis of Electroencephalographic Data in Serious Games. Appl. Sci. 2021, 11, 2480. [Google Scholar] [CrossRef]
- Sourina, O.; Liu, Y. EEG-Based Serious Games. In Subconscious Learning via Games and Social Media; Springer: Berlin/Heidelberg, Germany, 2015; pp. 135–152. [Google Scholar]
- Mondéjar, T.; Hervás, R.; Johnson, E.; Gutiérrez-López-Franca, C.; Latorre, J.M. Analyzing EEG Waves to Support the Design of Serious Games for Cognitive Training. J. Ambient. Intell. Humaniz. Comput. 2019, 10, 2161–2174. [Google Scholar] [CrossRef]
- da Costa, N.M.C.; Bicho, E.; Ferreira, F.; Vilhena, E.; Dias, N.S. A Multivariate Randomized Controlled Experiment about the Effects of Mindfulness Priming on Eeg Neurofeedback Self-Regulation Serious Games. Appl. Sci. 2021, 11, 7725. [Google Scholar] [CrossRef]
- Kim, H.; Miyakoshi, M.; Kim, Y.; Stapornchaisit, S.; Yoshimura, N.; Koike, Y. Electroencephalography Reflects User Satisfaction in Controlling Robot Hand through Electromyographic Signals. Sensors 2023, 23, 277. [Google Scholar] [CrossRef]
- Vasiljevic, G.A.M.; de Miranda, L.C. Brain–Computer Interface Games Based on Consumer-Grade EEG Devices: A Systematic Literature Review. Int. J. Hum. Comput. Interact. 2020, 36, 105–142. [Google Scholar] [CrossRef]
- Padfield, N.; Camilleri, K.; Camilleri, T.; Fabri, S.; Bugeja, M. A Comprehensive Review of Endogenous EEG-Based BCIs for Dynamic Device Control. Sensors 2022, 22, 5802. [Google Scholar] [CrossRef]
- Song, Z.; Fang, T.; Ma, J.; Zhang, Y.; Le, S.; Zhan, G.; Zhang, X.; Wang, S.; Li, H.; Lin, Y.; et al. Evaluation and Diagnosis of Brain Diseases Based on Non-Invasive BCI. In Proceedings of the 2021 9th International Winter Conference on Brain-Computer Interface (BCI), Gangwon, Republic of Korea, 22–24 February 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 1–6. [Google Scholar]
- Angulo Medina, A.S.; Aguilar Bonilla, M.I.; Rodríguez Giraldo, I.D.; Montenegro Palacios, J.F.; Cáceres Gutiérrez, D.A.; Liscano, Y. Electroencephalography-Based Brain-Computer Interfaces in Rehabilitation: A Bibliometric Analysis (2013–2023). Sensors 2024, 24, 7125. [Google Scholar] [CrossRef] [PubMed]
- Zapała, D.; Zabielska-Mendyk, E.; Augustynowicz, P.; Cudo, A.; Jaśkiewicz, M.; Szewczyk, M.; Francuz, P. The Effects of Handedness on Sensorimotor Rhythm Desynchronization and Motor-Imagery BCI Control. Sci. Rep. 2020, 10, 2087. [Google Scholar] [CrossRef] [PubMed]
- Perez Vidal, A.F.; Oliver Salazar, M.A.; Salas Lopez, G. Development of a Brain-Computer Interface Based on Visual Stimuli for the Movement of a Robot Joints. IEEE Lat. Am. Trans. 2016, 14, 477–484. [Google Scholar] [CrossRef]
- Pérez-Vidal, A.F.; Garcia-Beltran, C.D.; Martínez-Sibaja, A.; Posada-Gómez, R. Use of the Stockwell Transform in the Detection of P300 Evoked Potentials with Low-Cost Brain Sensors. Sensors 2018, 18, 1483. [Google Scholar] [CrossRef]
- Aggensteiner, P.M.; Brandeis, D.; Millenet, S.; Hohmann, S.; Ruckes, C.; Beuth, S.; Albrecht, B.; Schmitt, G.; Schermuly, S.; Wörz, S.; et al. Slow Cortical Potentials Neurofeedback in Children with ADHD: Comorbidity, Self-Regulation and Clinical Outcomes 6 Months after Treatment in a Multicenter Randomized Controlled Trial. Eur. Child. Adolesc. Psychiatry 2019, 28, 1087–1095. [Google Scholar] [CrossRef]
- Newson, J.J.; Thiagarajan, T.C. EEG Frequency Bands in Psychiatric Disorders: A Review of Resting State Studies. Front. Hum. Neurosci. 2019, 12, 521. [Google Scholar] [CrossRef]
- Li, Y.; Long, J.; Yu, T.; Yu, Z.; Wang, C.; Zhang, H.; Guan, C. An EEG-Based BCI System for 2-D Cursor Control by Combining Mu/Beta Rhythm and P300 Potential. IEEE Trans. Biomed. Eng. 2010, 57, 2495–2505. [Google Scholar] [CrossRef]
- Diaz-Romero, D.J.; Rincon, A.M.R.; Miguel-Cruz, A.; Yee, N.; Stroulia, E. Recognizing Emotional States with Wearables While Playing a Serious Game. IEEE Trans. Instrum. Meas. 2021, 70, 2506312. [Google Scholar] [CrossRef]
- Francillette, Y.; Boucher, E.; Bouchard, B.; Bouchard, K.; Gaboury, S. Serious Games for People with Mental Disorders: State of the Art of Practices to Maintain Engagement and Accessibility. Entertain. Comput. 2021, 37, 100396. [Google Scholar] [CrossRef]
- Jeunet, C.; Glize, B.; McGonigal, A.; Batail, J.M.; Micoulaud-Franchi, J.A. Using EEG-Based Brain Computer Interface and Neurofeedback Targeting Sensorimotor Rhythms to Improve Motor Skills: Theoretical Background, Applications and Prospects. Neurophysiol. Clin. 2019, 49, 125–136. [Google Scholar] [CrossRef]
- Voss, P.; Thomas, M.E.; Cisneros-Franco, J.M.; de Villers-Sidani, É. Dynamic Brains and the Changing Rules of Neuroplasticity: Implications for Learning and Recovery. Front. Psychol. 2017, 8, 274878. [Google Scholar] [CrossRef] [PubMed]
- Li, L. The Differences Among Eyes-Closed, Eyes-Open and Attention States: An EEG Study. In Proceedings of the 2010 International Conference on Computational Intelligence and Software Engineering, Wuhan, China, 10–12 December 2010; IEEE: Piscataway, NJ, USA, 2010; pp. 1–4. [Google Scholar]
- Gonzalez, R.C.; Woods, R.E. Digital Image Processing, 4th ed.; Pearson: London, UK, 2018; ISBN 9780133356724. [Google Scholar]
- Pactitis, S.A. Active Filters: Theory and Design; CRC Press: Boca Raton, FL, USA, 2011; ISBN 9781420054774. [Google Scholar]
- Abd-Alrazaq, A.; Al-Jafar, E.; Alajlani, M.; Toro, C.; Alhuwail, D.; Ahmed, A.; Reagu, S.M.; Al-Shorbaji, N.; Househ, M. The Effectiveness of Serious Games for Alleviating Depression: Systematic Review and Meta-Analysis. JMIR Serious Games 2022, 10, e32331. [Google Scholar] [CrossRef] [PubMed]
- Hassan, A.; Pinkwart, N.; Shafi, M. Serious Games to Improve Social and Emotional Intelligence in Children with Autism. Entertain. Comput. 2021, 38, 100417. [Google Scholar] [CrossRef]
- Ning, H.; Li, R.; Ye, X.; Zhang, Y.; Liu, L. A Review on Serious Games for Dementia Care in Ageing Societies. IEEE J. Transl. Eng. Health Med. 2020, 8, 1400411. [Google Scholar] [CrossRef]
- Vieira, C.; Ferreira da Silva Pais-Vieira, C.; Novais, J.; Perrotta, A. Serious Game Design and Clinical Improvement in Physical Rehabilitation: Systematic Review. JMIR Serious Games 2021, 9, e20066. [Google Scholar] [CrossRef]
- Alchalabi, A.E.; Shirmohammadi, S.; Eddin, A.N.; Elsharnouby, M. FOCUS: Detecting ADHD Patients by an EEG-Based Serious Game. IEEE Trans. Instrum. Meas. 2018, 67, 1512–1520. [Google Scholar] [CrossRef]
- Lussier-Desrochers, D.; Massé, L.; Simonato, I.; Lachapelle, Y.; Godin-Tremblay, V.; Lemieux, A. Evaluation of the Effect of a Serious Game on the Performance of Daily Routines by Autistic and ADHD Children. Adv. Neurodev. Disord. 2023, 7, 566–578. [Google Scholar] [CrossRef]
- Giannaraki, M.; Moumoutzis, N.; Papatzanis, Y.; Kourkoutas, E.; Mania, K. A 3D Rhythm-Based Serious Game for Collaboration Improvement of Children with Attention Deficit Hyperactivity Disorder (ADHD). In Proceedings of the 2021 IEEE Global Engineering Education Conference (EDUCON), Vienna, Austria, 21–23 April 2021; pp. 1217–1225. [Google Scholar] [CrossRef]
- Alchalabi, A.E.; Nour, A.; Shirmohammadi, S. More Attention, Less Deficit: Wearable EEG-Based Game for Focus Improvement. In Proceedings of the 2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH), Perth, WA, Australia, 2–4 April 2017; ISBN 9781509054824. [Google Scholar]
- Alim, A.; Imtiaz, M.H. Automatic Identification of Children with ADHD from EEG Brain Waves. Signals 2023, 4, 193–205. [Google Scholar] [CrossRef]
- Huang, H.; Shin, M.C.; Lee, J.; Yoon, S.H. Effective Neurofeedback Training of Large Electroencephalogram Signals Using Serious Video Games. IEEE Access 2023, 11, 112175–112188. [Google Scholar] [CrossRef]
- Aloisio, A.; Marzano, G.; Bonanni, R.; Ricci, M. Serious Games for ADHD: A Narrative Review. In Proceedings of the Digital Innovations for Learning and Neurodevelopmental Disorders, Rome, Italy, 24–25 May 2024. [Google Scholar]
- AlShorman, O.; Masadeh, M.; Alzyoud, A.; Bin Heyat, M.B.; Akhtar, F.; Rishipal. The Effects of Emotional Stress on Learning and Memory Cognitive Functions: An EEG Review Study in Education. In Proceedings of the 2020 Sixth International Conference on e-Learning (econf), Sakheer, Bahrain, 6–7 December 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 177–182. [Google Scholar]
- Pérez Vidal, A.F.; Rumbo Morales, J.Y.; Ortiz Torres, G.; Sorcia Vázquez, F.d.J.; Cruz Rojas, A.; Brizuela Mendoza, J.A.; Rodríguez Cerda, J.C. Soft Exoskeletons: Development, Requirements, and Challenges of the Last Decade. Actuators 2021, 10, 166. [Google Scholar] [CrossRef]
- Gordleeva, S.Y.; Lobov, S.A.; Grigorev, N.A.; Savosenkov, A.O.; Shamshin, M.O.; Lukoyanov, M.V.; Khoruzhko, M.A.; Kazantsev, V.B. Real-Time EEG-EMG Human-Machine Interface-Based Control System for a Lower-Limb Exoskeleton. IEEE Access 2020, 8, 84070–84081. [Google Scholar] [CrossRef]
- Roh, C.H.; Lee, W.B. A Study of the Attention Measurement Variables of a Serious Game as a Treatment for ADHD. Wirel. Pers. Commun. 2014, 79, 2485–2498. [Google Scholar] [CrossRef]
Subject | Detection Efficiency (%) | Subject Average (%) | ||
---|---|---|---|---|
T1 | T2 | T3 | ||
1 | 76 | 80 | 85 | 80 |
2 | 94 | 92 | 90 | 92 |
3 | 88 | 77 | 83 | 83 |
Overall Average of Participants | 85 |
Subject | Voluntary Blink Detection Efficiency (%) | Subject Average | ||
---|---|---|---|---|
T1 | T2 | T3 | ||
1 | 84 | 97 | 86 | 89 |
2 | 70 | 81 | 95 | 82 |
3 | 76 | 87 | 78 | 80 |
Overall Average of Participant | 84 |
Subject | Test | Phase 1 (s) | Phase 2 (s) | Phase 3 (s) | Phase 4 (s) | Phase 5 (s) | Total (s) |
---|---|---|---|---|---|---|---|
S1 | T1 | 9.89 | 7 | 21.10 | 18.62 | 7.74 | 64.35 |
S1 | T2 | 12.39 | 7 | 16.28 | 18.44 | 17.66 | 71.77 |
S1 | T3 | 10.11 | 7 | 21.04 | 14.68 | 9.38 | 62.21 |
S1 | Average | 10.8 ± 1.13 | 7 ± 0 | 19.47 ± 2.26 | 17.25 ± 1.82 | 11.59 ± 4.34 | 66.11 ± 4.10 |
S2 | T1 | 6.40 | 7 | 30.93 | 23.09 | 4.73 | 72.15 |
S2 | T2 | 5.46 | 7 | 18.54 | 20.82 | 6.97 | 58.79 |
S2 | T3 | 4.42 | 7 | 13.47 | 17.00 | 4.01 | 45.90 |
S2 | Average | 5.43 ± 0.81 | 7 ± 0 | 20.98 ± 7.33 | 20.30 ± 2.51 | 5.24 ± 1.26 | 58.95 ± 10.72 |
S3 | T1 | 4.00 | 7 | 19.00 | 23.60 | 7.37 | 60.97 |
S3 | T2 | 6.02 | 7 | 22.04 | 21.39 | 6.86 | 63.31 |
S3 | T3 | 5.36 | 7 | 17.2 | 15.54 | 4.7 | 49.8 |
S3 | Average | 5.13 ± 0.84 | 7 ± 0 | 19.41 ± 2.00 | 20.18 ± 3.4 | 6.31 ± 1.16 | 58.03 ± 5.90 |
S4 | T1 | 5.48 | 7 | 16.22 | 21.69 | 11.05 | 61.44 |
S4 | T2 | 3.9 | 7 | 11.9 | 17.01 | 3.51 | 43.32 |
S4 | T3 | 3.43 | 7 | 13.81 | 14.34 | 3.90 | 42.48 |
S4 | Average | 4.27 ± 0.88 | 7 ± 0 | 13.98 ± 1.77 | 17.68 ± 3.04 | 6.15 ± 3.47 | 49.08 ± 8.75 |
S5 | T1 | 4.19 | 7 | 18.33 | 12.64 | 18.6 | 60.76 |
S5 | T2 | 7.32 | 7 | 21.38 | 22.48 | 3.79 | 61.97 |
S5 | T3 | 6.62 | 7 | 20.37 | 16.84 | 4.89 | 55.72 |
S5 | Average | 6.04 ± 1.34 | 7 ± 0 | 20.03 ± 1.27 | 17.32 ± 4.03 | 9.09 ± 6.74 | 59.48 ± 2.71 |
S6 | T1 | 7.12 | 7 | 30.52 | 21.65 | 6.31 | 72.6 |
S6 | T2 | 10.51 | 7 | 13.79 | 17.07 | 6.81 | 55.18 |
S6 | T3 | 3.38 | 7 | 11.14 | 15.08 | 6.37 | 42.97 |
S6 | Average | 7 ± 2.91 | 7 ± 0 | 18.48 ± 8.58 | 17.93 ± 2.75 | 6.5 ± 0.22 | 56.92 ± 12.16 |
Overall Average | 6.44 ± 2.6 | 7 ± 0 | 18.73 ± 5.35 | 18.44 ± 3.27 | 7.48 ± 4.24 | 58.09 ± 9.55 |
Subject | Test | Active State (s) | Relaxed State (s) | Total Test (s) | |
---|---|---|---|---|---|
Level 1 | Level 2 | ||||
S1 | T1 | 37.95 | 26.4 | 0 | 64.35 |
S1 | T2 | 46.56 | 22.47 | 2.74 | 71.77 |
S1 | T3 | 37.17 | 23.23 | 1.81 | 62.21 |
S1 | Average | 40.56 ± 4.25 | 24.03 ± 1.70 | 1.52 ± 1.14 | 66.11 ± 4.10 |
S2 | T1 | 39.02 | 28.74 | 4.39 | 72.15 |
S2 | T2 | 31.46 | 23.24 | 4.09 | 58.79 |
S2 | T3 | 17.1 | 24.91 | 3.89 | 45.90 |
S2 | Average | 29.19 ± 9.09 | 25.63 ± 2.30 | 4.12 ± 0.21 | 58.95 ± 10.72 |
S3 | T1 | 23.35 | 35.35 | 2.27 | 60.97 |
S3 | T2 | 22.19 | 39.57 | 1.55 | 63.31 |
S3 | T3 | 18.66 | 29.9 | 1.24 | 49.8 |
S3 | Average | 21.40 ± 1.99 | 34.94 ± 3.96 | 1.69 ± 0.43 | 58.03 ± 5.90 |
S4 | T1 | 31.02 | 29.66 | 0.76 | 61.44 |
S4 | T2 | 15.92 | 26.16 | 1.24 | 43.32 |
S4 | T3 | 13.49 | 26.39 | 2.6 | 42.48 |
S4 | Average | 20.14 ± 7.75 | 27.4 ± 1.6 | 1.53 ± 0.78 | 49.08 ± 8.75 |
S5 | T1 | 34.86 | 25.01 | 0.89 | 60.76 |
S5 | T2 | 34.47 | 26.99 | 0.51 | 61.97 |
S5 | T3 | 26.72 | 28.56 | 0.44 | 55.72 |
S5 | Average | 32.02 ± 3.75 | 26.85 ± 1.45 | 0.61 ± 0.2 | 59.48 ± 2.7 |
S6 | T1 | 44.39 | 24.53 | 3.68 | 72.6 |
S6 | T2 | 28.78 | 21.79 | 4.61 | 55.18 |
S6 | T3 | 22.46 | 17.38 | 3.13 | 42.97 |
S6 | Average | 31.88 ± 9.22 | 21.23 ± 2.95 | 3.8 ± 0.61 | 56.92 ± 12.16 |
Overall Average | 29.2 ± 9.58 | 26.68 ± 4.89 | 2.21 ± 1.44 | 58.09 ± 9.55 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Vidal, A.F.P.; Cervantes, J.-A.; Rumbo-Morales, J.Y.; Sorcia-Vázquez, F.D.J.; Ortiz-Torres, G.; Moncada, C.A.C.; Arias, I.d.l.T. Development of RelaxQuest: A Serious EEG-Controlled Game Designed to Promote Relaxation and Self-Regulation with a Potential Focus on ADHD Intervention. Appl. Sci. 2024, 14, 11173. https://doi.org/10.3390/app142311173
Vidal AFP, Cervantes J-A, Rumbo-Morales JY, Sorcia-Vázquez FDJ, Ortiz-Torres G, Moncada CAC, Arias IdlT. Development of RelaxQuest: A Serious EEG-Controlled Game Designed to Promote Relaxation and Self-Regulation with a Potential Focus on ADHD Intervention. Applied Sciences. 2024; 14(23):11173. https://doi.org/10.3390/app142311173
Chicago/Turabian StyleVidal, Alan F. Pérez, José-Antonio Cervantes, Jesse Y. Rumbo-Morales, Felipe D. J. Sorcia-Vázquez, Gerardo Ortiz-Torres, Christian A. Castro Moncada, and Ignacio de la Torre Arias. 2024. "Development of RelaxQuest: A Serious EEG-Controlled Game Designed to Promote Relaxation and Self-Regulation with a Potential Focus on ADHD Intervention" Applied Sciences 14, no. 23: 11173. https://doi.org/10.3390/app142311173
APA StyleVidal, A. F. P., Cervantes, J. -A., Rumbo-Morales, J. Y., Sorcia-Vázquez, F. D. J., Ortiz-Torres, G., Moncada, C. A. C., & Arias, I. d. l. T. (2024). Development of RelaxQuest: A Serious EEG-Controlled Game Designed to Promote Relaxation and Self-Regulation with a Potential Focus on ADHD Intervention. Applied Sciences, 14(23), 11173. https://doi.org/10.3390/app142311173