Intelligent Human–Computer Interaction: Combined Wrist and Forearm Myoelectric Signals for Handwriting Recognition
Abstract
:1. Introduction
2. Materials and Methods
2.1. Study Population
2.2. Experimental Protocol
2.3. Data Pre-Processing and Feature Extraction
2.4. Classification Algorithms
2.5. Pattern Recognition Tests
2.6. Performance Metrics and Statistical Analysis
3. Results
4. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Phinyomark, A.; Quaine, F.; Charbonnier, S.; Serviere, C.; Tarpin-Bernard, F.; Laurillau, Y. EMG feature evaluation for improving myoelectric pattern recognition robustness. Expert Syst. Appl. 2013, 40, 4832–4840. [Google Scholar] [CrossRef]
- Khushaba, R.N. Correlation analysis of electromyogram signals for multiuser myoelectric interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 2014, 22, 745–755. [Google Scholar] [CrossRef] [PubMed]
- Khushaba, R.N.; Nazarpour, K. Decoding hd-emg signals for myoelectric control-how small can the analysis window size be? IEEE Robot. Autom. Lett. 2021, 6, 8569–8574. [Google Scholar] [CrossRef]
- Tigrini, A.; Verdini, F.; Fioretti, S.; Mengarelli, A. On the Decoding of Shoulder Joint Intent of Motion from Transient EMG: Feature Evaluation and Classification. IEEE Trans. Med. Robot. Bionics 2023, 5, 1037–1044. [Google Scholar] [CrossRef]
- Kwon, Y.; Dwivedi, A.; McDaid, A.J.; Liarokapis, M. Electromyography-based decoding of dexterous, in-hand manipulation of objects: Comparing task execution in real world and virtual reality. IEEE Access 2021, 9, 37297–37310. [Google Scholar] [CrossRef]
- Sugiarto, T.; Hsu, C.L.; Sun, C.T.; Hsu, W.C.; Ye, S.H.; Lu, K.T. Surface EMG vs. high-density EMG: Tradeoff between performance and usability for head orientation prediction in VR application. IEEE Access 2021, 9, 45418–45427. [Google Scholar] [CrossRef]
- Jiang, X.; Liu, X.; Fan, J.; Ye, X.; Dai, C.; Clancy, E.A.; Chen, W. Measuring neuromuscular electrophysiological activities to decode HD-sEMG biometrics for cross-application discrepant personal identification with unknown identities. IEEE Trans. Instrum. Meas. 2022, 71, 4009515. [Google Scholar] [CrossRef]
- Ctrl-labs at Reality Labs; Sussillo, D.; Kaifosh, P.; Reardon, T. A generic noninvasive neuromotor interface for human-computer interaction. bioRxiv 2024. [CrossRef]
- Beltrán Hernández, J.G.; Ruiz Pinales, J.; López Rodríguez, P.; López Ramírez, J.L.; Aviña Cervantes, J.G. Multi-Stroke handwriting character recognition based on sEMG using convolutional-recurrent neural networks. Math. Biosci. Eng. 2020, 17, 5435–5448. [Google Scholar]
- Tigrini, A.; Verdini, F.; Scattolini, M.; Barbarossa, F.; Burattini, L.; Morettini, M.; Fioretti, S.; Mengarelli, A. Handwritten Digits Recognition from sEMG: Electrodes Location and Feature Selection. IEEE Access 2023, 11, 58006–58015. [Google Scholar] [CrossRef]
- Liao, Y.Y.; Tseng, H.Y.; Lin, Y.J.; Wang, C.J.; Hsu, W.C. Using virtual reality-based training to improve cognitive function, instrumental activities of daily living and neural efficiency in older adults with mild cognitive impairment. Eur. J. Phys. Rehabil. Med. 2019, 56, 47–57. [Google Scholar] [CrossRef] [PubMed]
- Van Drempt, N.; McCluskey, A.; Lannin, N.A. A review of factors that influence adult handwriting performance. Aust. Occup. Ther. J. 2011, 58, 321–328. [Google Scholar] [CrossRef] [PubMed]
- Danna, J.; Velay, J.L.; Vietminh, V.; Capel, A.; Petroz, C.; Gondre, C.; Thoret, E.; Aramaki, M.; Ystad, S.; Kronland-Martinet, R.; et al. Handwriting movement sonification for the rehabilitation of dysgraphia. In Proceedings of the 10th International Symposium on Computer Music Multidisciplinary Research (CMMR)-Sound, Music & Motion, Marseille, France, 15–18 October 2013; pp. 200–208. [Google Scholar]
- Danna, J.; Velay, J.L. Basic and supplementary sensory feedback in handwriting. Front. Psychol. 2015, 6, 169. [Google Scholar] [CrossRef] [PubMed]
- Shukla, A.W.; Ounpraseuth, S.; Okun, M.S.; Gray, V.; Schwankhaus, J.; Metzer, W.S. Micrographia and related deficits in Parkinson’s disease: A cross-sectional study. BMJ Open 2012, 2, e000628. [Google Scholar] [CrossRef] [PubMed]
- Ziliotto, A.; Cersosimo, M.G.; Micheli, F.E. Handwriting rehabilitation in Parkinson disease: A pilot study. Ann. Rehabil. Med. 2015, 39, 586–591. [Google Scholar] [CrossRef] [PubMed]
- Nackaerts, E.; Broeder, S.; Pereira, M.P.; Swinnen, S.P.; Vandenberghe, W.; Nieuwboer, A.; Heremans, E. Handwriting training in Parkinson’s disease: A trade-off between size, speed and fluency. PLoS ONE 2017, 12, e0190223. [Google Scholar] [CrossRef] [PubMed]
- Kao, H.S. Calligraphy therapy: A complementary approach to psychotherapy. Asia Pac. J. Couns. Psychother. 2010, 1, 55–66. [Google Scholar] [CrossRef]
- Kao, H.S.; Lam, S.P.; Kao, T.T. Chinese calligraphy handwriting (CCH): A case of rehabilitative awakening of a coma patient after stroke. Neuropsychiatr. Dis. Treat. 2018, 14, 407–417. [Google Scholar] [CrossRef] [PubMed]
- Pąchalska, M.; Moskała, M.; MacQueen, B.D.; Polak, J.; Wilk-Frańczuk, M. Early neurorehabilitation in a patient with severe traumatic brain injury to the frontal lobes. Med. Sci. Monit. 2010, 16, CS157–CS167. [Google Scholar] [PubMed]
- Impedovo, D.; Pirlo, G. Dynamic handwriting analysis for the assessment of neurodegenerative diseases: A pattern recognition perspective. IEEE Rev. Biomed. Eng. 2019, 12, 209–220. [Google Scholar] [CrossRef] [PubMed]
- Drotár, P.; Mekyska, J.; Rektorová, I.; Masarová, L.; Smékal, Z.; Faundez-Zanuy, M. Evaluation of handwriting kinematics and pressure for differential diagnosis of Parkinson’s disease. Artif. Intell. Med. 2016, 67, 39–46. [Google Scholar] [CrossRef] [PubMed]
- Huang, G.; Zhang, D.; Zheng, X.; Zhu, X. An EMG-based handwriting recognition through dynamic time warping. In Proceedings of the 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, Buenos Aires, Argentina, 31 August–4 September 2010; pp. 4902–4905. [Google Scholar]
- Linderman, M.; Lebedev, M.A.; Erlichman, J.S. Recognition of handwriting from electromyography. PLoS ONE 2009, 4, e6791. [Google Scholar] [CrossRef] [PubMed]
- Botros, F.S.; Phinyomark, A.; Scheme, E.J. Electromyography-based gesture recognition: Is it time to change focus from the forearm to the wrist? IEEE Trans. Ind. Inform. 2020, 18, 174–184. [Google Scholar] [CrossRef]
- Botros, F.S.; Phinyomark, A.; Scheme, E.J. Day-to-Day Stability of Wrist EMG for Wearable-Based Hand Gesture Recognition. IEEE Access 2022, 10, 125942–125954. [Google Scholar] [CrossRef]
- Khushaba, R.N.; Kodagoda, S.; Takruri, M.; Dissanayake, G. Toward improved control of prosthetic fingers using surface electromyogram (EMG) signals. Expert Syst. Appl. 2012, 39, 10731–10738. [Google Scholar] [CrossRef]
- Singh, S.K.; Chaturvedi, A. Leveraging deep feature learning for wearable sensors based handwritten character recognition. Biomed. Signal Process. Control 2023, 80, 104198. [Google Scholar] [CrossRef]
- Englehart, K.; Hudgins, B. A robust, real-time control scheme for multifunction myoelectric control. IEEE Trans. Biomed. Eng. 2003, 50, 848–854. [Google Scholar] [CrossRef] [PubMed]
- Van Oudenaarde, E.; Oostendorp, R. Functional relationship between the abductor pollicis longus and abductor pollicis brevis muscles: An EMG analysis. J. Anat. 1995, 186, 509–515. [Google Scholar] [PubMed]
- Tigrini, A.; Mengarelli, A.; Cardarelli, S.; Fioretti, S.; Verdini, F. Improving EMG signal change point detection for low SNR by using extended Teager-Kaiser energy operator. IEEE Trans. Med. Robot. Bionics 2020, 2, 661–669. [Google Scholar] [CrossRef]
- Lorrain, T.; Jiang, N.; Farina, D. Influence of the training set on the accuracy of surface EMG classification in dynamic contractions for the control of multifunction prostheses. J. Neuroeng. Rehabil. 2011, 8, 25. [Google Scholar] [CrossRef] [PubMed]
- Hudgins, B.; Parker, P.; Scott, R.N. A new strategy for multifunction myoelectric control. IEEE Trans. Biomed. Eng. 1993, 40, 82–94. [Google Scholar] [CrossRef] [PubMed]
- Du, Y.C.; Lin, C.H.; Shyu, L.Y.; Chen, T. Portable hand motion classifier for multi-channel surface electromyography recognition using grey relational analysis. Expert Syst. Appl. 2010, 37, 4283–4291. [Google Scholar] [CrossRef]
- Huang, H.; Xie, H.B.; Guo, J.Y.; Chen, H.J. Ant colony optimization-based feature selection method for surface electromyography signals classification. Comput. Biol. Med. 2012, 42, 30–38. [Google Scholar] [CrossRef] [PubMed]
- Mengarelli, A.; Tigrini, A.; Fioretti, S.; Verdini, F. Identification of neurodegenerative diseases from gait rhythm through time domain and time-dependent spectral descriptors. IEEE J. Biomed. Health Inform. 2022, 26, 5974–5982. [Google Scholar] [CrossRef] [PubMed]
- Bishop, C.M.; Nasrabadi, N.M. Pattern Recognition and Machine Learning; Springer: New York, NY, USA, 2006; Volume 4. [Google Scholar]
- Taborri, J.; Palermo, E.; Rossi, S. Automatic detection of faults in race walking: A comparative analysis of machine-learning algorithms fed with inertial sensor data. Sensors 2019, 19, 1461. [Google Scholar] [CrossRef] [PubMed]
- Wahid, M.F.; Tafreshi, R.; Langari, R. A multi-window majority voting strategy to improve hand gesture recognition accuracies using electromyography signal. IEEE Trans. Neural Syst. Rehabil. Eng. 2019, 28, 427–436. [Google Scholar] [CrossRef] [PubMed]
- Farrell, T.R. Determining delay created by multifunctional prosthesis controllers. J. Rehabil. Res. Dev. 2011, 48, xxi–xxxviii. [Google Scholar] [CrossRef] [PubMed]
- Wang, S.; Hayashi, S.; Jin, Z.W.; Kim, J.H.; Yamamoto, M.; Murakami, G.; Abe, S. Distally-extending muscle fibers across involved joints: Study of long muscles and tendons of wrist and ankle in late-term fetuses and adult cadavers. Anat. Cell Biol. 2023, 56, 46–53. [Google Scholar] [CrossRef] [PubMed]
- Micera, S.; Carpaneto, J.; Raspopovic, S. Control of hand prostheses using peripheral information. IEEE Rev. Biomed. Eng. 2010, 3, 48–68. [Google Scholar] [CrossRef] [PubMed]
- Okorokova, E.; Lebedev, M.; Linderman, M.; Ossadtchi, A. A dynamical model improves reconstruction of handwriting from multichannel electromyographic recordings. Front. Neurosci. 2015, 9, 389. [Google Scholar] [CrossRef]
- Al-Timemy, A.H.; Serrestou, Y.; Khushaba, R.N.; Yacoub, S.; Raoof, K. Hand gesture recognition with acoustic myography and wavelet scattering transform. IEEE Access 2022, 10, 107526–107535. [Google Scholar] [CrossRef]
- Li, C.; Ma, Z.; Yao, L.; Zhang, D. Improvements on EMG-based handwriting recognition with DTW algorithm. In Proceedings of the 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, Japan, 3–7 July 2013; pp. 2144–2147. [Google Scholar]
Nouns | Adjectives | Verbs | |||
---|---|---|---|---|---|
Word | Label | Word | Label | Word | Label |
time | 1 | good | 11 | be | 21 |
person | 2 | new | 12 | have | 22 |
year | 3 | first | 13 | do | 23 |
way | 4 | last | 14 | say | 24 |
day | 5 | log | 15 | get | 25 |
thing | 6 | great | 16 | make | 26 |
man | 7 | little | 17 | go | 27 |
world | 8 | own | 18 | know | 28 |
life | 9 | other | 19 | take | 29 |
hand | 10 | old | 20 | see | 30 |
TD | Integrated EMG | IEMG |
Mean Absolute Value | MAV | |
Variance of sEMG | VAR | |
Root Mean Square | RMS | |
Waveform Length | WL | |
Difference Absolute Mean Value | DAMV | |
Difference Absolute Standard Deviation Value | DASDV | |
Zero Crossing | ZC | |
Myopulse Percentage Rate | MYOP | |
Willison Amplitude | WAMP | |
Slope Sign Change | SSC | |
Fuzzy Entropy | FuzEN | |
Weighted Permutation Entropy | WPermEN | |
Histogram of EMG, 10-bins | HIST | |
Auto-Regressive Coefficients, 4th Order | AR | |
Cepstrum coefficients of the 4th Order AR process | CC | |
FD | Mean Frequency | MNF |
Median Frequency | MDF | |
Peak Frequency | PKF | |
Total Power | TTP | |
1st Spectral Moment | SM1 | |
2nd Spectral Moment | SM2 | |
3rd Spectral Moment | SM3 | |
Frequency Ratio | FR | |
Power Spectrum Ratio | PSR | |
Variance of Central Frequency | VCF |
Subject | KNN | SVM | LDA | ||||||
---|---|---|---|---|---|---|---|---|---|
Acc | F1 | MCC | Acc | F1 | MCC | Acc | F1 | MCC | |
1 | 0.82 | 0.82 | 0.82 | 0.64 | 0.65 | 0.62 | 0.37 | 0.38 | 0.32 |
2 | 0.87 | 0.82 | 0.82 | 0.66 | 0.68 | 0.64 | 0.33 | 0.33 | 0.26 |
3 | 0.85 | 0.85 | 0.84 | 0.60 | 0.61 | 0.58 | 0.31 | 0.31 | 0.23 |
4 | 0.86 | 0.86 | 0.85 | 0.63 | 0.63 | 0.61 | 0.39 | 0.41 | 0.35 |
5 | 0.83 | 0.83 | 0.83 | 0.63 | 0.64 | 0.62 | 0.38 | 0.42 | 0.38 |
6 | 0.88 | 0.88 | 0.88 | 0.68 | 0.68 | 0.66 | 0.42 | 0.42 | 0.38 |
Average | 0.85 | 0.84 | 0.84 | 0.64 | 0.65 | 0.62 | 0.37 | 0.37 | 0.32 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Tigrini, A.; Ranaldi, S.; Verdini, F.; Mobarak, R.; Scattolini, M.; Conforto, S.; Schmid, M.; Burattini, L.; Gambi, E.; Fioretti, S.; et al. Intelligent Human–Computer Interaction: Combined Wrist and Forearm Myoelectric Signals for Handwriting Recognition. Bioengineering 2024, 11, 458. https://doi.org/10.3390/bioengineering11050458
Tigrini A, Ranaldi S, Verdini F, Mobarak R, Scattolini M, Conforto S, Schmid M, Burattini L, Gambi E, Fioretti S, et al. Intelligent Human–Computer Interaction: Combined Wrist and Forearm Myoelectric Signals for Handwriting Recognition. Bioengineering. 2024; 11(5):458. https://doi.org/10.3390/bioengineering11050458
Chicago/Turabian StyleTigrini, Andrea, Simone Ranaldi, Federica Verdini, Rami Mobarak, Mara Scattolini, Silvia Conforto, Maurizio Schmid, Laura Burattini, Ennio Gambi, Sandro Fioretti, and et al. 2024. "Intelligent Human–Computer Interaction: Combined Wrist and Forearm Myoelectric Signals for Handwriting Recognition" Bioengineering 11, no. 5: 458. https://doi.org/10.3390/bioengineering11050458
APA StyleTigrini, A., Ranaldi, S., Verdini, F., Mobarak, R., Scattolini, M., Conforto, S., Schmid, M., Burattini, L., Gambi, E., Fioretti, S., & Mengarelli, A. (2024). Intelligent Human–Computer Interaction: Combined Wrist and Forearm Myoelectric Signals for Handwriting Recognition. Bioengineering, 11(5), 458. https://doi.org/10.3390/bioengineering11050458