Next Article in Journal
Optimization of Glulam Regular Double-Tapered Beams for Agroforestry Constructions
Next Article in Special Issue
Patient Reported Outcome Measures (PROMs) Amongst Lower Extremity Agonist–Antagonist Myoneural Interface (AMI) Amputees
Previous Article in Journal
Online Audio-Visual Information on the Treatment of OSA with Mandibular Advancement Devices: Analysis of Quality, Reliability and Contents
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 13
Issue 9
10.3390/app13095728
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Early Predictability of Grasping Movements by Neurofunctional Representations: A Feasibility Study

by
Eike Jakubowitz
1,
Thekla Feist
1,
Alina Obermeier
1,
Carina Gempfer
1,
Christof Hurschler
1,
Henning Windhagen
1,2 and
Max-Heinrich Laves
3,4,*
1
Laboratory for Biomechanics and Biomaterials, Department of Orthopaedic Surgery, Hannover Medical School, Anna-von-Borries-Strasse 1-7, 30625 Hannover, Germany
2
Department of Orthopaedic Surgery, Hannover Medical School, Anna-von-Borries-Strasse 1-7, 30625 Hannover, Germany
3
Institute of Mechatronic Systems, Leibniz University Hannover, An der Universität 1, 30823 Garbsen, Germany
4
Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology, Am Schwarzenberg-Campus 3, 38987 Hamburg, Germany
*
Author to whom correspondence should be addressed.
Appl. Sci. 2023, 13(9), 5728; https://doi.org/10.3390/app13095728
Submission received: 1 March 2023 / Revised: 18 April 2023 / Accepted: 20 April 2023 / Published: 6 May 2023
(This article belongs to the Special Issue Exoskeletal Prosthetics: Recent Developments, Innovations and Challenges)
Browse Figures
Versions Notes

Abstract

:
Human grasping is a relatively fast process and control signals for upper limb prosthetics cannot be generated and processed in a sufficiently timely manner. The aim of this study was to examine whether discriminating between different grasping movements at a cortical level can provide information prior to the actual grasping process, allowing for more intuitive prosthetic control. EEG datasets were captured from 13 healthy subjects who repeatedly performed 16 activities of daily living. Common classifiers were trained on features extracted from the waking-state frequency and total-frequency time domains. Different training scenarios were used to investigate whether classifiers can already be pre-trained by base networks for fine-tuning with data of a target person. A support vector machine algorithm with spatial covariance matrices as EEG signal descriptors based on Riemannian geometry showed the highest balanced accuracy (0.91 ± 0.05 SD) in discriminating five grasping categories according to the Cutkosky taxonomy in an interval from 1.0 s before to 0.5 s after the initial movement. Fine-tuning did not improve any classifier. No significant accuracy differences between the two frequency domains were apparent (p > 0.07). Neurofunctional representations enabled highly accurate discrimination of five different grasping movements. Our results indicate that, for upper limb prosthetics, it is possible to use them in a sufficiently timely manner and to predict the respective grasping task as a discrete category to kinematically prepare the prosthetic hand.

1. Introduction

Kinematics of the upper limb can only be mimicked inadequately with myoelectric prostheses due to their control [1]. Addressing more than one DOF results usually in sequential and not simultaneous movements. This is due to the sensitivity of surface electromyography (sEMG) to crosstalk effects, as well as muscle differentiation, which must be learned by the amputee and requires high cognitive effort [2,3]. Despite the recent introduction of multi-articular prosthetic hands with multiple DOFs (e.g., iLimb (Össur, Reykjavik, Iceland) or BeBionic (Otto Bock)), this sEMG control principle remains unchanged. However, independent finger movements are technically possible since each finger is driven by its own actuator [4]. In contrast, hands only perform pre-programmed grasping patterns, which must be selected by the amputee. Only after selection, the amputee can guide his prosthetic hand to the grasping of an object. This action is still activated by the traditional myoelectric ON/OFF mechanism, after which the hand automatically performs the grasping movement. This control regime avoids intuitive, fluid, and comparably natural grasping kinematics [2] and often even leads to the rejection of myoelectric prostheses [5].

1.1. Motion Classes Due to Activation Patterns

In addition to targeted muscle reinnervation, current research has focused on the recognition of motion-typical EMG patterns to overcome this problem [6]. This pattern recognition is of particular interest, because even the central nervous system (CNS) does not selectively drive the 30-plus muscles to address the 20-plus DOFs of the hand [7,8]. Rather, it controls the interaction of all muscles together, which allows for the simultaneous control of all DOFs [9]. To do this, encoded synergies are stored in the CNS to be retrieved and combined in a task-specific manner, whereby the hand kinematic task is executed as a function of the corresponding muscle contraction patterns [10]. These activation patterns can then be assigned to defined motion classes for prosthetic control [11], and machine learning algorithms allow for a classification of muscle synergies with accuracies of over 90% after training [12].

1.2. Source Network Upstreaming

With the aim to further increase accuracy, recent studies have even proposed deep neural network approaches [13,14,15,16]. However, these need to be trained with relatively large datasets. In a specific application, amputees may thus be highly strained when generating data for their individual mapping. Such data can, however, be obtained beforehand using an aggregation of datasets from multiple subjects. Thus, a general mapping can first be pre-trained (source network), which is then combined into a target network by a second neural network trained with data from the target subject. Using data from 17 subjects, Côte-Allard et al. showed systematically and significantly improved neural network performance when supplemented by an upstream source network generated by pre-training using sEMG data from 19 other subjects [16]. The resulting target network achieved accuracy of over 98% for the classification of seven hand gestures. However, this accuracy was achieved by offline data processing. Online processing, as would be necessary for prosthesis control, can only be achieved by sliding time windows that last several hundred milliseconds and is therefore again associated with comparatively large timing offsets [2].

1.3. Timing in Hand Pre-Shaping

Regarding the kinematics of smooth movements that merge into each other, Jeannerod found that the maximum grip aperture (MGA) in healthy subjects was accomplished when the hand was on its way towards grasping the object, i.e., during the reaching phase [17]. Sivakumar et al. have provided the latest figures for this kinematic pattern [18]. Accordingly, in the reaching phase, a hand initially lying on a table takes almost exactly 1000 ms to reach a grasping object placed in front of it, whereby the primary preparation of the hand for a subsequent grasping task, i.e., the MGA, is already completed after approximately 61–65% of the reaching phase’s execution. Thus, the information about the following grasping posture must already be available 600 ms after the start of the reaching phase at the latest for prosthetic movements to appear fluid in a kinematic sequence. Such pattern recognition during reaching movements is often not possible for sEMG due to a number of overlapping influences, such as signal crosstalk or cognitively highly demanding efforts. This would require additional information to the simple ON/OFF control scheme, much earlier than is possible with sEMG pattern recognition.

1.4. Neurofunctional Synergies

In an impressive MRI study, Leo et al. found that kinematic grasping synergies are also represented on a cortical level [19]. Using an intracortical electrode array in a patient with quadriplegia as a brain–computer interface (BCI), these neurofunctional representations were also confirmed by Colachis et al. in a much smaller area of the primary motor cortex [20]. They found that neural synergies were correlated with different grasping objects of comparable size or mass and that neural signals could be used to train an artificial neural network for the functional electrical stimulation of one of the paretic forearms. In this way, seven functional hand postures could be generated for the manipulation of different objects. Other research groups have shown that the motor commands required for kinematic coordination are generated simultaneously by the sensorimotor cortex [21,22], i.e., during movement. In contrast, an earlier MRI study found that although their planning and initiation can be represented in a complementary manner by primary motor cortex areas, hand gestures are mainly represented by activations in the supplementary motor area up to 1900 ms before the actual movement [23]. However, this special feature was disregarded in previous BCI studies that aimed to discriminate grasping using artificial neural networks, since the cortical representation was only generated by action recognition approaches and not physically [24,25,26,27]. Thus, electroencephalography (EEG) data were only used from 500 ms after the actual stimulus, which—if it can be applied to the present work—again falls within the time range of the MGA for a primary hand preparation during the reaching phase, i.e., much too late. Whether far earlier hand posture discrimination using human EEG data is feasible in the field of upper limb prosthetic control remains completely unexplored. However, the working group around Müller-Putz impressively could show that in the low-frequency EEG signal range between 0.1 and 3.0 Hz, promising discriminations of grasping objects and types with multi-class linear discriminant analysis were possible when the planning phase was taken also into account or was even fused with the execution phase [28,29].

1.5. Derived Study Aims

Therefore, we investigate whether EEG data can be used to discriminate different grasping postures in a timely manner to generate a signal for prosthetic control. To increase the potential for real-world applications, we also investigate whether it makes sense to train a classifier first with training data and then to tune this using EEG data from a target person, as has already been performed for sEMG classifications.
The aims of this work are as follows:
  • To study the suitability of the currently used classifiers for discriminating grasping postures in general;
  • To identify the most meaningful way to categorize grasping motions to achieve a good balance between achievable discrimination accuracy and adequate grasping postures for prosthetics;
  • To study whether it is, in principle, possible to consider a time range up to the actual MGA completion for the classification, or whether it is even possible to focus on a time range before movement initiation, to better meet the critical problem regarding timing in hand pre-shaping;
  • To identify whether only waking-state frequency bands are suitable, as found in other studies, or whether the complete EEG band is more suitable due to earlier and/or other neurofunctional representations compared to previous BCI studies;
  • To investigate whether the straining risk of an amputee in generating his individual classifier can be avoided using a pre-trained source network.

2. Methods

All procedures described in this study were approved by the Ethics Committee of Hannover Medical School (No. 3364) and conducted in accordance with the Declaration of Helsinki. The subjects provided written informed consent to participate. Thirteen healthy subjects S1–S13 (mean age = 42.5 ± 16.2 years; f:m = 9:4; all right-handed) were included in the study. The inclusion criterion was an age of between 18 and 75 years. Non-inclusion criteria were upper limp movement restrictions, neuronal impairments, and cognitive deficits.

2.1. Study Protocol

The volunteers were asked to perform a set of 30 activities of daily living (ADLs) in a randomized order. Sixteen ADLs (including eight transitive and eight tool-mediated; Figure 1) were selected based on classical human grasping taxonomies according to Cutkosky [30] and Feix et al. [31], while considering typical grasping patterns of multi-articular prosthetic hands for the later defined grasping posture categorizations (Table 1). According to an sEMG synergy study, subjects S1–S11 repeated the ADLs three times, and their EEG data were used for classifier pre-training. Two additional test subjects, S12 and S13, which were selected from the pool of volunteers without particular rationale, executed the ADLs with 30 repetitions for the subsequent classifier fine-tuning. This resulted in a dataset size of 528 EEG samples from S1–S11 and 960 additional samples from S12 and S13. Subjects sat at a height-adjustable table with the marked initial position of the hand and the initial and target positions of grasping objects on the surface. To ensure that movements were as natural as possible, no precise execution instructions were given. Subjects were only asked to execute the ADL (for example, “take a sip from the mug”), to place their right forearm relaxed with the flat palm on the initial position before and after the ADL execution, and to wait approximately 2 s before the ADL execution after the stimulus, which was given by a verbal command. Using DV cameras (Pilot pi640gc, Basler, Ahrensburg, Germany) synchronized with an EEG measuring system (actiCHamp Plus, Brain Products GmbH, Gilching, Germany), videos of ADLs were recorded at 200 Hz in the frontal and sagittal plane. Using a single-polar leakance, the neuronal activity of the brain was recorded at 500 Hz by 32 electrodes placed on the scalp with an electrode cap (actiCAP Standard 32, Easycap GmbH, Herrsching, Germany) according to the 10–20 system [32].

2.2. Grasping Posture Categorizations

According to the typical grasping patterns of multiarticular prosthetic hands, the ADLs were assigned to six specific categories before the training (G6, first row in Table 1). Given that it is not yet known which grasping postures show synergic neurofunctional representations before and during the reaching phase, up to the MGA, two further categorizations were considered. The first one (C5, second row in Table 1) corresponds to the grasping taxonomy according to Cutkosky. It was not possible to assign the reaching phase endpoints of ADLs 2, 5, 7, 10, 11, 13, and 16 shown in Figure 1 using this taxonomy, and these ADLs were thus not included in the categorization. The second categorization (F4, third row in Table 1) was based on the Feix grasping taxonomy, but the reaching phase endpoints of ADLs 2, 5, and 8 shown in Figure 1 could not be assigned and remained unconsidered for categorization. The classes of the different categorizations are used as prediction targets for the classifiers in Section 2.4 and the following. As their names suggest, G6 contains six classes, C5 contains five classes, and F4 contains four classes (see Table 1).

2.3. Data Processing and Epoching

The EEG signal had 31 channels and was processed separately for each subject using the Toolbox EEGLAB [33] for MATLAB (Vers. R2020a, The MathWorks Inc., Natick, MA, USA). The raw data were notch- (50 Hz) and bandpass-filtered using the finite impulse response filter. To investigate possible influences of frequency bands on the classification results, the bandpass filtering was performed using the combined alpha and beta sub-bands of the waking state (fα-β = 8–30 Hz), as well as using the whole EEG signal spectrum covering sub-bands from delta up to gamma (fδ-γ = 0.5–100 Hz). To separate actual brain signals from interference signals resulting from muscle and eye artifacts and superimposed brain signals, a linear decomposition was performed using independent component analysis [33]. Confining events, such as “initial movement” and “object grasped”, were identified manually with the aid of synchronized videos, according to the protocol of Sivakumar et al. [18], which allowed us to separate the reaching phase from the action phase (Figure 2). In all ADLs, the reaching phase had a duration of at least 800 ms, which resulted in the earliest occurring MGA interval of 488–520 ms [18]. For the discrimination, two data epochings of different durations were chosen within the limits of when neuronal representations should already be reliably detectable [23], but at which point the hand is not yet preparing for the respective grasping posture, i.e., fulfilling a pre-shaping for the grasping object. Hence, discrimination must be accomplished before the MGA is reached; the duration T1 was thus defined as 500 ms after the initial movement, which is in line with grasping phase latencies [34]. A long epoch EL resulted from the total sum of two equal lasting time windows, T2 and T3, prior to the initial movement (EL = T1 + T2 + T3 = 1500 ms [−1000 ms ≤ EL ≤ 500 ms]). To determine whether an earlier discrimination might allow a prosthetic device more time for hand preparation execution for grasping tasks, a short epoch ES was defined prior to the initial movement (ES = T2 + T3 = 1000 ms [−1000 ms ≤ ES ≤ 0 ms]).

2.4. Feature Extraction and Classifiers

Due to the high complexity and dimensionality of EEG signals, the discrimination accuracy depends heavily on feature extraction methods [35]. Two main extraction methods were chosen in the present study, as follows: the common spatial pattern (CSP) method, which is the most commonly applied filter for band-power feature extractions in hand movements [36]; and the xDAWN spatial filtering with covariance matrix estimation, which has been increasingly used in more recent work, and also allows for time point feature extractions [37]. The former computes spatial filters for signal separation by maximizing variance ratios and the latter computes spatial filters that maximize the ratios between the signals and the signals plus noise. After feature extraction, all classifiers were trained to predict the classes of the considered categorizations (see Section 2.2). The following state-of-the-art matrix and tensor classifiers regarding EEG-based BCIs were compiled and adapted.
(1) Multinomial arranged logistic regression (LR): This uses variable vectors representing coefficients of the input variables to assume likelihoods [38]. The regression model was adapted to the data using maximum likelihood estimation for regression parameters.
(2) A linear discriminant analysis (LDA) [39]: Discrimination was achieved by estimating the optimal discriminant function.
(3) A support vector machine (SVM) classifier: EEG data were separated into their classes using a hyperplane. Since EEG data cannot be linearly separated, the radial basis function kernel was used.
(4) The LR approach was repeated within the tangent space (TS). Thus, a projection of covariance matrices to TS was performed using Riemannian geometry [25,40].
(5) The SVM approach was also repeated within the TS using Riemannian geometry [25,40].
(6) A further classifier computing the Riemannian minimum distance to mean (MDM) value [26] was applied. This calculates a geometric mean value for each class from the training data and then assigns an undefined trial to the class corresponding to the closest value [41]. However, class-interesting features within covariance matrices can be obscured by noise.
(7) To counteract this, the MDM classifier was additionally extended by geodesic filtering (FgMDM) to further maximize the variance between classes and further minimize it within classes [25,40].
The LR, SVM, and LDA classifiers were implemented using the Python library scikit-learn with their respective default regularization; the MDM classifier was implemented using the Python library pyriemann, which introduces regularization by estimating the shrunk Ledoit–Wolf covariance matrices [42]. Figure 3 shows the compiled classifiers as a function of the two feature extraction methods.

2.5. Validation and Hyper-Parameters

To determine the initial ability of the algorithms, the data were randomly split into nine blocks of learning and one block of test data of the same size, using the repeated ten-fold stratified cross validation (CV) method [43]. For optimization, automatic hyper-parameter tuning that applied all possible parameter settings (Figure 3) was performed using a grid search on the nine training sets. The combinations with the highest performance value in relation to the k-fold average were finally selected. This optimization was separately performed for the pre-training dataset from subjects S1–S11, and for each of the fine-tuning datasets from S12 and S13, which resulted in three different hyper-parameter tunings for each classifier. To avoid generalization errors, a ten-fold stratified CV was applied with the most suitable hyper-parameters for the final evaluation of all algorithms. The final results are reported on the hold-out test set that is not used for training or hyper-parameter tuning. For the fine-tuning stage, this results in a training set size of 243 to 432 samples and a test set size of 27 to 48 samples, depending on the used categorization. Since the building of categories, described in Section 2.2, was partially imbalanced, the balanced accuracy (BA) was calculated as the performance metric for algorithms [44], and standard deviations were used to indicate the stability of the result. To analyze statistical significance, a k-fold cross-validated two-sided Student’s t-test with k = 10 was performed for all used models resulting from classifiers and feature extractions, and learning scenarios explained in the following [45].

2.6. Learning Scenarios

In order to analyze the most effective machine learning approach for a specific application in upper limb prosthetics, four successive learning scenarios A–D were developed and calculated (Figure 4). The simplest scenario A served as a reference and practically means that data from exactly one target person are directly used for hyper-parameter optimizations to teach classifiers for this target person. In this case, these were the target subjects S12 or S13. Scenario B represents a classifier that is pre-trained with data of several foreign subjects (S1–S11), so that the final classifiers can be used directly for the target subject S12 or S13 without a new hyper-parameter optimization. For the target subject—or the amputee—this would mean no further effort at all, since a finished product can be directly used. Scenario C is comparable to scenario B, but, in addition, the hyper-parameters were adjusted with the fine-tuning datasets of target subject S12 or S13 after the pre-training with the data of foreign subjects S1–S11. As done in EMG studies [16], the purpose of this approach is to analyze whether a source classifier can already be prepared for later fine-tuning with the data of the target subject to possibly achieve an improvement by upstream learning. Scenario D behaves similarly to scenario C, with the difference that data from S13 were used for pre-training and data from S12 were used for fine-tuning, and vice versa. This would prove whether the effort on the side of the manufacturer for such a classifier can possibly be reduced by less data instead of using a dataset from several test subjects, as in scenario C. All calculations were performed on the same workstation (processor: 2.80 GHz Intel® Core™ i5, Intel Corporation, Santa Clara, CA, USA).

3. Results

3.1. Classifiers

Since target subjects S12 and S13 basically showed comparable results in all analyses, the findings are illustrated using S12 as the representative. Apart from the LR algorithm, there was always lower discrimination accuracy for scenario B (fine-tuning after pre-training with S1–S11 data without hyper-parameter adjustments for the target subject) compared to all other scenarios within the G6 categorization (Figure 5). For the LDA and SVM classifiers, this reduced accuracy of scenario B was particularly pronounced. There were no significant differences between scenarios A (fine-tuning without pre-training), C (fine-tuning with pre-training using S1–S11 data and hyper-parameter adjustments for the target subject), and D (fine-tuning with pre-training using S12/S13 data and hyper-parameter adjustments for the target subject) for all classifiers (p > 0.4). Furthermore, for both target subjects S12 and S13, classifiers within the tangent space TS_LR and TS_SVM always resulted in the highest accuracies, regardless of which of the three categorizations, which of the two frequency bands, and which of the two epochs was used. Regarding the combination of G6 with the combined delta and gamma frequency band on the one hand, and with the long epoch on the other hand, the highest accuracies could be found in all three scenarios A, C, and D on average with BA = 0.74 ± 0.06 (mean ± standard deviation) and BA = 0.79 ± 0.07, respectively (Figure 5). Given that this result was comparable for all other conditions, the LR, LDA, SVM, MDM, and FgMDM classifier results are omitted from the manuscript.

3.2. Grasping Posture Categorizations

On inspecting the remaining classifiers to compare the grasping posture categorizations, the significantly highest accuracies were achieved using the C5 categorization according to the Cutkosky taxonomy (p < 0.008) [30]. On excluding scenario B again, these average accuracies were BA = 0.88 ± 0.07 for the TS_LR algorithm, and BA = 0.90 ± 0.06 for the TS_SVM algorithm for the combined delta and gamma frequency band and the long epoch (Figure 6). In this case, a trend towards maximum accuracy values was found for the TS_LR algorithm with BA = 0.90 ± 0.06 in scenario C, and with BA = 0.91 ± 0.05 for the TS_SVM algorithm in scenario A. Across all scenarios, lower accuracies were observed in the G6 categorization with the TS_LR algorithm with mean BA = 0.73 ± 0.06, and for the TS_SVM algorithm with mean BA = 0.78 ± 0.07 (averaged over all scenarios). On excluding scenario D (fine-tuning with pre-training using S12/S13 data and hyper-parameter adjustments for the target subject), a comparably high accuracy result for S12 and the TS_SVM algorithm was found for the F4 categorization. Due to the general dominance of the C5 categorization, the G6 and the F4 categorization results are omitted in the following. However, it is important to note that a different number of grasps fall into the categorizations and a direct comparison between categorizations has to be considered with caution.

3.3. Frequency Bands and Epoch Length

There were no significant differences in the accuracies on comparing both frequency bands (p > 0.07, see Figure 7). When comparing the epochs, TS_SVM performed significantly better for the long epoch (EL) for scenarios A and C in both frequency bands (p < 0.032) and for scenario D in the extended frequency band f = 0.5–100 Hz (p = 0.039). TS_LR also performed significantly better for the longer epoch for scenario C in the extended frequency band (p = 0.004). Other significant differences were not observed. Overall, there was a general tendency towards higher accuracy for the long epoch, irrespective of the frequency band. The highest and most constant accuracy that already became evident during the grasping posture categorization was with the TS_SVM algorithm, with a BA = 0.91 ± 0.05 for scenario A (fine-tuning without pre-training) with the combined delta and gamma frequency band and the long epoch; this value did not differ from those of scenarios C or D.

4. Discussion

The present study investigated whether EEG data can be used to discriminate between different grasping movements for a prosthetic application, i.e., within a sufficient time window. To this aim, different combinations of classifiers, feature extraction methods, time windows, EEG frequency bands, and developed learning scenarios were analyzed. We also examined whether pre-training with data from different subjects would enable the subsequent fine-tuning of classifiers for a target subject, which could simplify the efforts of amputees when training their individual prosthetic control.
To our knowledge, the present results are the first to have used neurofunctional representations for a real multiclass prediction within the required time window for upper limb prosthetics, i.e., before preparation of the hand movement sequence for the respective grasping task. In contrast to other EEG studies [24,25,26,27], our results are based on real, physically executed grasping kinematics and not on action recognition. Previous studies with similar aims but lower accuracies used only a two-class approach to discriminate between lateral and palmar grasps [46,47]. Additionally, when discrimination was investigated for more than two different grasping types, either only a single object for the respective grasping pattern was investigated [48], standardized grasping taxonomies were not considered [27], or signal intervals unsuitable for prosthetic tasks were used [28,29]. However, the discrimination accuracy of ADLs found in the present study seems to depend on several factors, which will now be discussed.

4.1. Pre-Training

Scenarios were designed to be of increasing complexity to allow for a systematic comparison of direct classifier training with the data of a target subject as a reference (scenario A) up to a fine-tuning using the data of several subjects (scenario C), or data from one subject (scenario D) for pre-training. Following prior sEMG studies [16,49], it was originally assumed that direct training with the data of a target subject (scenario A) would result in less accurate discrimination, since the classifier could not exploit information from prior training. In contrast, pre-training with data from multiple subjects, but without further adjustment of the hyper-parameters of the target subject’s data (scenario B), resulted in lower accuracies in almost all investigated conditions, even though the amount of training data was larger. Thus, the adaptation of the hyper-parameters to a particular target subject, as was consistently the case in all the following scenarios, seems to play a crucial role in the discrimination success of physical hand kinematics. Nevertheless, the influence of such adaptations seems to vary considerably depending on the classifier used. For example, the TS_LR and TS_SVM algorithms were shown to be relatively robust, whereas the MDM and LDA algorithms were far more sensitive regarding hyper-parameter changes (with consistently lower accuracies in scenario B). In addition, a notable difference in hyper-parameter adjustments was observed between the S1–S11 pre-training dataset and the S12 and S13 target training datasets. Comparatively fewer filters were used for the pre-training than for the target datasets for CSP filtering. For example, this ratio was two (pre-training) to twenty (fine-tuning) when using the LDA algorithm. The smaller number of filters was probably due to the complexity of finding uniform features in the data of multiple subjects for pre-training, whereas the target datasets contained EEG signals from only a single subject with presumably comparable features. However, in both cases (pre-training and fine-tuning), the mutual filters were defined by the highest eigenvalues, and should therefore have extracted similar features. These filter settings via hyper-parameter optimization remained almost unchanged from scenario to scenario since calculations were always performed using the same data and thus with the same isolated features. This also explains why scenarios A, C, and D resulted in similar discrimination accuracies, while scenario B resulted in comparatively lower accuracy. In scenario B, hyper-parameter optimization was omitted during the fine-tuning process. Thus, a clear selection of the “best scenario” is not possible when excluding scenario B, which means that there was no clear advantage of using a pre-trained classifier. Based on these findings, it seems sufficient to train a classifier directly using the EEG data of the target person, whereby 30 repetitions of ADLs seemed to provide a sufficient basis. Capturing these data would require an intensive session with an amputee, but it is nonetheless manageable. In addition, no decisive difference in discrimination accuracy was found for scenarios C and D. Pre-training data did not have an influence on the result.

4.2. Classifiers

A consistent trend emerged when looking at the algorithms used, independent of the subject, the feature extraction method, epoch, frequency band, categorization of grasping postures, and scenario. Namely, the TS_LR and TS_SVC algorithms consistently achieved the most accurate discriminations, whereas the MDM algorithm achieved the lowest accuracies, with a few exceptions. The values of the LDA and LR algorithms were consistently at least 10% below those of the TS_LR and TS_SVM algorithms. In contrast, the SVM and FgMDM algorithms often provided more accurate discriminations than the LDA and LR algorithms, but lower accuracies than the TS algorithms. Thus, the SVM and FgMDM algorithms can be considered as mediocre classifiers for early EEG data.

4.3. Categorization of Grasping Postures

The 16 ADLs were selected according to the initially developed categorization G6, which included six typical grasping patterns of multi-articular prosthetic hands. Thus, at least two ADLs were assigned to each category. However, since it was not clear at the beginning of the study which neurofunctional representation would evoke movement-typical features, we considered two additional categorizations. For the C5 categorization, seven ADLs had to be excluded since they did not correspond to any category of the Cutkosky taxonomy. Thus, we decided to include a categorization according to the Feix taxonomy; while this contains fewer categories, only three ADLs had to be excluded. The different classification accuracies can be explained by this categorization diversity. There were fewer yet more evenly distributed ADLs in the C5 categorization, which resulted in algorithms that were more likely able to distinguish features within early EEG signals, favoring the discrimination result. In contrast, there were two to four ADLs per group in the G6 categorization, which apparently led to less favorable weightings of relevant features. Even more unfavorable weightings appeared in the F4 categorization. Either this type of categorization should not be considered for such a prosthetic task in general, or the allocation of ADLs to categories did not correspond to synergic features of neurofunctional representations.

4.4. Epoching

To investigate the influence of the time domain on the discrimination accuracy, the epoch was reduced to one second before the initial movement. While shortened epochs can avoid motion artifacts [49], earlier discrimination is more favorable for technical use and offers a longer interval for a prosthetic hand system to prepare for the respective grasping task. However, with the data used in the present study, an early epoch decreased the discrimination accuracy of all algorithms. This could be because the information content of cortical activity seems to peak at approximately 250 ms after the initial movement [47], and thus would be excluded from classification, resulting in accuracy losses of around 4–10%. However, according to Combrisson and Jerbi [50], the classification performance remained different even for the four-category problem of the Feix taxonomy categorization, which is indicative of the validity of an early discrimination prior to the initial movement.

4.5. Frequency Bands

Most BCI studies use bandpass filters to process their EEG signals only within the sub-frequencies of the waking state [51]. For example, Sburlea and Müller-Putz [34], who investigated virtual grasping objects, found the best categorization of grasping movements during the reaching phase as a function of neurofunctional representations in parietal brain regions in the lower beta frequency band. In contrast, we found a tendency toward better but at least comparable discrimination accuracy using an extended band that also included delta to gamma frequencies (fδ,γ = 0.5–100 Hz). This could be because, contrary to most other studies [24,25,26,27,34,46,51], we included time windows well before the initial action recognition or the initial movement execution into the epoching, and the features of neuronal activity during this movement planning phase seem to be more prominent within these frequencies than in only the alpha and beta sub-bands. This should be investigated in more detail to avoid overlooking frequency components that are potentially useful for hand prosthetic applications.

4.6. Limitations

Although the present study demonstrates the potential of neurofunctional representations as a complementary control signal to achieve fluid prosthetic movements, it is subject to some limitations. First, we obtained data from healthy subjects. While amputees do indeed perform movements such as reaching for an object with the residual upper limb and the prosthetic device, the actual physical grasping movement no longer exists given that the corresponding limb—i.e., the human end effector—is missing. However, it depends on the amputation level and whether an amputation is acquired or congenital. Neurofunctional representations of grasping movements can be assumed to be at least partly comparable for acquired amputations, since primary motor neurons were once functional and are still physically present. In the case of dysmelia, fundamentally different representations can be expected, because these must be generated by action recognition of hand and arm movements that have never been performed [52]. Furthermore, the relatively small datasets must be mentioned. Data for the pre-training consisted of only three repetitions per subject and ADL, which resulted in approximately the same number of data as in the fine-tuning dataset. This undoubtedly had a considerable effect on the discrimination accuracy of the pre-training. Thus, this dataset should be expanded for further investigations that go beyond a feasibility investigation. Nevertheless, this limitation explains why an advantage of pre-training could not clearly be shown with the present data. As well as electrode placement with a measuring cap and electrode gel, another limitation is the cumbersome consideration of all 31 EEG channels, which would be practically impossible to transfer to a prosthetic application. Nevertheless, the present study demonstrates the initial feasibility of such a neurofunctional approach, which is certainly worth pursuing further. Since Schwarz et al. [29] have already shown that the number and setup of electrodes have a significant effect on the performance, subsequent studies should investigate the characteristics of these representations in more detail to allow for a considerable channel reduction. Indeed, Schwarz et al. [46] have presented very promising results in their comparative study, in which they used a dry EEG system consisting of eleven electrodes that covered only the superior orbital part of the brain. Although this system had lower accuracy for the discrimination of different reach and grasping actions from the alpha and beta frequency bands in general, the authors found a performance decrease of around 6% compared to a conventional multichannel gel-based EEG system, such as the one used in our study. A further limitation is that the distribution of ADLs to categorizations varied. A more equal distribution of ADL numbers within categories should be obtained in future work to achieve even better discrimination accuracies. Under such conditions, it would be worth investigating whether the C5 categorization identified in the present study as advantageous represents an approach that can be applied to prosthetics control. In addition, the algorithms and scenarios used in this study processed simultaneous information from multiple spectral features across multiple brain areas as a type of black box. Thus, further information about which motion parameters are encoded by the signals and how they are accurately represented cortically in time (i.e., as a function of planned or performed grasping action) and space should be a focus of future investigations.

5. Conclusions

In summary, neurofunctional representations seem to be usable to discriminate physical grasping movements, with high accuracy of up to 91%. With the present EEG dataset and a TS_SVM classifier, it was possible to predict nine out of ten grasping movements correctly. Thus, in principle, it would be possible to use this discrimination in upper limb prosthetics. Using sEMG, a complementary control approach would be conceivable, in which the kinematic hand preparation is predicted as a discrete class from EEG, based on which the already pre-programed grasping pattern of multi-articular hand prostheses can be implemented. Workarounds such as co-contractions or task selections due to gesture control could be omitted and may lead to more intuitive control for the more fluid preparation of the prosthetic hand for the upcoming task by automatic pre-shaping. Nevertheless, the discrimination accuracy depends on several factors. For example, the selected categorization of the grasping postures seems to play a decisive role, and the distribution and number of comparable ADLs also appear to be important. The most accurate predictions were achievable using the categorization of ADLs according to the Cutkosky taxonomy, with which a comparable selection of grasping postures could potentially be implemented for multi-articular prosthetic hands. Consideration of the interval of neurofunctional representations for the training of classifiers may also have a crucial effect. Given that it is possible to establish discrimination prior to the actual hand preparation for the respective grasping task, an interval that extends into the reaching phase should be chosen. Within such an interval, far more decisive representations can influence the discrimination accuracy than intervals occurring before the initial movement, i.e., during movement planning. In contrast, the frequency sub-bands had no decisive influence on the discrimination accuracy. Regarding a real-world discriminating application, the notched raw signal may also be usable, and could considerably reduce efforts during online data processing.

Author Contributions

E.J. and H.W. acquired the funding; E.J., A.O., H.W. and M.-H.L. designed and coordinated the study; E.J., T.F., A.O. and C.G. performed the data collection; T.F. processed and analyzed the data; E.J., T.F. and M.-H.L. interpreted the results, conducted the literature search, wrote the manuscript, and generated the figures; C.H. and H.W. revised the manuscript; E.J. and M.-H.L. drafted the manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

This study received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No. 688857, called “SoftPro”.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Ethics Committee of Hannover Medical School (No. 3364).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Not applicable.

Acknowledgments

The authors would like to thank the volunteers for their efforts and patience during the study; Marc-Nils Wahalla from the Institute of Microelectronic Systems (Leibniz University Hannover) for helping with the measurement equipment; and Ahmad Sasah, Benjamin Fleischer-Lück, and Juliana Clavijo Rincón for their assistance in data collection, coding, and data processing.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

General Acronyms
AcronymDefinition
ADLsactivities of daily living
BAbalanced accuracy
BCIbrain–computer interface
C5grasping category, 5 classes, Cutkowsky taxonomy
CNScentral nervous system
CVcross-validation
DOFdegree of freedom
DVdigital video
EEGelectroencephalography
EL, ESlong and short epoch
F4grasping category, 4 classes, Feix taxonomy
G6grasping category with 6 classes
MGAmaximum grip aperture
MRImagnetic resonance imaging
S1-S13subjects 1–13
sEMGsurface electromyography
T1, T2, T3time intervals 1–3 lasting 500 ms each
Feature Extraction and Classifier Acronyms
AcronymDefinition
CSPcommon spatial pattern filter
FgMDMMDM classifier extended by geodesic filtering
LDAlinear discriminant analysis classifier
LRlinear regression classifier
MDM(Riemannian) minimum distance to mean
SVMsupport vector machine classifier
TStangent space
TS_LRLR classifier in tangent space
TS_SVMSVM classifier in tangent space
xDAWNspatial filter with covariance matrix estimation

References

  1. Engdahl, S.M.; Gates, D.H. Differences in quality of movements made with body-powered and myoelectric prostheses during activities of daily living. Clin. Biomech. 2021, 84, 105311. [Google Scholar] [CrossRef]
  2. Ison, M.; Artemiadis, P. The role of muscle synergies in myoelectric control: Trends and challenges for simultaneous multifunction control. J. Neural Eng. 2014, 11, 051001. [Google Scholar] [CrossRef] [PubMed]
  3. Ciancio, A.L.; Cordella, F.; Barone, R.; Romeo, R.A.; Bellingegni, A.D.; Sacchetti, R.; Davalli, A.; Di Pino, G.; Ranieri, F.; Di Lazzaro, V.; et al. Control of prosthetic hands via the peripheral nervous system. Front. Neurosci. 2016, 10, 116. [Google Scholar] [CrossRef]
  4. Belter, J.T.; Segil, J.L.; Dollar, A.M.; Weir, R.F. Mechanical design and performance specifications of anthropomorphic prosthetic hands: A review. J. Rehabil. Res. Dev. 2013, 50, 599–618. [Google Scholar] [CrossRef]
  5. Biddiss, E.; Chau, T.T. Upper limb prosthesis use and abandonment: A survey of the last 25 years. Prosthet. Orthot. Int. 2007, 31, 236–257. [Google Scholar] [CrossRef]
  6. Resnik, L.; Huang, H.; Winslow, A.; Crouch, D.L.; Zhang, F.; Wolk, N. Evaluation of EMG pattern recognition for upper limb prosthesis control: A case study in comparison with direct myoelectric control. J. Neuroeng. Rehabil. 2018, 15, 23. [Google Scholar] [CrossRef] [PubMed]
  7. Savescu, A.; Cheze, L.; Wang, X.; Beurier, G.; Verriest, J.P. A 25 degrees of freedom hand geometrical model for better hand attitude simulation. SAE Tech. Pap. 2004. [Google Scholar] [CrossRef]
  8. Jarque-Bou, N.J.; Vergara, M.; Sancho-Bru, J.L.; Gracia-Ibáñez, V.; Roda-Sales, A. A calibrated database of kinematics and EMG of the forearm and hand during activities of daily living. Sci. Data. 2019, 6, 270. [Google Scholar] [CrossRef]
  9. Li, S.; He, J.; Sheng, X.; Liu, H.; Zhu, X. Synergy-driven myoelectric control for EMG-based prosthetic manipulation: A case study. Int. J. Human Robot. 2014, 11, 1450013. [Google Scholar] [CrossRef]
  10. Tresch, M.C.; Saltiel, P.; d’Avella, A.; Bizzi, E. Coordination and localization in spinal motor systems. Brain Res. Brain Res. Rev. 2002, 40, 66–79. [Google Scholar] [CrossRef] [PubMed]
  11. Vujaklija, I.; Shalchyan, V.; Kamavuako, N.; Jiang, N.; Marateb, H.R.; Farina, D. Online mapping of EMG signals into kinematics by autoencoding. J. Neuroeng. Rehabil. 2018, 15, 21–30. [Google Scholar] [CrossRef]
  12. Scheme, E.; Englehart, K. Electromyogram pattern recognition for control of powered upper-limb prostheses: State of the art and challenges for clinical use. J. Rehabil. Res. Dev. 2011, 48, 643–659. [Google Scholar] [CrossRef] [PubMed]
  13. Allard, U.C.; Nougarou, F.; Fall, C.L.; Giguere, P.; Gosselin, C.; Laviolette, F.; Gosselin, B. A convolutional neural network for robotic arm guidance using sEMG based frequency-features. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Republic of Korea, 9–14 October 2016; pp. 2464–2470. [Google Scholar]
  14. Atzori, M.; Cognolato, M.; Muller, H. Deep learning with convolutional neural networks applied to electromyography data: A resource for the classification of movements for prosthetic hands. Front. Neurorobot. 2016, 10, 9. [Google Scholar] [CrossRef] [PubMed]
  15. Du, Y.; Jin, W.; Wei, W.; Hu, Y.; Geng, W. Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation. Sensors. 2017, 17, 458. [Google Scholar] [CrossRef]
  16. Cote-Allard, U.; Fall, C.L.; Drouin, A.; Campeau-Lecours, A.; Gosselin, C.; Glette, K.; Laviolette, F.; Gosselin, B. Deep learning for electromyographic hand gesture signal classification using transfer learning. IEEE Trans. Neural Syst. Rehabil. Eng. 2019, 27, 760–771. [Google Scholar] [CrossRef] [PubMed]
  17. Jeannerod, M. The timing of natural prehension movements. J. Mot. Behav. 1984, 16, 235–254. [Google Scholar] [CrossRef]
  18. Sivakumar, P.; Quinlan, D.J.; Stubbs, K.M.; Culham, J.C. Grasping performance depends upon the richness of hand feedback. Exp. Brain Res. 2021, 239, 835–846. [Google Scholar] [CrossRef]
  19. Leo, A.; Handjaras, G.; Bianchi, M.; Marino, H.; Gabiccini, M.; Guidi, A.; Scilingo, E.P.; Pietrini, P.; Bicchi, A.; Santello, M.; et al. A synergy-based hand control is encoded in human motor cortical areas. Elife 2016, 15, e13420. [Google Scholar] [CrossRef]
  20. Colachis, S.C., IV.; Bockbrader, M.A.; Zhang, M.; Friedenberg, D.A.; Annetta, N.V.; Schwemmer, M.A.; Skomrock, N.D.; Mysiw, W.J.; Rezai, A.R.; Bresler, H.S.; et al. Dexterous control of seven functional hand movements using cortically-controlled transcutaneous muscle stimulation in a person with tetraplegia. Front. Neurosci. 2018, 4, 208. [Google Scholar] [CrossRef]
  21. Houweling, S.; Beek, P.J.; Daffertshofer, A. Spectral changes of interhemispheric crosstalk during movement instabilities. Cereb. Cortex. 2010, 20, 2605–2613. [Google Scholar] [CrossRef]
  22. Seeber, M.; Scherer, R.; Müller-Putz, G.R. EEG oscillations are modulated in different be-havior-related networks during rhythmic finger movements. J. Neurosci. 2016, 36, 11671–11681. [Google Scholar] [CrossRef] [PubMed]
  23. Erdler, M.; Beisteiner, R.; Mayer, D.; Kaindl, T.; Edward, V.; Windischberger, C.; Lindinger, G.; Deecke, L. Supplementary motor area activation preceding voluntary movement is detecta-ble with a whole-scalp magnetoencephalography system. Neuroimage 2000, 11, 697–707. [Google Scholar] [CrossRef] [PubMed]
  24. Gaur, P.; Pachori, R.B.; Wang, H.; Prasad, G. A multi-class EEG-based BCI classification using multivariate empirical mode decomposition based filtering and Riemannian geometry. Expert Syst. Appl. 2018, 95, 201–211. [Google Scholar] [CrossRef]
  25. Barachant, A.; Bonnet, S.; Congedo, M.; Jutten, C. Multiclass brain-computer interface classifycation by Riemannian geometry. IEEE Trans. Biomed. Eng. 2012, 59, 920–928. [Google Scholar] [CrossRef]
  26. Blankertz, B.; Tomioka, R.; Lemm, S.; Kawanabe, M.; Muller, K.-R. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process. Mag. 2008, 25, 41–56. [Google Scholar] [CrossRef]
  27. Roy, R.; Sikdar, D.; Mahadevappa, M. Chaotic behaviour of EEG responses with an identical grasp posture. Comput. Biol. Med. 2020, 123, 103822. [Google Scholar] [CrossRef]
  28. Sburlea, A.I.; Wilding, M.; Müller-Putz, G.R. Disentangling human grasping type from the ob-ject’s intrinsic properties using low-frequency EEG signals. Neuroimage Rep. 2021, 1, 100012. [Google Scholar] [CrossRef]
  29. Schwarz, A.; Ofner, P.; Pereira, J.; Sburlea, A.I.; Müller-Putz, G.R. Decoding natural reach-and-grasp actions from human EEG. J. Neural Eng. 2018, 15, 016005. [Google Scholar] [CrossRef]
  30. Cutkosky, M.R. On grasp choice, grasp models, and the design of hands for manufacturing tasks. IEEE Trans Robot. Autom. 1989, 5, 269–279. [Google Scholar] [CrossRef]
  31. Feix, T.; Pawlik, R.; Schmiedmayer, H.; Romero, J.; Kragic, D. A comprehensive grasp taxonomy. In Robotics, Science and Systems: Workshop on Understanding the Human Hand for Advancing Robotic Manipulation; 2009; Available online: https://ps.is.mpg.de/publications/feix-rssws-2010 (accessed on 30 March 2023).
  32. Klem, G.H.; Lüders, H.O.; Jasper, H.H.; Elger, C. The ten-twenty electrode system of the international federation. The International Federation of Clinical Neurophysiology. Electroen-Cephalogr Clin. Neurophysiol. Suppl. 1999, 52, 3–6. [Google Scholar]
  33. Delorme, A.; Makeig, S. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics. J. Neurosci. Methods 2004, 134, 9–21. [Google Scholar] [CrossRef] [PubMed]
  34. Sburlea, A.I.; Müller-Putz, G.R. Exploring representations of human grasping in neural, muscle and kinematic signals. Sci. Rep. 2018, 8, 16669. [Google Scholar] [CrossRef] [PubMed]
  35. Chen, W.; Wang, S.; Zhang, X.; Yao, L.; Yue, L.; Qian, B.; Li, X. EEG-based motion intention recognition via multi-task RNNs. In Proceedings of the 2018 SIAM International Conference on Data Mining, San Diego, CA, USA, 3–5 May 2018; pp. 279–287. [Google Scholar]
  36. Ramoser, H.; Müller-Gerking, J.; Pfurtscheller, G. Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans. Rehabil. Eng. 2000, 8, 441–446. [Google Scholar] [CrossRef] [PubMed]
  37. Rivet, B.; Souloumiac, A.; Attina, V.; Gibert, G. xDAWN algorithm to enhance evoked poten-tials: Application to brain-computer interface. IEEE Trans. Biomed. Eng. 2009, 56, 2035–2043. [Google Scholar] [CrossRef]
  38. Prabhat, A.; Khuller, V. Sentiment classification on big data using naive bayes and logistic regression. In Proceedings of the International Conference on Computer Communication and Informatics (ICCCI), Coimbatore, India, 5–7 January 2017; pp. 1–5. [Google Scholar]
  39. Sita, J.; Nair, G. Feature extraction and classification of EEG signals for mapping motor area of the brain. In Proceedings of the International Conference on Control Communication and Computing (ICCC), Thiruvananthapuram, India, 13–15 December 2013; pp. 463–468. [Google Scholar]
  40. Barachant, A.; Bonnet, S.; Congedo, M.; Jutten, C. Riemannian geometry applied to BCI classification. In Latent Variable Analysis and Signal Separation; Vigneron, V., Zarzoso, V., Moreau, E., Gribonval, R., Vincent, E., Eds.; LVA/ICA, Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2010; Volume 6365. [Google Scholar]
  41. Lotte, F.; Congedo, M.; Lécuyer, A.; Lamarche, F.; Arnaldi, B. A review of classification algorithms for EEG-based brain-computer interfaces. J. Neural. Eng. 2007, 4, R1–R13. [Google Scholar] [CrossRef]
  42. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
  43. Rodríguez, J.D.; Pérez, A.; Lozano, J.A. Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 30, 569–575. [Google Scholar] [CrossRef]
  44. Tharwat, A. Classification assessment methods. Appl. Comput. Inform. 2021, 17, 168–192. [Google Scholar] [CrossRef]
  45. Dietterich, T.G. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput. 1998, 10, 1895–1923. [Google Scholar] [CrossRef]
  46. Schwarz, A.; Escolano, C.; Montesano, L.; Müller-Putz, G.R. Analyzing and decoding natural reach-and-grasp actions using gel, water and dry EEG systems. Front. Neurosci. 2020, 14, 849. [Google Scholar] [CrossRef]
  47. Schwarz, A.; Pereira, J.; Lindner, L.; Müller-Putz, G.R. Combining frequency and time-domain EEG features for classification of self-paced reach-and-grasp actions. In Proceedings of the 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 23–27 July 2019; pp. 3036–3041. [Google Scholar]
  48. Agashe, H.A.; Paek, A.Y.; Zhang, Y.; Contreras-Vidal, J.L. Global cortical activity predicts shape of hand during grasping. Front. Neurosci. 2015, 9, 121. [Google Scholar] [CrossRef] [PubMed]
  49. Ameri, A.; Akhaee, M.A.; Scheme, E.; Englehart, K. A deep transfer learning approach to reducing the effect of electrode shift in EMG pattern recognition-based control. IEEE Trans. Neural Syst. Rehabil. Eng. 2020, 28, 370–379. [Google Scholar] [CrossRef] [PubMed]
  50. Combrisson, E.; Jerbi, K. Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J. Neurosci. Methods. 2015, 250, 126–136. [Google Scholar] [CrossRef] [PubMed]
  51. Yahya, N.; Musa, H.; Ong, Z.Y.; Elamvazuthi, I. Classification of motor functions from electroencephalogram (EEG) Signals based on an integrated method comprised of common spatial pattern and wavelet transform framework. Sensors 2019, 19, 4878. [Google Scholar] [CrossRef] [PubMed]
  52. Wesselink, D.B.; van den Heiligenberg, F.M.; Ejaz, N.; Dempsey-Jones, H.; Cardinali, L.; Tarall-Jozwiak, A.; Diedrichsen, J.; Makin, T.R. Obtaining and maintaining cortical hand representation as evidenced from acquired and congenital handlessness. Elife 2019, 8, e37227. [Google Scholar] [CrossRef] [PubMed]
Applsci 13 05728 g001 550
Figure 1. Reaching phase endpoints of performed ADLs. Transitive: (1) drink from a glass [reaching phase endpoint: grasp a glass], (2) receive a call [grasp a telephone], (3) use a book [grasp a book], (4) eat an apple [grasp an apple], (5) put on a hat [grasp a hat], (6) drink from a mug [grasp a mug], (7) remove a key from a lock [grasp a key], and (8) transfer a pencil [grasp a pencil with tripod grip]. Tool-mediated: (9) pour a glass of water [grasp a bottle], (10) make a serve [grasp a racket], (11) brush teeth [grasp a toothbrush], (12) use a laptop [grasp the screen of a laptop for unfolding], (13) draw a line [grasp a pencil with a lateral pinch grip], (14) open a door [grasp a door knob], (15) transfer a tennis ball [grasp a tennis ball], and (16) unscrew a bottle (grasp a bottle top).
Figure 1. Reaching phase endpoints of performed ADLs. Transitive: (1) drink from a glass [reaching phase endpoint: grasp a glass], (2) receive a call [grasp a telephone], (3) use a book [grasp a book], (4) eat an apple [grasp an apple], (5) put on a hat [grasp a hat], (6) drink from a mug [grasp a mug], (7) remove a key from a lock [grasp a key], and (8) transfer a pencil [grasp a pencil with tripod grip]. Tool-mediated: (9) pour a glass of water [grasp a bottle], (10) make a serve [grasp a racket], (11) brush teeth [grasp a toothbrush], (12) use a laptop [grasp the screen of a laptop for unfolding], (13) draw a line [grasp a pencil with a lateral pinch grip], (14) open a door [grasp a door knob], (15) transfer a tennis ball [grasp a tennis ball], and (16) unscrew a bottle (grasp a bottle top).
Applsci 13 05728 g001
Applsci 13 05728 g002 550
Figure 2. Experimental timeline with the confining events “initial movement” and “object grasped” of the reaching phase lasting at least 800 ms. The MGA of the hand for this phase was between 61 and 65%; therefore, for epoching reasons, it was subdivided into the three time windows, T1 to T3, which lasted 500 ms each. MGA = maximum grip aperture.
Figure 2. Experimental timeline with the confining events “initial movement” and “object grasped” of the reaching phase lasting at least 800 ms. The MGA of the hand for this phase was between 61 and 65%; therefore, for epoching reasons, it was subdivided into the three time windows, T1 to T3, which lasted 500 ms each. MGA = maximum grip aperture.
Applsci 13 05728 g002
Applsci 13 05728 g003 550
Figure 3. Combined classifiers with feature extraction methods (gray) and the applied hyper-parameter optimizations (white). Settings of the optimization algorithms were selected based on the multinomial option from the Python library (not necessary for the Riemannian geometry).
Figure 3. Combined classifiers with feature extraction methods (gray) and the applied hyper-parameter optimizations (white). Settings of the optimization algorithms were selected based on the multinomial option from the Python library (not necessary for the Riemannian geometry).
Applsci 13 05728 g003
Applsci 13 05728 g004 550
Figure 4. Scenarios (A–D) exemplified using S12 as the target subject for the fine-tuning of classifiers. The same procedure was applied when using S13 as the target subject by an exchange of S12 data and S13 data in all scenarios, and vice versa.
Figure 4. Scenarios (A–D) exemplified using S12 as the target subject for the fine-tuning of classifiers. The same procedure was applied when using S13 as the target subject by an exchange of S12 data and S13 data in all scenarios, and vice versa.
Applsci 13 05728 g004
Applsci 13 05728 g005 550
Figure 5. Balanced discrimination accuracies (mean ± sd over all grasps) as a function of the grasping posture categorization G6 of the applied classifiers linear regression (LR), linear discriminant analysis (LDA), support vector machine (SVM), LR in tangent space (TS_LR), SVM in tangent space (TS_SVM), Riemann minimum distance to mean (MDM), MDM extended by geodesic filtering (FgMDM), and scenarios A–D from Figure 4 using bandpass-filtered (fδ-γ = 0.5–100 Hz) and epoched (−1000 ms ≤ EL ≤ 500 ms) EEG signals.
Figure 5. Balanced discrimination accuracies (mean ± sd over all grasps) as a function of the grasping posture categorization G6 of the applied classifiers linear regression (LR), linear discriminant analysis (LDA), support vector machine (SVM), LR in tangent space (TS_LR), SVM in tangent space (TS_SVM), Riemann minimum distance to mean (MDM), MDM extended by geodesic filtering (FgMDM), and scenarios A–D from Figure 4 using bandpass-filtered (fδ-γ = 0.5–100 Hz) and epoched (−1000 ms ≤ EL ≤ 500 ms) EEG signals.
Applsci 13 05728 g005
Applsci 13 05728 g006 550
Figure 6. Balanced discrimination accuracies (mean ± sd over all grasps) of the classifiers LR in tangent space (TS_LR) and SVM in tangent space (TS_SVM), and scenarios A–D from Figure 4 as a function of grasping posture categorizations G6, C5, and F4 from Table 1 using bandpass-filtered (fδ-γ = 0.5–100 Hz) and epoched (−1000 ms ≤ EL ≤ 500 ms) EEG signals. The C5 categorization yields significantly better results (p < 0.008).
Figure 6. Balanced discrimination accuracies (mean ± sd over all grasps) of the classifiers LR in tangent space (TS_LR) and SVM in tangent space (TS_SVM), and scenarios A–D from Figure 4 as a function of grasping posture categorizations G6, C5, and F4 from Table 1 using bandpass-filtered (fδ-γ = 0.5–100 Hz) and epoched (−1000 ms ≤ EL ≤ 500 ms) EEG signals. The C5 categorization yields significantly better results (p < 0.008).
Applsci 13 05728 g006
Applsci 13 05728 g007 550
Figure 7. Balanced discrimination accuracies (mean ± sd over all grasps) of the classifiers LR in tangent space (TS_LR) and SVM in tangent space (TS_SVM), and scenarios A–D from Figure 4 trained with the grasping posture categorization C5 as a function of EEG signal frequency bands and epochs EL (−1 s–0.5 s) and ES (−1 s–0 s).
Figure 7. Balanced discrimination accuracies (mean ± sd over all grasps) of the classifiers LR in tangent space (TS_LR) and SVM in tangent space (TS_SVM), and scenarios A–D from Figure 4 trained with the grasping posture categorization C5 as a function of EEG signal frequency bands and epochs EL (−1 s–0.5 s) and ES (−1 s–0 s).
Applsci 13 05728 g007
Table
Table 1. Grasping posture categorizations for multiarticular prosthetic hands (G6), and for the grasping taxonomies according to Cutkosky [29] (C5) and Feix [30] (F4).
Table 1. Grasping posture categorizations for multiarticular prosthetic hands (G6), and for the grasping taxonomies according to Cutkosky [29] (C5) and Feix [30] (F4).
Var.No.Grip PatternADLs
G61Power prismatic grip1, 9, 10
2Power prismatic adducted thumb2, 5
3Power lateral pinch3, 12
4Power circular grip4, 6, 14, 15
5Key pinch grip7, 16
6Precision8, 11, 13
C51Power prehensile lateral pinch3, 12
2Power prehensile circular sphere4, 15
3Power prehensile prismatic wrap1, 9
4Precision circular sphere6, 14
5Precision prismatic thumb 3 finger8
F41Power palm1, 4, 9, 15
2Intermediate side7, 10, 11, 13, 16
3Precision pad 2–5 thumb abduction6, 14
4Precision pad 2–5 thumb adduction3, 12
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Jakubowitz, E.; Feist, T.; Obermeier, A.; Gempfer, C.; Hurschler, C.; Windhagen, H.; Laves, M.-H. Early Predictability of Grasping Movements by Neurofunctional Representations: A Feasibility Study. Appl. Sci. 2023, 13, 5728. https://doi.org/10.3390/app13095728

AMA Style

Jakubowitz E, Feist T, Obermeier A, Gempfer C, Hurschler C, Windhagen H, Laves M-H. Early Predictability of Grasping Movements by Neurofunctional Representations: A Feasibility Study. Applied Sciences. 2023; 13(9):5728. https://doi.org/10.3390/app13095728

Chicago/Turabian Style

Jakubowitz, Eike, Thekla Feist, Alina Obermeier, Carina Gempfer, Christof Hurschler, Henning Windhagen, and Max-Heinrich Laves. 2023. "Early Predictability of Grasping Movements by Neurofunctional Representations: A Feasibility Study" Applied Sciences 13, no. 9: 5728. https://doi.org/10.3390/app13095728

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop