Combining Electromyography and Tactile Myography to Improve Hand and Wrist Activity Detection in Prostheses

Abstract

Despite recent advances in prosthetics and assistive robotics in general, robust simultaneous and proportional control of dexterous prosthetic devices remains an unsolved problem, mainly because of inadequate sensorization. In this paper, we study the application of regression to muscle activity, detected using a flexible tactile sensor recording muscle bulging in the forearm (tactile myography—TMG). The sensor is made of 320 highly sensitive cells organized in an array forming a bracelet. We propose the use of Gaussian process regression to improve the prediction of wrist, hand and single-finger activation, using TMG, surface electromyography (sEMG; the traditional approach in the field), and a combination of the two. We prove the effectiveness of the approach for different levels of activations in a real-time goal-reaching experiment using tactile data. Furthermore, we performed a batch comparison between the different forms of sensorization, using a Gaussian process with different kernel distances.

1. Introduction

Most of the clinically available non-invasive externally-powered hand prostheses are controlled by identifying myoelectric signals corresponding to finger, hand and wrist activations. Surface electromyography (sEMG) sensors allow for capturing the electrical activity of the muscles of the forearm, which is used to estimate the user intent using machine learning methods [1]. Most research efforts in this area have focused on myoelectric prostheses, by exploring the control of one, two or multiple degrees of freedom (DOFs) [2,3,4] or by exploiting hand synergies [5,6]. Even by means of this technology, simultaneous and proportional control of multiple DOFs remains a major challenge [7]; the problem is that even when state-of-the-art machine learning algorithms are used to interpret sEMG data, robust (reliable) hand activity detection is not yet possible in daily living activities with a small number of sEMG sensors.

In order to try to provide a solution to this problem, we propose the exploiting of tactile myography (TMG) to detect hand and wrist movements as a complementary source of information to augment the traditional sEMG sensors. TMG is high-resolution pressure information, related to the deformations induced by muscle activity in the body [8,9]. In [10], a shape-conformable tactile bracelet was developed, comprising 320 sensors (taxels) measuring the bulging of the muscles around the full circumference of the forearm. A preliminary comparison between TMG and sEMG during a real-time target-reaching experiment showed that detection using linear regression applied directly to the data of the bracelet tends to outperform detection using sEMG [11]. In this work specifically, we propose the use of Gaussian process regression (GPR) [12] to improve the intent detection with TMG data and to combine information of the two sources. The choice of GPR is motivated by the fact that only a limited number of samples is needed to train the method. Despite that the data contains the typical type of patterns that could be treated by deep learning strategies, the use of such an approach would require a large training dataset to be efficient. Such a dataset would require many hours of data collection with each amputee, which would result in a non-user-friendly system. As TMG data have a matrix form, we aim to exploit the data structure to learn a controller from few examples. This is achieved using a tensor Gaussian process [13,14], which is an extension of the Gaussian process to matrix data (or data organized as higher-order arrays). We extend this method to multimodal data and perform regression using TMG and sEMG to detect user intents. Our contribution is twofold. First, we validate the use of the Gaussian process for TMG data with a real-time goal-reaching task involving hand and wrist movement detection. Second, we propose a GPR approach to combine multimodal data and compare the use of sEMG, TMG and a combination of both for hand and wrist movement detection. For both experiments, the intended activations were associated with input signal patterns by relying on the values of an animated hand model that the participants had to imitate. This approach could introduce some mismatches compared to the use of a dataglove or a vision system to detect the hand pose. The aim of using this method was to test if such a protocol could be used in future studies with amputees, for whom data cannot be collected by directly measuring the movements of the hand. The paper is organized as follows. Section 2 and Section 3 present the sensors and regression methods used in the experiments. Section 4 describes the experimental protocol. The results are presented and discussed in Section 5 and Section 6. Section 7 concludes the paper.

2. Materials

Muscle activity was acquired using two sensor devices. The first was based on TMG, and the process is known as high-resolution force myography (HD-FMG) or residual kinetic imaging (RKI). This idea was introduced in the prosthetics community by Craelius, Phillips and others [8,9], who demonstrated that this method could intensify patterns of individual finger motions. The second device was based on sEMG, which is the standard detection method in clinical applications. The bracelets were placed on the proximal end of the forearm, at the position with the greatest muscle bulk, as shown in Figure 1a.

2.1. Surface Electromyography Bracelet

Ten Ottobock MyoBock 13E200=50 placed on a Velcro strap, as shown in Figure 1b, were used to gather sEMG data. The sensors were linked to the wireless acquisition device developed in [15], which can gather up to 32 sensors by multiplexing the channels and gathering these through the 12-bit analog-to-digital converter (ADC) of a MSP430 microcontroller that transmitted the data via Bluetooth to the computer. In our case, only 10 sensors were used because of the limited space on the forearm. The functional block representation of the acquisition device is shown by Figure 2. Each off-the-shelf sEMG sensor provided amplification, rectification and filtering on-board. Such sensors are standardly used in prosthetic sockets for myocontrol.

2.2. Tactile Bracelet

The tactile bracelet developed in [10] was composed of 7 to 10 modules, each having $4\times 8$ cells, resulting in a total of 320 input sensors. Each module was linked to the main data acquisition board that gathered the signals with 12-bit precision via serial peripheral interface (SPI) and sent them through a serial port via a mini USB cable (see Figure 3). On each module lay a soft, porose conductive foam whose resistance decreased when pressure was applied. The variable cell resistance was then converted to a voltage value and, after passing through a standard voltage divider, was sent through an ADC to obtain the sensor output. TMG gathers a high-spatial resolution (5 mm) image of the pressure exerted at the surface of the forearm by the volumetric changes of the muscles due to movements. All details regarding the device can be found in [10], while further details regarding the resistive-based approach to TMG employed in the device are available in [16].

3. Methods

3.1. Gaussian Process Regression

Gaussian processes are a class of probabilistic models that define a posterior over functions given a set of input and output data. A dataset of N observations ${\left\{({\mathit{x}}_n,{\mathit{y}}_n)\right\}}_{n=1}^N$ or $\{\mathit{X},\mathit{Y}\}$ is encoded as a joint distribution, which is assumed to be Gaussian with some mean $\mathit{\mu }$ and covariance or kernel matrix $\mathit{K}$, as follows:

$$p\left(\mathit{y}\right|\mathit{X})\sim \mathcal{N}(\mathit{\mu },\mathit{K})$$

(1)

The covariance $\mathit{K}\in {\mathbb{R}}^{N\times N}$ is computed using a kernel function k as a measure of similarity so that ${\mathit{K}}_{ij}=k({\mathit{x}}_i,{\mathit{x}}_j)$. If two input points are similar according to the kernel, the output of the function at those points will also be similar. The predictive distribution of a new output ${\mathit{y}}_{\ast }$ corresponding to a new input ${\mathit{x}}_{\ast }$ can then be inferred as

$$p\left({\mathit{y}}_{\ast }\right|{\mathit{x}}_{\ast },\mathit{X},\mathit{y})\sim \mathcal{N}({\mathit{\mu }}_{\ast },{\Sigma }_{\ast })$$

(2)

where

$$\begin{array}{cc}\hfill {\mathit{\mu }}_{\ast }& =k({\mathit{x}}_{\ast },\mathit{X}){\left(k(\mathit{X},\mathit{X})+{\sigma }^2\mathit{I}\right)}^{-1}\mathit{Y},\hfill \\ \hfill {\Sigma }_{\ast }& =k({\mathit{x}}_{\ast },{\mathit{x}}_{\ast })-k({\mathit{x}}_{\ast },\mathit{X}){\left(k(\mathit{X},\mathit{X})+{\sigma }^2\mathit{I}\right)}^{-1}k(\mathit{X},{\mathit{x}}_{\ast })\hfill \end{array}$$

(3)

3.2. Gaussian Process Regression for Matrix-Valued Data

GPR can be adapted to matrix and higher-order data by adapting the kernel function computation [13]. Given a dataset of N observations ${\left\{({\mathit{X}}_n,{\mathit{y}}_n)\right\}}_{n=1}^N$, the kernel is defined as a product of positive semi-definite factor kernels:

$$k({\mathit{X}}_i,{\mathit{X}}_j)=k({\mathit{X}}_i,{\mathit{X}}_j)k({\mathit{X}}_i^{\top },{\mathit{X}}_j^{\top })$$

(4)

where ${\mathit{X}}^{\top }$ is the transpose of $\mathit{X}$.

3.3. Gaussian Process Regression for Multimodal Data

Given a dataset of N multimodal observations ${\left\{({\mathit{X}}_n^1,\dots ,{\mathit{X}}_n^M,{\mathit{x}}_n^1,\dots ,{\mathit{x}}_n^P,{\mathit{y}}_n)\right\}}_{n=1}^N$, where the inputs are matrix and/or vector-valued data, we propose to define the kernel as the following product:

$$k(\{{\mathit{X}}_i,{\mathit{x}}_i\},\{{\mathit{X}}_j,{\mathit{x}}_j\})=\prod _{m=1}^{M}k({\mathit{X}}_i^m,{\mathit{X}}_j^m)\prod _{p=1}^{P}k({\mathit{x}}_i^p,{\mathit{x}}_j^p)$$

(5)

3.4. Kernels

We consider factor kernels in the form of radial basis function (RBF) kernels defined as

$$k({\mathit{x}}_i,{\mathit{x}}_j)=\exp \left(-\frac{d({\mathit{x}}_i,{\mathit{x}}_j)}{2{\beta }^2}\right)$$

(6)

where $d({\mathit{x}}_i,{\mathit{x}}_j)$ is a distance measure and $\beta$ is a scaling parameter defining the variance of the Gaussian. We investigated different distance measures for matrix and vector-valued data, namely

• the Kullback–Leibler (KL) divergence $d_{\mathrm{KL}}=\mathrm{KL}\left(p\left({\mathit{X}}_i\right|{\mathit{\mu }}_i,{\Sigma }_j)\parallel q\left({\mathit{X}}_j\right|{\mathit{\mu }}_j,{\Sigma }_j)\right)$ by treating all matrix data as a Gaussian generative model [14];
• two different distances for covariance or symmetric positive definite (SPD) matrices, $d_{\mathrm{logSPD}}={\parallel \ln \left({\Sigma }_i\right)-\ln \left({\Sigma }_j\right)\parallel }_{\mathrm{F}}$ and $d_{\mathrm{SPD}}={\parallel {\Sigma }_i-{\Sigma }_j\parallel }_{\mathrm{F}}$, where $\Sigma =\mathit{X}$ if $\mathit{X}$ is a SPD matrix and $\Sigma =\mathrm{cov}\left(\mathit{X}\right)$ otherwise [17];
• the Euclidean distance $d_{\mathrm{Eucl}}=\parallel \mathrm{vec}\left({\mathit{X}}_i\right)-\mathrm{vec}\left({\mathit{X}}_j\right)\parallel$ for matrices and $d_{\mathrm{Eucl}}=\parallel {\mathit{x}}_i-{\mathit{x}}_j\parallel$ for vectors.

4. Experiments

4.1. Participants

Nine non-amputated right-handed participants joined both experiments (two females and seven males, aged $31\pm 6$ with a minimum age of 24 and a maximum age of 45 years old). None of the participants had medical issue that could interfere with the measurements. Moreover, three participants knew the material from previous experiments. The procedure was thoroughly explained to each participant before the trial, and it was conducted according to the World Medical Association (WMA) Declaration of Helsinki; informed consent was obtained from all the participants in written form.

4.2. Experimental Setup

For both experiments, each participant was asked to sit comfortably in a chair in front of a monitor. The tactile and sEMG bracelets were fitted on the upper part of their right forearm, the tactile bracelet being further up than the sEMG sensors, as shown in Figure 1a. Ground truth was obtained by using the values of an animated visual stimulus in the form of a 9-DOF hand model. The stimulus was displayed on the monitor. This method relies on the assumption that the participant replicates the hand model movements with reasonable accuracy. For a study with non-amputees, it has the drawback of possibly reducing the precision in the prediction of the intent detection as a result of the delay required by the participant to adapt. However, as we forecasted future studies to be conducted with amputees, we could not collect data using precise systems such as motion capture or a position-sensing glove, as these involve directly measuring the movement of the hand. Because ground truth data cannot be collected on amputees, the approach described above was required to associate intended activations with input signal patterns.

4.3. Experiment 1: Real-Time Goal-Reaching Task with Tactile Myography

This experiment aimed to bring the analyses of TMG data performed in [11,18] one step further by applying a non-linear regression method to achieve real-time detection.

During the first part of the experiment, namely the training part, the participants were asked to imitate the movements of an animated hand model (see Figure 4). They were presented the following sequence of movements three times: wrist flexion, wrist extension, wrist supination, thumb flexion, index flexion and little-finger flexion. Wrist and finger movements were executed in the air and by pressing fingers on the table, respectively. Each stimulus followed a cycle of 14 s, consisting of a transition phase (2 s), an activation phase (6 s), a transition phase (2 s) and a relaxing phase (4 s). The data were captured only during the activation and rest phases, and the transition phases allowed the participant to reach full activation and rest, respectively. Data acquired from the tactile bracelet and the visual stimulus were synchronized by linearly interpolating the samples of each channel. TMG data were low-pass filtered (first-order Butterworth filter with cut-off frequency of 1 Hz) to remove high-frequency disturbances. We then trained two different regression models, namely, ridge regression (RR) and GPR, using a RBF kernel with a Euclidean distance. The GPR parameter $\beta$ was optimized for each participant using cross-validation on training data. Each training dataset was randomly shuffled; then, 40% was used to train each model, and the test was performed on the remaining 60%. This procedure was repeated 10 times with a different random shuffle each time. The obtained $\beta$ values ranged between 1 and 4 depending on the participant. The final model was trained on the whole training set with the optimized parameters.

The second part of the experiment consisted of a real-time goal-reaching task, in which the participants controlled a second hand model. For each method, 54 tasks were administrated to the participants (6 movements used for training at 3 activation levels, repeated 3 times). Half of the participants began with RR, while the other half began with GPR. For each method, the participants experienced the control of the virtual hand without constraints before beginning the goal-reaching tasks. For each task, the first hand model would perform a movement to either one-third, two-thirds or full activation. The participants controlled the second hand model and were asked to have it mimic the action of the first. Performing the detection for several activation levels allowed us to determine whether the method was adapted for proportional control. A task was considered successful if the participant could maintain the desired movement at the desired activation level for 1.5 s. The movement was considered as maintained as long as the difference between the first and the second hand model was below a fixed threshold. If the task was not successfully fulfilled after 15 s, it was considered to have failed. A visual marker (smiling or sad face) appearing on the monitor indicated the result of each task (see Figure 5). A video of this experiment can be found in the supplementary material (Video S1).

4.4. Experiment 2: Combination of Electromyography and Tactile Myography

With this experiment, we extended the preliminary comparison between sEMG and TMG performed in [11] with a comparison between sEMG, TMG and both inputs considered together. Moreover, we compared a simple linear regression method (RR) and GPR using a RBF kernel with different measures of distance. The data collection process was similar to that for the training part of the first experiment. The participants, wearing the tactile and sEMG bracelets, were asked to imitate movements of an animated hand model. They were presented the same sequence of six movements three times. The main difference to the previous experiment was that the data were recorded during the whole cycle of each stimulus in order to obtain the whole range of activation from rest to complete finger and wrist movement. The transition phases were extended from 2 to 5 s to allow the participants to follow the model more precisely.

Data from the TMG, sEMG sensors and visual stimulus were synchronized by linearly interpolating the samples of each channel. This was possible because each and every sample gathered from the input devices, as well as the stimulus values, were accurately timestamped using the computer’s performance counter. TMG and sEMG data were low-pass filtered (first-order Butterworth filter with cut-off frequency of 1 Hz) to remove high-frequency disturbances. We selected the training dataset as data recorded at zero and full activation and the testing dataset as data in the transition parts, corresponding to the whole range of intermediate activation levels. This choice was made in order to evaluate the performance of the models applicable to amputees in forecasted future studies, as they cannot provide accurate intermediate training data. For each participant, the RR and GPR models were trained on the training data using inputs from sEMG, TMG or both. The following distance measures were used in the RBF kernels of GPR:

• sEMG signal as input: $d_{\mathrm{Eucl}}$, $d_{\mathrm{SPD}}$ and $d_{\mathrm{logSPD}}$;
• TMG signal as input: $d_{\mathrm{Eucl}}$, $d_{\mathrm{SPD}}$, $d_{\mathrm{logSPD}}$ and $d_{\mathrm{KL}}$;
• sEMG and TMG signals as input: all combinations of $d_{\mathrm{Eucl}}$, $d_{\mathrm{SPD}}$ and $d_{\mathrm{logSPD}}$ for sEMG coupled with $d_{\mathrm{Eucl}}$, $d_{\mathrm{SPD}}$ and $d_{\mathrm{KL}}$ for TMG.

In the case of sEMG data, the two distances $d_{\mathrm{SPD}}$ and $d_{\mathrm{logSPD}}$ were computed for data arranged in spatial covariances with a 400 ms sliding window and an increment of 40 ms. The parameters of each model were selected by cross-validation of the training data. The applied cross-validation process was the same as for the first experiment. The final model was trained on the whole training set with the optimized parameters and tested on data corresponding to the whole range of intermediate activation levels for each participant. A video of this experiment can be found in the supplementary material (Video S2).

5. Results

5.1. Experiment 1: Real-Time Goal-Reaching Task with Tactile Myography

We compared the performance of RR and GPR ($d_{\mathrm{Eucl}}$) by computing the ratio of successful tasks (success rate—SR), the time needed to accomplish successful tasks (time to complete task—TCT) and, in the case of unsuccessful tasks, the time spent within the goal (time in the target—TIT). We also computed the root-mean-square error (RMSE) between the ground truth and the prediction for both methods.

Table 1. Performance comparison in term of success rate (SR), time to complete the task (TCT) and time in the target (TIT) between Gaussian process regression (GPR) and ridge regression (RR) in experiment 1.

Regression Method	SR (%)	TCT (s)	TIT (s)	RMSE
GPR ($d_{\mathrm{Eucl}}$)	$55.35\%\pm 17.28\%$	$4.58\pm 3.02$	$0.13\pm 0.46$	$0.58\pm 0.08$
RR	$35.60\%\pm 12.24\%$	$5.56\pm 3.46$	$0.23\pm 0.56$	$0.78\pm 0.12$

shows the average and the standard deviation of the mentioned results. Figure 6 and Figure 7 show evaluations in terms of SR, and in terms of TCT and TIT, respectively. The SR, TCT and TIT for each participant are available in Appendix A (Figure A1 and Figure A2).

A paired-samples Student’s t-test showed a statistically significant difference between the SR of the two methods ($p<0.05$). The performance of GPR outperformed the performance of RR by ∼20%. Moreover, we observe that the TCT of GPR was reduced by ∼1 s compared to RR. However, the TIT obtained with RR was slightly shorter, although the mean TIT was small for both methods, with a high standard deviation compared to the average value. As expected given the different SR, TCT and TIT values, GPR outperformed RR in terms of the RMSE.

Figure 8 shows the RMSE for each finger and wrist movement that the participants had to execute for both regression methods. We observe that GPR outperformed RR for all movements, especially for wrist flexion and supination. Moreover, little-finger flexion was the most difficult movement to predict for both methods. Figure 9 shows the RMSE computed for each activation level for both regression methods. The RMSE obtained with GPR was smaller and had a smaller variance than the RMSE obtained with RR. We observed a slight increase of the RMSE with the activation level. This may have been due to interferences between the DOFs that are more important for a higher level of activation; for example, flexing one finger completely tends to induce flexion of the other fingers, especially for a complete flexion.

5.2. Experiment 2: Combination of Electromyography and Tactile Myography

We compared the performance of GPR with different distance measures and RR, with sEMG, TMG or both as the input, by computing the RMSE between the ground truth and the predictions.

Table 2. Performance in term of root mean square error (RMSE) of ridge regression (RR) and Gaussian process regression (GPR) with different distances using surface electromyography (sEMG) or tactile myography (TMG) as input for the regression model.

sEMG	RMSE		TMG	RMSE
GPR ($d_{\mathrm{Eucl}}$)	$0.69\pm 0.12$		GPR ($d_{\mathrm{Eucl}}$)	$0.31\pm 0.05$
GPR ($d_{\mathrm{logSPD}}$)	$0.46\pm 0.05$		GPR ($d_{\mathrm{logSPD}}$)	$0.31\pm 0.05$
GPR ($d_{\mathrm{SPD}}$)	$0.58\pm 0.05$		GPR ($d_{\mathrm{SPD}}$)	$0.32\pm 0.06$
			GPR ($d_{\mathrm{KL}}$)	$0.38\pm 0.12$
RR	$0.47\pm 0.06$		RR	$0.41\pm 0.23$

and

Table 3. Performance in term of root mean square error (RMSE) of ridge regression (RR) and Gaussian process regression (GPR) with different distances combining surface electromyography (sEMG) and tactile myography (TMG) as input for the regression models.

	sEMG	${\mathit{d}}_{\mathbf{Eucl}}$	${\mathit{d}}_{\mathbf{logSPD}}$	${\mathit{d}}_{\mathbf{SPD}}$
TMG
${\mathit{d}}_{\mathbf{Eucl}}$		$0.29\pm 0.05$	$0.31\pm 0.05$	$0.31\pm 0.06$		RR	$0.39\pm 0.25$
${\mathit{d}}_{\mathbf{SPD}}$		$0.31\pm 0.05$	$0.32\pm 0.06$	$0.32\pm 0.06$
${\mathit{d}}_{\mathbf{KL}}$		$0.31\pm 0.06$	$1.15\pm 2.08$	$2.02\pm 5.01$

show the average and the standard deviation of the RMSE for the participants, for unimodal (sEMG or TMG) and multimodal inputs, respectively. These results are also reported in Figure 10.

We observe that TMG outperformed sEMG for all the regression methods in the case of a unimodal input. Paired-samples Student’s t-tests applied to errors of any combination of methods showed statistically significant differences between sEMG and TMG ($p<0.05$). Only GPR using the distance $d_{\mathrm{logSPD}}$ improved the prediction using sEMG input compared to RR. In the case of TMG, GPR outperformed RR for all distances; the better predictions were obtained using $d_{\mathrm{logSPD}}$ and $d_{\mathrm{Eucl}}$. However, we chose not to use $d_{\mathrm{logSPD}}$ for TMG input in the multimodal models, as its computation time was greatly increased compared to $d_{\mathrm{SPD}}$ and both distances were similar to each other. Although GPR using $d_{\mathrm{KL}}$ had the higher averaged RMSE value, it performed better than GPR using other distances for two participants (2 and 6). The RMSE for each participant is available in Appendix A (Figure A3 and Figure A4 for sEMG and TMG inputs, respectively).

RR using both sEMG and TMG as the input outperformed unimodal RR. A paired-samples t-test showed a statistically significant difference between errors obtained with TMG and with sEMG coupled with TMG using RR ($p<0.05$). However, in the case of GPR, the only combination of sEMG and TMG improving the RMSE compared to TMG was to use $d_{\mathrm{Eucl}}$ for both inputs. The scores of the other combinations of distances were similar to those obtained using only TMG. The high RMSE value of $d_{\mathrm{KL}}$ for TMG combined with $d_{\mathrm{logSPD}}$ or $d_{\mathrm{SPD}}$ for sEMG was due to the large error obtained for three participants (5, 6 and 7). The RMSE remained close to the RMSE of other distances for the rest of the participants. The RMSE values for all the participants are available in Appendix A (Figure A5).

Figure 11 shows the RMSE for each finger and wrist movement that the participant had to execute for GPR with different combinations of distances. We observe that all the methods had a high and low RMSE for the same movements. Wrist supination had the best RMSE, while index flexion and wrist extension seemed to be the most difficult to detect.

Table 4. Computation (comp.) time for the different kernel distances using surface electromyography (sEMG) or tactile myography (TMG) as input for the regression model. The time to compute training distances is the time needed to compute the distances between all training data points. The time to compute testing distances is the time needed to compute the distance of a new test data point with all training data points.

sEMG	Training Distances Comp. Time (s)	Testing Distances Comp. Time (s)
$d_{\mathrm{Eucl}}$	$0.37\pm 0.01$	$0.001\pm 0.001$
$d_{\mathrm{logSPD}}$	$34.18\pm 0.96$	$0.046\pm 0.001$
$d_{\mathrm{SPD}}$	$2.75\pm 0.1$	$0.006\pm 0.001$
TMG	Training Distances Comp. Time (s)	Testing Distances Comp. Time (s)
$d_{\mathrm{Eucl}}$	$1.35\pm 0.06$	$0.003\pm 0.000$
$d_{\mathrm{SPD}}$	$1.94\pm 0.06$	$0.077\pm 0.007$
$d_{\mathrm{KL}}$	$49.31\pm 2.1$	$0.184\pm 0.010$

shows the time to compute all the training distances between training data points and the time to compute the distances between a new test data point and the training data points. Computation times are shown for distances applied to sEMG and TMG data separately. These had to be added when both sensors were used as the input. The computation of these distances caused the difference in computation times for GPR using different kernel distances. All methods were programmed in Matlab and were run on a 3.5 GHz six-core CPU. We observe that $d_{\mathrm{Eucl}}$ was the fastest to compute for both input sensors; $d_{\mathrm{logSPD}}$ and $d_{\mathrm{KL}}$ were more expensive than other distances in the case of sEMG and TMG data as input, respectively.

6. Discussion

In the first experiment, we showed that GPR outperformed RR to detect hand and wrist movements at different activation levels in terms of the SR, TCT and RMSE. GPR and RR obtained similar TIT values. These results confirm the batch analysis presented in [18] with GPR and RR on zero and full activation values. In comparison with the preliminary real-time experiment using RR presented in [11], we obtained a ∼20% lower SR, a ∼1 s lower TIT and similar TCT values. The lower SR and TIT results may have been explained by the fact that we tested individual finger movements that are usually more difficult to detect and can be confounded with some wrist movements. Furthermore, we often observed difficulties in discriminating index- and little-finger flexion and finger and wrist flexion during the real-time experiment, especially using RR. This often results in the activation of several DOFs rather than a unique DOF; for example, the index was often flexed along with the little finger during tasks involving only the little finger. Most failed tasks seemed to be due to this kind of interaction, which induced a large difference between the poses of the hand model and the controlled hand. It also induced a zero TIT for these failed tasks. Thus, the presence of many outliers in Figure 7 is explained by the fact that only a minority of failed tasks had a non-zero TIT. Difficulties in discriminating index- and little-finger flexion and finger and wrist flexion may also explain the slightly higher RMSE observed for those movements.

In the second experiment, we confirmed that TMG outperformed sEMG using RR, and we showed that the performance was also improved using GPR. The first assessment is in line with the comparison presented in [11], and the second confirms the results from [18] and from our first experiment. The RMSE values obtained with TMG in this experiment were slightly lower than those of the previous experiment. This may have been explained by the fact that the second experiment was performed in batch and because training and testing data are parts of a continuous set of data (i.e., in the whole dataset, zero and full activation are considered as training data, while transitions are considered as testing data).

The RMSE values we obtained showed only a small improvement for certain methods when combining sEMG and TMG compared to TMG. The RMSE of GPR with different combinations of distances were similar, except those for which the KL divergence was used for TMG data that performed very poorly for some participants. When combining sEMG and TMG, the Euclidean and SPD distances may have been preferred over the log SPD distance for sEMG, as the latter had a longer computation time and did not seem to improve the detection compared to other distances. Moreover, the Euclidean and SPD distances may have been preferred to the KL divergence for TMG data, as they were more stable over the participants and required less time to be computed. Similarly to the first experiment, the RMSE values computed for each movement suggest that wrist supination is the easiest to detect. Overall, the comparison between TMG and combined sEMG and TMG in the second experiment was performed in batch, and we are not aware of any such comparison in the literature. Thus, we consider the results of this comparison as preliminary. Moreover, the choice of RBF kernels with different distances was motivated by the fact that these are known to work well in practice and are easy to tune. However, it would be interesting to test GPR with other kernels, for example, with Matérn kernels. Further analysis and real-time evaluations are required and will be performed in future work.

Two further considerations must be put forward. Firstly, we used the stimulus values as ground truth, because of the impossibility of obtaining reliable data from amputees—a technique that was has already been introduced multiple times in literature [19,20]. That ground truth does not exactly match the “intent” of the subjects is an unavoidable drawback of this approach, and its solution relies on the adaptation of the subjects themselves to the visual stimuli. This fact has not been greatly investigated as far as we know, and it has a clear impact on the practical usability of all systems, such as that presented in this paper. Secondly, and even more importantly, it has even been shown [21,22] that offline accuracy hardly reflects practical usability. Therefore, a deeper investigation of the proposed technique in a real-sized daily living setup is necessary.

7. Conclusions

We presented a real-time goal-reaching experiment using TMG data as the input, by comparing a linear and a non-linear regression method (RR and GPR) to detect different levels of activation in finger and wrist movements. We proposed a GP-based regression approach to combine multimodal inputs and compared the efficiency of sEMG, TMG and a combination of the two in a batch analysis performed on data collected during a second experiment. We then compared the use of different RBF kernel distances for GPR applied to these data.

Results of both real-time and batch experiments showed that the information obtained from 320 sensors of the tactile bracelet could effectively be used to detect graded activation levels of finger and wrist movements. We confirmed that TMG outperformed sEMG to achieve proportional and simultaneous control of several DOFs. Furthermore, the performance of TMG is increased using GPR compared to a simple linear regression method, namely, RR.

The results of the comparison between TMG and the combination of both TMG and sEMG should be considered as preliminary and suggest that combining both sensors could improve the detection using RR and GPR with certain kernels. In future work, we plan to perform a comparison in the form of real-time goal-reaching experiments and to further analyze the potential use of multimodal inputs.