2.1. Dataset
We used a publicly available dataset called Non-Invasive Adaptive Prosthetics (Ninapro) to validate the proposed scheme. To date, a total of ten datasets (DB1~DB10) have been curated, comprising physiological metrics from 180 trans-radial amputees and intact subjects (i.e., electromyographic signals, kinematics, inertial measurements, etc.).
As shown in
Figure 1, the devices used in this dataset to record sEMG signals and motion data of the hand were the OttoBock myograph and the CyberGlove II data glove, respectively. The OttoBock consists of 10 MyoBock 13 E200-50 electrodes, of which 8 electrodes were placed uniformly underneath the elbow joint, and the remaining 2 electrodes were placed on the flexor and the extensor muscles. The raw sEMG signals were also amplified, filtered, and rectified. The CyberGlove II uses specialized resistive bending sensing technology to accurately detect 22 angle data points in real time using 22 sensors. The sEMG and angle data were both sampled at a frequency of 100 Hz [
27,
28].
Reference [
21] used data from eight participants in DB2 at random to estimate the continuous motion of the grasping action, and Reference [
29] only used data from the first five subjects in DB5 to complete the finger joint angle estimation. This is because selecting a small number of subjects that include as much information about all the subjects as possible, rather than using all subjects’ data directly, reduces the computational cost of model training. Therefore, we selected the first eight subjects from Ninapro DB1, and
Table 1 provides some detailed information about these subjects. The wrist, as a complex joint in the upper limb of the human body that connects the hand and forearm, is responsible for performing various fine motor activities in daily life. Therefore, we selected three types of movements closely related to the wrist from DB1, as shown in
Figure 2. These movements included wrist flexion (WF), wrist ulnar deviation (WUD), and wrist extension closed hand (WECH).
Figure 3 illustrates the target angles selected to effectively capture angle data for these three movement types.
2.2. Pre-Processing
The sampling frequency of the sEMG signal and kinematic data of the Ninapro DB1 was 100 Hz. In addition, because the original signal has been filtered and rectified, we did not need to repeat the operation of the data set to directly apply it. Considering the strong nonlinearity and short-time validity of sEMG, it is necessary to segment the sEMG data and extract the feature information of each part. Specifically, we divided the data for all subjects in DB1 into windows of with a length of 100 (0.1 s) with 10 (0.01 s) overlaps, and the next data point k + 1 of the last data point k of the recording window was recorded as the observed value for that window.
The time domain method, frequency domain method, and time–frequency domain method are the most commonly utilized techniques for sEMG analysis. Reference [
29] chose six features that can be classified into temporal and frequency domains. The time domain group consisted of Mean Absolute Value (MAV), Root Mean Square (RMS), and Variance (VAR). In comparison, the frequency domain group consisted of Mean Power (MNP), Mean Frequency (MNF), and median frequency (MDF). Every feature in each domain was used independently. The ideal features for their study were then identified by comparing the performance of each feature. We tested the six features mentioned in their work using our experimental settings, and the results are shown in
Table 2. The frequency domain features performed slightly better, but required threefold more training time. The
RMSE and
R2 scores of MAV and RMS were in the middle-upper level among the six features, while their training times were the shortest. In terms of continuity and real time capability, the time-domain features performed better than the frequency domain and time-frequency domain features [
30]. We ended up using two time-domain features representing muscle fatigue and activity levels, MAV and RMS, as the feature for extraction. This reduces the time spent on feature extraction while ensuring that complete information is extracted. These are the two best features that best represent the specific characteristics that the author used [
21,
31]. The equations are as follows:
and
2.4. The TCN-LSTM Hybrid Model
A TCN is essentially a special modification of the CNN architecture to make a convolutional neural network that is more suitable for time series tasks. Currently, TCNs are widely applied and have achieved significant results in various research fields such as wind power forecasting and traffic estimation [
32,
33]. However, their application in joint angle prediction is relatively rare. In our study, we utilized a TCN to extract deep information from sEMG signals to enhance wrist angle prediction. The main characteristics of the TCN network include three key features: causal convolution, dilated convolution, and residual block [
19].
Causal convolution strictly follows the law of causation. The output
at time
t of layer
k depends only on the data
~
before time
t of layer
k − 1, so no information about the future leaks into the past. For a one-dimensional input sequence
and a convolution filter
, the causal convolution at
xt can be defined as
However, the length of the history sequence processed by causal convolution is limited, and when processing tasks with longer history records, multiple layers or large convolution cores are required to increase the convolution’s receptive field, increasing the computational burden. Based on this, researchers proposed using dilated convolution to address the problem [
34]. Dilated convolution involves skipping some input elements to enable filters to be applied over a larger region than their actual size. The larger receptive field means that more historical data can be captured with fewer parameters and fewer hidden layers [
35]. As illustrated in
Figure 3, larger filters were generated from the original filter by introducing zeros. This can be mathematically expressed using the following formula:
where
d is the dilation rate, which usually takes a value of 1, 2, 4, 8, etc. (2 to the power) and
is the convolution kernel size. The architecture of a dilated causal convolution is shown in
Figure 4.
When dealing with complex prediction tasks such as long sequences, this particular convolutional structure requires continuous updating of weight parameters, leading to an increase in the computational cost of the network. Therefore, the residual connection is used to replace the last convolution layer in the TCN, and the deep features extracted by the TCN are fused with the original manual features to realize the cross-layer transmission of the network, which not only reduces the operating cost of the network but also effectively solves the problems of gradient disappearance and explosion [
36]. This structure is shown in
Figure 5.
LSTM is a kind of network that can capture long-term dependencies in time series by modifying hidden layer neurons based on a RNN [
37,
38]. This advantage is contingent upon the cell state and the gated structure. In
Figure 6, the fundamental composition of the hidden layer in LSTM is illustrated.
represents the activation vector,
signifies the hidden state, and the gating structure encompasses an input gate that regulates the information to be added to the cell state at the current time. Additionally, there is a forget gate that determines which information from the cell state at a prior moment should be discarded, and a hidden output gate that governs the information from the cell state to be transmitted to the next moment [
39,
40]. The update of the LSTM unit at time step j can be described as
where
represents the sigmoid function,
represents the hyperbolic tangent function,
represents the connection between
and
, and
,
,
, and
represent the activation vectors of the input gate, the forgetting gate, the cell alternative value, and the output gate, respectively.
represents element-by-element multiplication, and
,
,
, and
are the weight matrices on each gate, respectively, to map the input data and previously hidden states to the activation value of the opposite door.
,
,
, and
are the corresponding paranoia vectors that adjust the activation threshold of the gate.
The integration of a TCN and LSTM offers a synergistic approach to sEMG signal processing. TCNs excel at capturing intricate deep features from sEMG signals, while LSTM effectively handles temporal dependencies within the input data. By combining these strengths, the proposed TCN-LSTM hybrid model facilitates nuanced feature extraction at both the global and local levels. Moreover, LSTM’s regularization capabilities mitigate overfitting concerns arising from excessive TCN layers, endowing the model with enhanced generalization capacity and bolstering its stability and robustness. The TCN-LSTM model is anticipated to perform exceptionally well in wrist angle prediction tasks owing to a number of crucial aspects, the details of which are discussed below.
Firstly, the long-term dependencies and inherent complex patterns in sEMG data are effectively managed by capturing features at different time scales. Additionally, by integrating TCN and LSTM modules, smooth feature extraction can be performed at both the global and local levels of sEMG data, thus facilitating a comprehensive feature learning capability. Furthermore, the model performs splicing of deep and handcrafted features, which improves the capacity to generate expressive outputs. In summary, this fusion not only strengthens the model’s capacity to interpret sEMG data, but also makes it more capable of handling complex sequence patterns and noise interference.
Figure 7 illustrates the comprehensive workflow of the proposed model. Initially, the sEMG signal’s manual features undergo a thorough analysis, employing the TCN in the first step. This involves utilizing a TCN architecture with three layers, with expansion coefficients of 1, 2, and 4, and employing a filter size of 15.
Next, the extracted deep features are fused with the manual features through a spliceover process in the second step. Subsequently, the LSTM regression model is constructed in the third step. Specifically, the LSTM architecture is configured with a 5-layer structure featuring 50 hidden units. Finally, the regression results undergo processing through a linear layer to obtain the ultimate prediction outcomes. The main parameters included a batch size of 32, an epoch of 50, an initial learning rate of 0.01, and a 50% reduction in the learning rate after 10 rounds of training.