Prediction of Upper Limb Action Intention Based on Long Short-Term Memory Neural Network

Abstract

The use of an inertial measurement unit (IMU) to measure the motion data of the upper limb is a mature method, and the IMU has gradually become an important device for obtaining information sources to control assistive prosthetic hands. However, the control method of the assistive prosthetic hand based on the IMU often has problems with high delay. Therefore, this paper proposes a method for predicting the action intentions of upper limbs based on a long short-term memory (LSTM) neural network. First, the degree of correlation between palm movement and arm movement is compared, and the Pearson correlation coefficient is calculated. The correlation coefficients are all greater than 0.6, indicating that there is a strong correlation between palm movement and arm movement. Then, the motion state of the upper limb is divided into the acceleration state, deceleration state and rest state. The rest state of the upper limb is used as a sign to control the assistive prosthetic hand. Using the LSTM to identify the motion state of the upper limb, the accuracy rate is 99%. When predicting the action intention of the upper limb based on the angular velocity of the shoulder and forearm, the LSTM is used to predict the angular velocity of the palm, and the average prediction error of palm motion is 1.5 rad/s. Finally, the feasibility of the method is verified through experiments, in the form of holding an assistive prosthetic hand to imitate a disabled person wearing a prosthesis. The assistive prosthetic hand is used to reproduce foot actions, and the average delay time of foot action was 0.65 s, which was measured by using the method based on the LSTM neural network. However, the average delay time of the manipulator control method based on threshold analysis is 1.35 s. Our experiments show that the prediction method based on the LSTM can achieve low prediction error and delay.

1. Introduction

The disabled are a special group in contemporary society. Physical defects have brought many inconveniences to their lives. In order to make up for the missing upper limbs of the handicapped and improve their self-care ability, upper limb prostheses are used to replace part of the functions of the lost limbs [1,2]. For the control of upper limb prostheses, predicting the user’s movement intention is as important as identifying the type of action of the upper limb. Action recognition usually focuses on the complete action performed by the upper limb. It is the result of doing the action, such as drinking water, putting on shoes, and brushing teeth [3]. In contrast, intent prediction not only identifies the types of actions performed by users, but also focuses on how to perform these actions. It is the process from “what to do” to “how to do it” [4]. Intention prediction is not only applied in the field of disability; it also plays an important role in the field of rehabilitation and healthcare [5,6,7].

In the human body, information sources that can be used to control the upper limb prosthesis mainly include electrophysiological signals and mechanical signals [8]. The IMU has become one of the main ways to measure mechanical signals. It can be used to measure the acceleration, angle, and other dynamic information of limbs. With the increasing perfection and popularization of wearable sensors, IMUs have developed widely in the field of action recognition and intent prediction [9,10]. Fuan et al. used inertial sensors to design a human action recognition system and achieved 95% accuracy [11]. Tong et al. put inertial sensors on the hands of patients with Parkinson’s disease to capture the acceleration of the wrist. In addition, the neural network model is used to identify hand tremors to achieve symptom recognition [12].

The use of the IMU to control assistive prosthetic hands has become a hot research topic. A key problem in controlling the assistive prosthetic hand is choosing an appropriate machine learning algorithm. In the face of large amount of data, machine learning algorithms can improve the efficiency of recognition, to a certain extent. After learning from a large amount of data, machine learning algorithms can identify and predict current activities based on new observation data. Liu et al. have designed a motion prediction system using the IMU. Based on the acceleration and angle data, a support vector machine is used to classify motion patterns with an average accuracy of 94.25% [13]. Yeaser et al. has proposed a classification method for predicting rollator user intent using the data collected by the IMU, and the KNN classification algorithm achieved 92.9% accuracy [14]. Although classifiers, such as SVM and KNN, can achieve a high action recognition rate, they do not have the function of memorizing long historical information, and are unsuccessful in experiments predicting human motion. Of course, there are some machine learning algorithms that can achieve better data predictions. Altan et al. have developed a new hybrid wind speed prediction model based on the LSTM network and the gray wolf optimizer decomposition method. The resulting model can capture the nonlinear characteristics of wind speed time series and has better predictive performance than a single prediction model, in terms of accuracy [15]. In the financial market, in order to make very high-precision price predictions for digital currencies, they have developed a new hybrid prediction model based on the LSTM neural network, empirical wavelet transform decomposition, and the cuckoo search algorithm. This hybrid model can capture digital, nonlinear properties of monetary time series [16].

Generally, in addition to the requirement of high accuracy for controlling the assistive prosthetic hand, the following requirements also need to be met to measure the quality of the control effect: (1) low prediction delay; and (2) the achievement of a smooth and continuous transition between different activities [17]. The essence of human activity data is time series data. That is, subsequent data have a certain correlation with previous data [18]. Therefore, after training the neural network with a large amount of data, based on the previous observation data, it is helpful to predict the change trend of the subsequent data, in order to achieve a smooth transition between different activities. In order to meet the requirement of low prediction delay, in the process of controlling the assistive prosthetic hand, it is necessary to predict the motion of the upper limb according to the observation data collected in real time. As time goes on, the volume of observational data is also increasing. Some machine learning algorithms need to analyze a complete observation sequence before making an action prediction, and the wide length of observation data will increase the running time of the algorithm, resulting in a high control delay. Examples of this are recurrent neural networks (RNN) and dynamic time warping algorithms. Although these algorithms can achieve high recognition accuracy, they are useless for solving low-latency problems [19,20].

In order to solve the problem of some algorithms not having the function to memorize long historical information, this paper designs a prediction model of action intention. With the memory function of the LSTM neural network, the LSTM is used to predict the motion data of the upper limbs, so as to reduce the control delay of the manipulator. Therefore, this paper aims to reduce the delay in controlling the assistive prosthetic hand, and a new method for predicting the action intention of the upper limb is proposed. This method can be used to predict the hand movement of the assistive prosthetic hand when the user completes a foot action. By predicting the angular velocity of the hand to judge the motion of the upper limb, the delay of controlling the prosthetic hand can be reduced, to some extent. This paper concerns healthy people performing necessary foot actions in daily life, including putting on shoes, putting on socks, and tying shoe laces. The IMU is used to collect the angular velocity data of the upper limb. Based on the motion data of the healthy people’s upper limbs, the correlation between palm movement and arm movement is analyzed while they perform foot movements. Based on the motion data of the arm, the LSTM network is used to predict the motion of the palm to achieve the goal of reducing the delay of controlling the manipulator. In addition, combined with the motion data of each part of the upper limb, the LSTM network is used to identify the motion state of the upper limb. Finally, based on the prediction results of the LSTM, the assistive prosthetic hand is controlled to reproduce the foot actions.

Section 2 introduces the process of data acquisition and the correlation analysis of the arm and the hand in detail. Section 3 describes the extraction method of the feature parameters and the key methods in the long short-term memory neural network model. Section 4 presents the experimental results in detail and discusses the findings. Finally, conclusions are given in Section 5.

2. An Overview

Three sets of the IMU were used to acquire the angular velocity of the upper limb during movement. The IMU was installed on the shoulder, forearm, and palm. Figure 1 shows the installation location of the sensors. The IMU collects data every 20 milliseconds, and the frequency is 50 Hz. The accuracy of the sensor is 0.2 degrees. The intact upper extremity of a healthy person usually includes the shoulder, forearm, and palm. The palm can be seen as an extension of the upper arm, and the movements of the fingers play an extremely important role in grasping objects.

The correlation between the movement of the arm and the palm of normal people was studied while they completed the foot action. The angular velocity data of the three parts of the upper limb was taken when performing the actions of putting on socks, putting on shoes, and tying shoe laces. The Pearson correlation coefficient was calculated between the arm data and the palm data, so as to measure the correlation between the arm and the palm.

Table 1. Pearson correlation coefficient for arm and palm.

	Pearson Correlation Coefficient
	Shoulder and Palm	Forearm and Palm
Putting on socks	0.7455	0.6761
Putting on shoes	0.8319	0.7053
Tying shoe laces	0.8079	0.8669

shows the Pearson correlation coefficient for the arm and the palm. It can be seen from the table that for different action types, the correlation coefficients between the arm and the palm are all greater than 0.6, indicating that the degree of correlation between the movement of the arm and the movement of the palm is strongly correlated. Therefore, this study uses a machine learning method to infer hand movements based on the movements of the arms.

In order to grasp objects steadily, a normal person will open or close the fingers when the upper limb is at rest. It can be seen that finding the time when the upper limb is at rest is the key to understanding the grasping intention of the upper limb and controlling the assistive prosthetic hand. When the user has the intention of grasping, the arm will move towards the target position under the driving of the limb. At this time, the switching of upper limb of motion state is a direct manifestation of grasping intention. During the movement of the upper limb towards the target position, the angular velocity of the upper limb will become larger and larger. Before approaching the target position, the value of the angular velocity will become smaller and smaller until it is close to 0. Through the kinematic analysis of the upper limb, the motion states of the upper limb can be divided into rest, acceleration, and deceleration. In the initial state, the motion state of the upper limb is at rest by default. Figure 2 shows the switching flow of the upper limb state. The acceleration state and the deceleration state can be switched with each other, but the rest state and the acceleration state, or the rest state and the deceleration state, cannot be switched with each other.

The moment when the upper limb is at rest is used as a sign to control the assistive prosthetic hand. In previous research, the motion state of the upper limb according to the collected observation data was judged. When the motion state of the upper limb was at rest, the assistive prosthetic hand was controlled. Since it is controlled after the motion state of the upper limb is at rest, this will increase the delay of controlling the assistive prosthetic hand. Therefore, according to the current observation data, the user’s motion state is first identified, and then the subsequent motion state of the upper limb is predicted. When the predicted result is at rest, the assistive prosthetic hand is controlled.

3. Implementation of Key Model and Methods

Based on the above correlation analysis, this paper uses a long short-term memory neural network to predict the motion data of the palm. In this section, the extracted time domain features are introduced in detail, as well as the design process of parameters when using LMTN.

3.1. Feature Extraction

Due to bias drift, geomagnetic interference, and other causes, the original data collected by the inertial sensor is mixed with noise; as a result, this paper uses the moving average filtering method to filter the original data. Extracting feature parameters is one of the important ways to characterize sequence data. When analyzing data collected by inertial sensors, the features that are often extracted are divided into three categories: time-domain features, frequency-domain features, and time-frequency features [21]. Considering that the sensor output is a set of time series data, this paper directly uses time domain features to analyze the angular velocity data of the upper limb. The characteristic parameters include the variance, difference, and maximum and minimum value of the angular velocity of each part of the upper limb.

Variance can be used to measure the dispersion of a set of data. When analyzing the angular velocity of the upper limb, the magnitude of the variance can represent the degree of fluctuation of a set of data, thus representing the range of movement of the upper limb. The equation for variance is shown in Equation (1).

$$VAR=\frac{{\displaystyle \sum _{i=1}^N{\left({\omega }_i-\overline{\omega }\right)}^2}}{N-1}.$$

(1)

Among them, ${\omega }_i$ represents the angular velocity of the i-th sampling point of a certain part of the upper limb, N represents the length of the signal window, and $\overline{\omega }$ represents the average value of this set of data.

The magnitude of angular acceleration can characterize the direction of motion. Therefore, in a set of data, the difference between two adjacent angular velocity data is recorded as $S{K}_i$. The magnitude of several differences can characterize the motion state of the upper limb. The equation for the difference is shown in Equation (2).

$$S{K}_i={\omega }_{i+1}-{\omega }_i,\hspace{1em}i=1,2,3...N-1.$$

(2)

In a set of data, the magnitude of the difference between the maximum value and the minimum value can represent the magnitude of the fluctuation of a set of data. The larger the difference, the larger the fluctuation. Therefore, the maximum and minimum values of a set of data are obtained, respectively, and the difference is then recorded as MN. The equation for MN is shown in Equation (3).

$$MN=\max ({\omega }_i)-\min ({\omega }_i).$$

(3)

3.2. Neural Network

The neural network model is an important component of machine learning. Human activity data are essentially a set of time series data. Recognizing human actions is actually classifying serialized data. In order to obtain better results of action recognition, it is necessary to analyze and judge the entire time series. The current action not only depends on the current data, but also has a relationship to the previous data. RNN can solve the problem of the traditional neural network model, which does not have the function of memorizing historical information. In RNN, the output result depends not only on the current input data, but also on the previous output, so the previous historical information can be fully utilized. However, some studies have shown that although RNN can handle the dependence of time series data, it is difficult to learn and preserve long-term historical information. The effect on processing long series data is not good [22]. The LSTM neural network, as a special recurrent neural network, effectively solves this problem through a unique gate structure.

The LSTM neural network consists of an input layer, hidden layer, and output layer. Among them, the hidden layer with memory function is the core of the LSTM neural network. Figure 3 shows the structure diagram of the hidden layer unit. The hidden layer unit includes an input gate, an output gate, and a forgetting gate, which are represented by $i_t$, $o_t$, and $f_t$:

$$i_t=\sigma (W_{xi}{x}_t+W_{hi}{h}_{t-1}+b_i).$$

(4)

$$o_t=\sigma (W_{xo}{x}_t+W_{ho}{h}_{t-1}+b_o).$$

(5)

$$f_t=\sigma (W_{xf}{x}_t+W_{hf}{h}_{t-1}+b_f).$$

(6)

Among them, $x_t$ is the input at the current moment, $h_{t-1}$ is the output of the previous unit in the hidden layer at the previous moment, $W_x$ is the input weight matrix, $W_h$ is the weight matrix between neurons in the hidden layer, and b is the bias term.

The steps in which the neural network processes the data are as follows:

(1)

According to the output $h_{t-1}$ at the previous moment and the current input $x_t$, the sigmoid function is used to calculate which information can pass through $c_t$.

(2)

Control the input of saving the previous information and the current information, and then calculate the ratio of the historical information and the current input through the tanh function calculation. The updated current unit state can be expressed as:

$$c_t=f_t{c}_{t-1}+i_t(tanh(W_{xc}{x}_t+W_{hc}{h}_{i-1}+b_c)).$$

(7)

(3)

The output of the sigmoid function does not consider the information $o_t$ learned at the previous moment, and then the new unit state information is filtered and compressed through the tanh function to make the information more stable. Finally, take the inner product of $o_t$ of the output gate and the new $c_t$ to obtain the hidden layer state $h_t$ at the current moment:

$$h_t=o_{t}tanh(c_t).$$

(8)

(4)

When the input of the time series data is completed at the last moment, the hidden layer state $h_t$ of the long short-term memory neural network model is used as the input to the output layer of the network model. Then use the softmax function to calculate the final predicted action probability y:

$$y=soft\max (W{h}_t+b).$$

(9)

3.3. Model Implementation Details

The action intention recognition of the upper limb includes two contents: (a) Identify the grasping intention of the upper limb, mainly to identify the motion state of the upper limb, and find the moment when the upper limb is stationary; (b) Predict the movement of the upper limb. It mainly predicts the angular velocity data of the subsequent time points according to the angular velocity of the upper limb. In the process of recognizing and predicting the movements of the upper limb, the LSTM neural network is used to analyze the angular velocity data of the upper limb. The flowchart of action intent recognition is shown in Figure 4.

For identifying the grasping intention of the upper limb, based on the training set data of the upper limb movement, a feature extraction is performed first, which contains four sets of feature vectors. Therefore, a feature matrix with N rows and four columns can be formed. The motion states of the upper limb are divided into rest, acceleration, and deceleration. The LSTM is used to learn the sample data, and the parameters of the neural network are set as shown in

Table 2. Parameter settings for network identification.

Parameter Name	Parameter Value
Number of hidden units	100
Number of classifications	3
Maximum number of training cycles	100

. Then, the trained network model is used to identify the observation data to judge the motion state of the upper limb.

For predicting the movements of the upper limb, the LSTM neural network is also used to train the preprocessed sample data, and the weight matrix W and the bias term b are obtained through continuous iterative learning.

Table 3. Parameter settings for network prediction.

Parameter Name	Parameter Value
The size of the training set	16,875
The dimension of the input data	1
Number of hidden layers	1
Number of hidden units	200
Number of training	250
The number of iterations	15
The way of gradient descent	SGDM
Learning rate	0.005
Loss function	Cross entropy
Weight initialization method	Gaussian distribution

lists the LSTM network parameter names and corresponding parameter values. The specific steps are as follows:

(1)

Initialization parameters. In the neural network model, the weight matrix and bias term need to be initialized first. The Gaussian distribution is used for the initialization of the weight matrix, where the mean of the Gaussian distribution is $\mu =0.6$ and the standard deviation $\sigma =0$, which is in line with the general weight distribution, and the initial bias is set to 0.

(2)

Calculate the error between the actual value and the predicted value. The LSTM neural network is used to predict the observed value, and the predicted value y of the output layer of the network is obtained through a series of formula calculations. Calculate the cross-entropy of the predicted value y and the true value $\widehat{y}$ as the error. The loss is as follows:

$$loss=-\frac{1}{n}{\displaystyle \sum _{i=1}^n[y^{(i)}\ln ({\widehat{y}}^{(i)})+(1-y^{(i)})\ln (1-{\widehat{y}}^{(i)})]}.$$

(10)

(3)

Determine the weight matrix and bias term. Calculate the gradient of the loss function loss to the weight matrix W and the bias term b, and backpropagate the obtained gradient to the front end of the network to adjust the parameters of each part of the network. Iteratively train the reduced loss function through momentum stochastic gradient descent until convergence is reached.

The paper uses MATLAB software to compile the code program of the LSTM neural network. MATLAB software has a neural network toolbox. The TrainNetwork function is used to train the parameters of the LSTM. The LSTM network model is implemented using the MATLAB toolbox. Based on the visual studio 2019 platform, the host computer software for data acquisition and analysis was developed. Training and testing are run on a PC with Intel Core i7-6500U CPU, 12 Gb DDR-III 2400 MHz RAM, and NVIDIA GeForce 940MX.

4. Experiment and Result Analysis

4.1. Experimental Equipment

The acquisition of upper limb movement data is completed by the data glove, and the experimental equipment is shown in Figure 5. The data glove can be worn on the shoulder, forearm, and palm of the upper limb, and each part integrates the MPU6050 inertial sensor, which can directly output angular velocity data with a sampling frequency of 50 Hz. Another piece of experimental equipment is the assistive prosthetic hand, which is connected to the single-chip microcomputer. When the single-chip computer receives the control command from the PC, it will immediately output high or low levels to control the fingers of the assistive prosthetic hand to open or close.

4.2. Activity Classification Results

First, the effect of our experiments on identifying the motion states of the upper limb is discussed. The rest state of the upper limb is recorded as “0”, the acceleration state is recorded as “1”, and the deceleration state is recorded as “2”. The experimental data of a volunteer putting on socks were randomly selected, with a total of 791 sampling points. The sample data are identified by using the LSTM. The recognition results are shown in Figure 6.

In order to compare the feasibility of the methods more effectively, the recognition results of the LSTM with those of SVM, KNN, and LDA are compared. For the same dataset, the experimental data are first preprocessed. Then, feature extraction is performed on the processed data to construct feature vectors and state vectors. Then, the feature vector and state vector are divided into ten parts, of which seven are used as the training set and three are used as the test set. The training set was used to train the model, and the test set was used to test the classification effect of the model. The recognition accuracy of each classifier is shown in Figure 7.

By analyzing Figure 7 and

Table 4. The recognition results of each classifier.

		LSTM	SVM	KNN	LDA
Putting on socks	Static	99.11%	99.55%	99.55%	99.11%
	Acceleration	98.73%	99.15%	99.57%	98.73%
	Deceleration	99.15%	98.73%	99.15%	98.31%
	Total	99.02%	99.23%	99.45%	98.80%
Putting on shoes	Static	99.65%	98.60%	99.65%	99.30%
	Acceleration	99.45%	98.35%	99.45%	99.45%
	Deceleration	99.36%	98.73%	99.36%	98.73%
	Total	99.52%	98.56%	99.52%	99.20%
Tying shoe laces	Static	99.73%	99.60%	99.07%	99.60%
	Acceleration	99.57%	99.15%	98.94%	98.94%
	Deceleration	99.15%	99.36%	98.30%	98.73%
	Total	99.53%	99.23%	98.82%	99.12%

, it can be found that:

(1)

For the same type of action, the four classifiers all obtained high accuracies. This is due to the fact that the feature extraction on the sample data is performed, instead of directly using the sequence data. Of the four classifiers, it was unable to determine which one performed the best, because for the same classifier, its recognition effect was also different. For example, in the LSTM, in the action of tying shoelaces, the classifier has a good recognition result, but in the action of putting on socks, the recognition result of the classifier is not as expected. However, in general, the LSTM neural network model can maintain a high accuracy rate.

(2)

The recognition result of the classifier has nothing to do with the specific action type. For the same classifier, in different types of actions, the classification effect of the classifier cannot be consistent. For example, in the action of putting on shoes, the recognition accuracy of SVM is lower; in the actions of putting on socks and tying shoelaces, the recognition accuracy of SVM is the same. However, for the LSTM, the recognition accuracy of the LSTM is low in the action of putting on socks. From this, it can be shown that the action type does not affect the recognition result of the classifier.

(3)

For the three different motion states, no matter which classifier is used, the recognition result of the rest state is higher. This is in line with our expectations, because, compared to the acceleration state and the deceleration state, the main effort was put into the moment of the rest state. The control of the assistive prosthetic hand occurs at the moment when the upper limb is at rest. The recognition of the acceleration state and the deceleration state helps to eliminate the influence of bad factors, thereby improving the recognition rate of the rest state.

4.3. Action Prediction Results

Figure 8 shows the predicted results of leg movements. Among them, the red curve is the predicted value of the palm angular velocity, and the blue curve is the actual value of the palm angular velocity. In the second section, the paper discusses the strong correlation between arm movement and palm movement. Therefore, based on the motion data of the arm, the LSTM neural network is used to predict the motion data of the palm. And the palm angular velocity collected by the inertial sensor is saved in real time. The root mean square error is used to represent the error between the true value and the predicted value. The root mean square errors of the three actions are shown in

Table 5. Root mean square error of three actions.

	Putting on Socks	Putting on Shoes	Tying Shoe Laces
RMSE	1.2347	1.4424	3.6361

.

It can be seen from the error curve graph in Figure 8 and Table 5 that:

(1)

The predicted value of the LTSM network and the actual value collected by the sensor almost coincide, indicating that, based on the motion data of the arm, the LSTM network can better predict the motion data of the palm.

(2)

In the actions of putting on socks and putting on shoes, the root mean square error is less than 1.5 rad/s. Due to the increase in the amount of data, the amount of data of tying shoe laces is twice that of the former, and the root mean square error is less than 4 rad/s. It can be seen that with the increase in motion data, the prediction error of the LSTM also increases, which is inevitable. Due to the process of prediction, the algorithm always uses the previous prediction output to predict the subsequent data. If there is a certain error in the previous prediction output, the subsequent prediction output will continuously enlarge the error. That is to say, without correcting the previous prediction data, the root mean square error of the LSTM neural network will increase with the increase in motion data.

(3)

It can be found in the error diagram that most of the errors occur in the peak and trough sections, which are the switching points between the acceleration state and the deceleration state. This also explains the low accuracy of identifying the acceleration state and the deceleration state. However, these two errors will not affect the control of the manipulator.

4.4. Experimental Verification

This experiment verifies the feasibility of the method by reproducing the hand movements of the assistive prosthetic hand. Using the LSTM to predict the angular velocity data of the hand, the delay in controlling the assistive prosthetic hand was reduced. This paper concerns foot actions in daily life, such as putting on socks, putting on shoes, and tying shoelaces. It reproduces these selected actions in the way that a normal person holds an assistive prosthetic hand, which is used as a substitute for a human hand. For the three different actions, this paper selects a frequently used step accordingly, as shown in Figure 9, where the first row is the flow chart of putting on socks, the second row is the flow chart of putting on shoes, and the third and fourth rows are the flowchart for tying shoelaces.

60 volunteers were recruited to complete the experiments. Of the volunteers, there were 30 males and 30 females, ranging in age from 20 to 50. The motion experiments of putting on socks, putting on shoes, and tying shoelaces were all performed by 20 volunteers, including 10 men and 10 women. Each volunteer wore data gloves as required and repeated the experimental action 50 times. 1000 datasets were collected for each action type, and a total of 3000 datasets were obtained. All volunteers underwent proficiency training before the experiment.

In the experiment, the moments when the upper limb was stationary and when the assistive prosthetic hand was opened or closed were recorded, in order to calculate the delay of controlling the assistive prosthetic hand. This was then compared with the delay time generated by our previously designed control method. The latency data for the two methods are shown in

Table 6. Delay time generated by two control methods.

Participant ID		1	2	3	4	5	6	7	8	9	10	Mean
Method 1	Putting on socks	0.66 s	0.57 s	0.55 s	0.61 s	0.72 s	0.68 s	0.59 s	0.58 s	0.64 s	0.72 s	0.632 s
	Putting on shoes	0.72 s	0.64 s	0.65 s	0.71 s	0.75 s	0.68 s	0.61 s	0.67 s	0.55 s	0.59 s	0.657 s
	Tying shoe laces	0.64 s	0.66 s	0.58 s	0.68 s	0.64 s	0.72 s	0.64 s	0.59 s	0.63 s	0.66 s	0.644 s
Method 2	Putting on socks	1.31 s	1.32 s	1.27 s	1.33 s	1.35 s	1.38 s	1.45 s	1.29 s	1.22 s	1.34 s	1.326 s
	Putting on shoes	1.25 s	1.22 s	1.33 s	1.28 s	1.52 s	1.38 s	1.33 s	1.28 s	1.32 s	1.44 s	1.335 s
	Tying shoe laces	1.18 s	1.18 s	1.22 s	1.35 s	1.41 s	1.47 s	1.36 s	1.33 s	1.25 s	1.27 s	1.302 s

. Compared with the previous control methods, the control method based on the LSTM neural network has greatly reduced the delay. The average delay time is around 0.65 s.

5. Conclusions

In order to reduce the delay in controlling the assistive prosthetic hand, this paper proposes a new method for predicting the action intention of the upper limb. Based on the correlation between the angular velocity of the arm and the angular velocity of the palm when a normal person completes a foot action, the LSTM is used to predict the angular velocity of the hand. The motion information of the upper limb is collected by the IMU, including the angular velocity of the shoulder, forearm, and palm. Whether the upper limb is still or not is used as a sign to control the assistive prosthetic hand. The motion states of the upper limbs are divided into acceleration, deceleration, and rest. The LSTM neural network is used to identify the motion state of the upper limb, by up to 99%. In the action prediction of the upper limb, the LSTM is used to predict the angular velocity data of the palm, and the error between the actual value and the predicted value is calculated by the root mean square error. For the actions of putting on shoes and putting on socks, the root mean square error is less than 1.5 rad/s; for the action of tying shoelaces, the root mean square error is less than 4 rad/s. Finally, the neural network model is applied to the experiment of controlling the assistive prosthetic hand. The delay in controlling the assistive prosthetic hand is recorded and compared with the delay produced by our previous control method, with an average delay time of 0.65 s. Based on the analysis of the experimental data, it can be concluded that the LSTM neural network can achieve low prediction error.

Taking the motion information of the upper limb of the human body as the information source for judging the intention of the upper limb can avoid the influence of factors, such as age, gender, and degree of amputation on the control of the prosthesis. The experimental results of this paper are of great help to research on the control method of assistive prosthetic hands. The research in this paper is limited to the design of the method, and there is still a long way to go for generalization to the applications needed by people with disabilities. In the future research, it is necessary to focus on the design of the upper limb prosthetic system, combine the control method and hardware device more effectively, and apply results to services for the disabled.