LSTM Recurrent Neural Network for Hand Gesture Recognition Using EMG Signals

Abstract

Currently, research on gesture recognition systems has been on the rise due to the capabilities these systems provide to the field of human–machine interaction, however, gesture recognition in prosthesis and orthesis has been carried out through the use of an extensive amount of channels and electrodes to acquire the EMG (Electromyography) signals, increasing the cost and complexity of these systems. The scientific literature shows different approaches related to gesture recognition based on the analysis of EMG signals using deep learning models, highlighting the recurrent neural networks with deep learning structures. This paper presents the implementation of a Recurrent Neural Network (RNN) model using Long-short Term Memory (LSTM) units and dense layers to develop a gesture classifier for hand prosthesis control, aiming to decrease the number of EMG channels and the overall model complexity, in order to increase its scalability for embedded systems. The proposed model requires the use of only four EMG channels to recognize five hand gestures, greatly reducing the number of electrodes compared to other approaches found in the literature. The proposed model was trained using a dataset for each gesture EMG signals, which were recorded for 20 s using a custom EMG armband. The model reached an accuracy of to 99% for the training and validation stages, and an accuracy of 87 ± 7% during real-time testing. The results obtained by the proposed model establish a general methodology for the reduction of complexity in the recognition of gestures intended for human.machine interaction for different computational devices.

1. Introduction

Hand gesture recognition systems have been improving greatly within the last two decades, mainly because of the need for more natural and accurate human–computer interaction systems [1]. Furthermore, Human Computer Interfaces (HCI) are becoming very important in one field in particular: upper limb prosthesis [2,3]; hands are known for being one of the most important and functional parts of our body, hence loss of them can result in a subject notable deterioration of life quality. This is mainly why the literature is rich in studies aiming for optimum ways to control upper limb active prosthesis [4]. Even with this amount of literature there are some major challenges regarding actual HCI techniques for upper limb prosthesis control [5], coming first, three of these main challenges are listed and explained:

• The first challenge involves hand gesture recognition in real-time [6]. Most studies regarding this topic focus primarily in finding novel algorithm-based methods in order to achieve high performance rates in gesture recognition. Nevertheless these scores are obtained in non-real-time tests where factors such as computational efficiency, window length and system overall performance are not taken in to account [7,8]. Furthermore, this kind of hand gesture recognition methods may be suitable for non-mobile applications such as sign language recognition [9], but not for active prosthesis which are mobile and independent systems.
• The second challenge is to find such a good performance classifier that it can be used in different types of amputees and still has a good accuracy rate in hand gesture recognition [10], moreover, the challenge is to find an appropriate classifier that can deliver good recognition rates in almost every amputee morphology, while keeping the prediction delay low and the overall system computational needs moderate.
• The third challenge is to narrow the gap between academic and industrial achievements. Both of them, at this point, are heading in totally different ways. While an academic display can be set in a known environment, the industrial one needs to be as robust as possible in order to increase the overall performance of the system when is used by a real world user [11].

Taking these issues, in the scientific literature three well-defined development trends can be seen. The first development trend focuses on the use of images to cluster hand gestures, highlighting the cameras as a common sensor that can be found in a great number of electronic devices nowadays [12], allowing to identify hand gestures both in static and dynamic environments, as well as in real time [13]. The main source of information in this kind of classification is images, which is why large data sets of pictures are taken, most of them containing multiple hands from different subjects, waving different signs and gestures, regardless of whether the model sorts static or dynamic movements [14]. The main data that it needs are images, either if it is a sequence of them (Dynamic movement classification) or just one of them (Static movement classification) [15], but there is an even lower level approach to this particular data set: Images are large matrices composed of RGB (Red, Green, Blue) pixel data. Then these pixels are the basic structural and functional units of an image. That is why some studies dig even deeper in to this kind of data and add depth to the image composition. Such grids are formed with RGB-D pixels [16,17]. This can be accomplished by using commercially available cameras such as the ones that can be found in a Kinect [18] or in the commercially available leap motion controller [19]. The next step in this kind of classification is to process those large grids of pixels and to obtain a good fitting model, capable of sorting different types of gestures. Some of those are based on Convolutional Neural Networks (CNN) [16,20], K-Nearest Neighbor (KNN) and Bayesian Neural Network [21]. In certain cases the model descriptors are changed in order to lower the computational demands [22,23] for real-time image processing.

The second development focus is the use of Electromyography (EMG) for hand gesture recognition. It is important to mention that the EMG shows the electrical muscular activity recorded by positioning electrodes on top of the muscular group of interest. In order to obtain valuable information for the characterization of the hand gesture, filtering, processing and clustering methods of EMG signal have emerged as highly relevant techniques [24]. Another batch of papers shows a lot of methods and variants, used in the literature, to increase the rate of accuracy of the process mentioned above, starting by using a large amount of electrodes, followed by the use of huge time resolution ADCs (Analog to digital converters), however the real variety in literature comes when it is time to choose the descriptors [24,25,26] and what algorithm to use for sorting gestures, from recurrent neural networks (RNN) [27,28], to Hidden Markov Models [29], all the way to gesture recognition based on Motor Unit Spike Trains (MUST) [30], nearly every algorithm has been used. Nevertheless deep learning models have been the most accurate and reliable, RNN have shown good results along with Long-short Term Memory (LSTM) layer in them [17,27,28].

The third development trend, which is starting to take off, is force myography (FMG). It is basically an alternative for EMG sensing, and it is based on the volumetric changes of muscle fibers while they contract or distract [31]. Some researchers claim that it is even better than EMG because it only needs force or pressure sensors attached to an armband, and the requirements for such signals processing do not demand as many filters or amplifiers as EMG does [29]. Here, the simple nature of both signals is different and that is why the approaches are distinct. Whilst the EMG is an electrical signal caused by small impulses created in the motor neurons when contracting the muscles [24], FMG is a volume change that can be sensed by using special transducers in order to know its magnitude. If a big picture of the process of this kind of hand gesture recognition is taken, much would be similar to the EMG signal acquiring and clustering process. Initially the signal is acquired by using sensors suitable for this approach. It is important to take into consideration the force or pressure range that the transducer is able to sense [32]; then these data are processed and lastly the gestures are classified using a various palette of algorithms such as Linear Discriminant Analysis (LDA) [5,27] and Recurrent Neural Networks[17].

Following the three challenges and the second trend mentioned before, Recurrent Neural Networks (RNN) emerge as promising models for gesture recognition classifiers based on EMG signals due to their ability to identify patterns in dynamic time series; RNNs have been widely used in different fields to forecast time series data and model energy system behavior [30,33,34]. In the field of Human–Machine Interfaces and electromyography, multiple studies have explored the application of RNNs in gesture recognition systems in order to determine whether is or not suitable for gesture prediction and prosthesis control. For example, Jabbari et al. has explored the use of stacked LSTM Recurrent Neural Networks for gesture recognition on amputees [35]; Samadani performed a comparative study evaluating unidirectional and bidirectional RNNs, using both Long-short Term Memory (LSTM) units and Gated Recurrent Units (GRU), also analyzing the effects of the attention mechanism, Samadani found that bidirectional LSTM units with attention yielded the best results [36]; Hu et al. proposed an attention-based hybrid CNN-RNN architecture for gesture recognition, this research compared the models accuracy on different popular gesture recognition datasets [37]; other studies like the ones presented by Zhang et al, Jiang et al. and Simão, proposed different RNNs architectures composed of multiple stacked recurrent layers and their comparison with other architectures such as CNN-RNN models [13,38,39].

As can be found in the literature, most of the studies use static data and online datasets in which the data was obtained in controlled environments, however, scientific literature has shown that models that often perform well on static data do not perform well in systems operating in real-time [40], making the results obtained only in static data misleading. Moreover, most of the research found uses a high number of EMG channels (eight or more) [32,35,36,37,38,41,42], which increases the computational needs and the prediction delay, making it hard to implement those systems in embedded devices and commercial applications [5]. This research aims to propose a full hand gesture recognition system capable of predicting different gestures in real-time operation using only four EMG channels. Concepts and design of the acquisition system and the time domain optimum descriptor will be discussed during the overall development of this paper, to highlight their importance in the development of the classifier.

This paper presents a novel hybrid RNN model composed of one LSTM (Long Short-Term Memory) layer and dense layers (fully connected layers) to classify in real-time different hand gestures using only four EMG channels. The model achieved an accuracy $99\%$ in the training and validation stages in the recognition of five hand gestures performed with static data. On the other hand, the proposed model was implemented on a personal computer that received the EMG data from a custom made EMG acquisition armband. The model achieved an accuracy of 87 ± 7% during real-time testing, proving to be suitable and scalable for real-time prosthesis control. The real-time tests revealed challenges that only occur during real-time operation, giving insights about possible solutions to this challenges to further optimize the system.

This article is structured as follows, Section 2 presents the methodology used to define the experimental set up of EMG system, the algorithm choosing and theoretic support are brought up during this section. Section 3 shows the main results obtained form the tests carried out by the system previously described; and a final section shows the conclusions (Section 4), setting ground for future work.

2. Methodology

Due to the complexity that frames the recognition of hand gesture in real time, this section presents a novel methodology and the steps followed to develop an integrated system for gesture recognition, showing key concepts that allow the implementation of a Recurrent Neural Network with LSTM and dense layers.

2.1. Experimental Set-Up and Data Acquisition

In order to set up a hand gesture classifier model, a recording of EMG signals from five different gestures was performed, which can be seen in Figure 1. These gestures were selected based on basic daily hand patterns, including two of the fundamental grasping motions, the power grip (Figure 1b) and the pinch grip (Figure 1c), which is a precision handling maneuver [43].

The EMG data was acquired using a custom four-channel EMG armband [44]. The EMG armband acquired the EMG signal in each channel with dry electrodes, then it conditioned the signal so a microcontrollers ADC could read it and finally it sended packages containing the four channels information via serial communication. The data of the four EMG channels were received by a personal computer (PC) in which the preprocessing and development of the model were performed.

Each gesture was recorded for 20 s with a sampling frequency of 1 KHz, allowing to obtain 20,000 samples per gesture. The sampling frequency was selected following the Nyquist sampling theorem, considering a maximum frequency for the EMG signal of 500 Hz [45,46]; it was also taken into consideration the computational cost of acquiring the four EMG channels and the maximum serial transmission speed of the microcontroller. An adequate selection of the sampling frequency is key to acquire all the signal components and perform a better mapping of the signal with the proposed classifier.

The acquired samples were divided into 100 windows of 200 samples; collecting a total of 500 examples to build the dataset. The 200 samples window was selected based on a literature review on hand prosthesis requirements and EMG window length selection; where a prosthesis requires an acting speed of 500 ms or less, in order to work smoothly [47,48,49,50]. Taking into consideration that the sample time is 1 ms, 200 sample windows allow an actuation speed of around 200–300 ms considering a serial communication with the computational model, which is within the required speed; moreover, researchers have found that window lengths of 200 ms or more allow to have reduced classification errors when working with EMG signals [45,51].

The dataset was randomly shuffled and then split into two parts: $80\%$ of the shuffled dataset was used for training, and the remaining $20\%$ was used as a validation set to perform hyper-parameter tuning and validation of the training.

It is important to note that the acquisition of the EMG signals was done following all the ethical guidelines set by law and the institution. The data was taken from eight test subjects with an age range between 20 and 30 years. All participants were healthy people. The experiments were approved by EIA University Ethical Committee (Code P201912-04), and also an informed consent was obtained from the participants.

2.2. Data Preprocessing

Once the EMG data were acquired, the four EMG signals were preprocessed so they could be used as inputs to the proposed model. The preprocessing is a fundamental step when working with EMG signals, since raw EMG signals are random and difficult to interpreter, the preprocessing step allows to extract valuable information to the raw EMG signal This information then can be used as input data to a classifier. For this step an EMG feature extraction process is generally conducted. According to scientific literature there are many known features of EMG signals [3]. However, research on EMG signal processing showed that there is not a single best feature extraction process or formula, and that it largely depends on the data and required characteristics of the system [52].

With this in mind, the preprocessing step conducted in this research was mainly focused on smoothing the signal and removing the unuseful noise of it, leaving the feature extraction for the proposed neural model [53,54]. The preprocessing method applied to the signals can be seen in Figure 2. In this process, initially the mean of the signal is subtracted in order to center it at zero mean; then, a rectification step is performed in which the absolute value is applied to the zero mean signal, this is a useful step during the preprocessing of the EMG signal [46]; then, a moving average filter was applied in order to reduce the noise of the signal [13] and to obtain its envelope, which is an useful characteristic when working with neural network models; finally the signal was normalized between 0 and 1 as recommended in the Deep Learning literature [55].

In general, the moving average is the most common filter in digital signal processing (DSP) due to its ease of implementation. In spite of its simplicity, the moving average filter is optimal for reducing random noise while retaining the signal, making it an outstanding filter for time domain encoded signals [56], thus allowing to obtain useful information about the EMG signal in the time domain, while keeping the computational cost low. Its formula is given by Equation (1).

$$y\left(i\right)=\frac{1}{M}\sum _{j=0}^{M-1}x(i-j)$$

(1)

For the moving average filter, a total of 20 delays for the filter period N was selected. This period allowed to smooth the signal reducing the noise, making it easier for the proposed LSTM model to identify the signal patterns that represent a hand gesture. Figure 3 shows the steps of the signal pre-processing for a window of 200 samples.

2.3. Recurrent Neural Network (RNN) Model

To perform a hand gesture classification in real-time based on the pre-processed EMG signals, a novel Recurrent Neural Network (RNN) model of LSTM type was proposed. The RNN structure with LSTM configuration has been found to be a good architecture for the identification of temporal patterns in time series [57]. In general many researchers found that the RNNs have the ability to retain information making it an excellent approach for time series applications [58]. In this context, the RNN proposed in this research has a hybrid architecture composed of dense layers wrapping a recurrent layer; it has been found that the approach where dense layers support the recurrent layer allows for an increased performance over models that only have the recurrent layer [59,60]. The overall architecture of the RNN can be seen in Figure 4. Each one of its layers will be explained in greater detail in further Section 2.3.1 and Section 2.3.2.

Figure 5 shows the basic structure of the recurrent layer for the proposed model. Here, each input X is composed of one sample of each EMG channel in an specific time t; where $T_x$ is the length of the window that represent a hand gesture to classify ($T_x=200$); and a represents the activation values passed in each time instant from one recurrent unit to the next one. These activation values a are propagated thru the network capturing dependencies between the different time steps and then the last activation $a^{\langle T_x\rangle }$ is used to calculate the output of the network.

The implementation of the proposed model and its training was done in Python using the Tensorflow library due to the fast and efficient way that this library gives in the implementation of deep learning models. However, in order to have a clear understanding of the models architecture, the next section presents the mathematical definitions for each component of the network.

2.3.1. Dense Layers

The dense layer represents the processing units that integrates a classic deep learning model. In the scientific literature it is common to find models that use linear functions to map an input to an specific output, however, in the case of signal processing where a specialized treatment of noise must be performed, such as when working with EMG signals as is done in this research, the literature recommends the use of activation functions of the hyperbolic tangent type. The vectorized implementation of the dense layer can be seen in Equation (2).

$$a^{\langle l\rangle }=tanh(W^{\langle l\rangle }{a}^{\langle l-1\rangle }+b^{\langle l\rangle });a^{\langle 0\rangle }=X$$

(2)

where:

$a^{\langle t-1\rangle }$ represents the previous layer activations. In the case of the first layer, they are equal to the input X which contains the value of each EMG channel.

$W^{\langle l\rangle }$ represents the current layer weights.

$b^{\langle l\rangle }$ represents the current layer bias.

2.3.2. Long Short-Term Memory (LSTM) Layer

The recurrent layer is made of Long Short-term Memory (LSTM) units, which is one of the state of the art units used in RNNs due to its capacity to memorize long term dependencies. Studies have shown that LSTM units outperform other state of the art recurrent units in complex dataset [61,62,63,64], making them a reliable choice for the recurrent layer architecture. The structure of a single LSTM unit can be seen in Figure 6.

Equations (3)–(8) show the vectorized implementation of each one of the items that make up the LSTM structure.

$${\tilde{c}}^{\langle t\rangle }=tanh(W_c[a^{\langle t-1\rangle },X^{\langle t\rangle }]+b_c)$$

(3)

$${\Gamma }_i^{\langle t\rangle }=tanh(W_i[a^{\langle t-1\rangle },X^{\langle t\rangle }]+b_i)$$

(4)

$${\Gamma }_f^{\langle t\rangle }=tanh(W_f[a^{\langle t-1\rangle },X^{\langle t\rangle }]+b_f)$$

(5)

$${\Gamma }_o^{\langle t\rangle }=tanh(W_o[a^{\langle t-1\rangle },X^{\langle t\rangle }]+b_o)$$

(6)

$$c^{\langle t\rangle }={\Gamma }_i^{\langle t\rangle }\ast {\tilde{c}}^{\left(t\right)}+{\Gamma }_f^{\langle t\rangle }\ast c^{\langle t-1\rangle }$$

(7)

$$a^{\langle t\rangle }={\Gamma }_o^{\langle t\rangle }\ast tanh\left(c^{\langle t\rangle }\right)$$

(8)

where:

$X^{\langle t\rangle }$ is an input vector containing the value of each EMG channel or the activation values of the previous dense layer in time t.

$a^{\langle t-1\rangle }$ represents the activations of the LSTM units in the previous time $t-1$.

$c^{\langle t-1\rangle }$ represents the memory values in the previous time $t-1$; $a^{\langle t\rangle }$ are the activations in the current time t.

$c^{\langle t\rangle }$ are the new memory values for time t.

$W_x$ represents the LSTM unit weights for each gate and $b_x$ represents the LSTM unit bias for each gate.

It is important to note that $[a^{\langle t-1\rangle },X^{\langle t\rangle }]$ is a matrix form by the concatenation of the previous time activations $a^{\langle t-1\rangle }$ and the current inputs $X^{\langle t\rangle }$.

The output layer integrates a SOFTMAX activation function to classify which gesture is being performed. The SOFTMAX activation function is one of the most widely used functions for multi-class classification problems because of its performance in predicting the probability of each class [55,65,66]. The SOFTMAX function predicts the probability of a class $C_j$ given an input X as shown in Equation (9). Its mathematical expression can be seen in Equation (10).

$${\widehat{y}}_j=P\left(C_j\right|X)$$

(9)

$${\widehat{y}}_j=\frac{t_j}{{\sum }_{j=1}^C{t}_j};t_j=e^{a_j}$$

(10)

where:

${\widehat{y}}_j$ is the output probability for the class j.

C is the total number of classes that are being predicted.

$a_j$ represents the linear combination of the weights and the previous layer activations, and can be defined as $a_j=W{a}^{\langle l-1\rangle }$.

Its important to note that the SOFTMAX output layer has the the same units or nodes as the number of classes that are being predicted.

2.4. Model Optimization

Once the model is defined, the next step was to select a cost function in order to optimize it through training and obtain the desired results in the classifier. In general, when using an output layer with a SOFTMAX activation function, a categorical cross entropy loss function $\mathcal{L}(\widehat{y},y)$ is selected, which allows optimizing the cost function $\mathcal{J}(W,b)$ that sums the loss over the training examples. The categorical cross entropy loss allows a correct penalization of incorrect predictions and improve the performance in multi-class classification problems compared with other loss functions like the mean squared error ($MSE$). The categorical cross entropy loss and the cost function are shown in Equations (11) and (12).

$$\mathcal{L}(\widehat{y},y)=-\sum _{j=1}^C{y}_{j}log\left({\widehat{y}}_j\right)$$

(11)

$$\mathcal{J}(W,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}(\widehat{y},y)$$

(12)

where:

${\widehat{y}}_j$ correspond to the models predicted probability for class j.

$y_j$ represents the correct class (label) of the training example.

C is the total number of classes and m is the number of training examples in the mini batch.

It is important to note that the model was optimized with a mini batch gradient descent strategy. The batch size was set during the hyper-parameter tuning process along with the number of training epochs (Training iterations). The optimization algorithm will be explained in greater detail in the next paragraph.

In order to minimize the cost function $\mathcal{J}(W,b)$, it is necessary to update the weights W and biases b of each layer. This must be done by an optimization algorithm such as gradient descent. In this research, the Adaptive Moment Estimation ADAM [67] algorithm was selected; that is, a combination of Gradient Descent With Momentum and Root Mean Square Propagation (RMSProp). The ADAM optimization algorithm is considered to be one of the best optimization algorithms to train neural networks [68,69,70].

2.4.1. Hyperparameter Tuning

One of the key steps during the model training or model optimization, is the process known as hyper-parameter tuning. The hyper-parameter are values used to control the learning of the model, for example the model architecture, the number of training epochs (Training iterations), the learning rate $\alpha$, among others. This process is performed with the objective of obtaining an optimal performance of the model [71,72,73], and correct issues like high bias and high variance.

The fundamental problem of hyper-parameter tuning consists in finding the sets of hyper-parameter $\lambda$ that minimize the generalization error ${\mathbb{E}}_{(D_{train},D_{val})\sim D}V(\mathcal{L},{\mathcal{A}}_{\lambda },D_{train},D_{val})$ of a machine learning algorithm $\mathcal{A}$ given a data set D (Divided in training $D_{train}$ and validation $D_{val}$). Where the hyper-parameters $\lambda$ belong to the configuration space ∧ and V is the loss function of the algorithm $\mathcal{A}$ parametrized by $\lambda$ (${\mathcal{A}}_{\lambda }$). Equation (13) shows the mathematical expression of this optimization problem [71].

$${\lambda }^{\ast }=\underset{\lambda \in \wedge }{argmin}{\mathbb{E}}_{(D_{train},D_{val})\sim D}V(\mathcal{L},{\mathcal{A}}_{\lambda },D_{train},D_{val})$$

(13)

The literature has proposed many techniques to address the problem of hyper-parameter tuning. The most widely used are: manual search, in which the hyper-parameters are tuned by hand with an iterative process; grid search, in which a finite set of values for the defined hyper-parameters are specified and then the combination of these values is evaluated; and random search [74], which follows the same principle of grid search but the set of values for the hyper-parameters is set randomly [71,73,74,75].

In this research, manual hyper-parameter tuning was performed due to its simplicity and low computational cost. This optimization was done by changing the following hyper-parameters: the learning rate $\alpha$, the batch size, the number of training epochs and the model architecture. With the future work in mind, in which the model might be embedded in a micro-controller to make the system portable; the architecture of the model started as simple as possible and the it was increased with more layers and hidden units in order to achieve the desired performance. The results of this process can be seen in greater detail in Section 3.1.

2.5. Metrics

Once the model is trained, it is important to define a way to determine its performance, this is when performance metrics come to play. There are many metrics to indicate the performance of a model, most of them based on the confusion matrix, which contains the values of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN), values that come in handy to calculate these metrics.

This research evaluated the performance of the gesture classifier using two metrics commonly used in the literature [76]. The first one was the accuracy, which can be mathematically defined as shown in Equation (14).

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(14)

As Equation (14) shows, the accuracy measures how much the model is correctly predicting on the entire data-set and represents the probability that a prediction made by the model is correct. Although this is an easily interpretable measure, it comes with the issue that it does not consider the data-set class distribution and gives the same weight to each prediction, which tends to hide strong classification errors for classes with few examples, since those classes are less relevant compared to the biggest ones, thus, this metric can be easily misinterpreted and give a false sense of performance for the model. This downside in this metric, generally occurs when the model is trained or validated with imbalanced data-sets in which some classes have fewer examples than other classes.

To solve the aforementioned disadvantage in the accuracy metric, a second metric is proposed that allows to validate the performance of the model; the proposed metric is the Macro F1-Score (multi-class version of the F1-Score), which can be interpreted as a harmonic mean of the model precision and recall. It is important to note that, considering the gesture classifier has multiple classes, it is necessary to average the precision and recall across all classes in order to calculate the Macro F1-Score. Equations (15)–(17) shows the necessary calculations to find the metric.

$$\begin{array}{cc}\hfill Precisio{n}_j& \hfill =\frac{T{P}_j}{T{P}_j+F{P}_j}\hfill \\ \hfill MacroPrecision& \hfill =\frac{{\sum }_{j=1}^{C}Precisio{n}_j}{C}\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill Recal{l}_j& \hfill =\frac{T{P}_j}{T{P}_j+F{N}_j}\hfill \\ \hfill MacroRecall& \hfill =\frac{{\sum }_{j=1}^{C}Recal{l}_j}{C}\hfill \end{array}$$

(16)

$$MacroF1-Score=2\left(\frac{MacroPrecision\ast MacroRecall}{MacroPrecision+MacroRecall}\right)$$

(17)

As Equations (15)–(17) show, the precision and recall are calculated for each class and then averaged across the C classes of the classifier.

The Macro F1-Score is useful to support the accuracy metric and validate the model performance even if the data-set is imbalanced. As mentioned earlier, the F1-Score requires the model precision, which is a measure of the model’s quality. It indicates if the model returns more relevant predictions (TP) than irrelevant ones (FP); on the other hand, the F1-Score also requires the model recall, which is a measure of the model’s quantity. It indicates the capacity of the model to return most of the relevant predictions (TP) without considering if irrelevant predictions are also returned. With this in mind, the F1-Score, being the harmonic mean between the precision and recall, gives equal weight to both metrics. Thus if the F1-Score is high, both metrics will also be high, more relevant predictions and among all positive predictions most of them are relevant. Moreover, considering that the F1-Score is based on the average metrics for each class, it gives the same weight to all classes, taking them all into consideration independently if some classes have less examples than other classes, thus, giving a better estimate of the model performance.

2.6. Experimental Validation

In order to analyze and evaluate the proposed model two main phases were performed. An initial phase where the model was trained and tuned according to the concepts mentioned in Section 2.4; and a second phase where the best performing models of the first phase, were tested in real time gesture classification. Both experimental stages validated the model performance using the metrics seen in Section 2.5.

In the first phase, training and tuning, the training/validation data-set was constructed as mentioned in the experimental set-up (Section 2.1. Once the data sets were formed, the next step was to train and tune the model, for this, a hyper-parameter tuning process was performed (Section 2.4.1). In this process, a total of six different LSTM-RNN architectures were proposed following the general architecture presented in Section 2.3; in this part, the LSTM-RNN architecture was increased in size in each one of the models, the first three proposed model consisted in only one LSTM layer and the last three were hybrid models composed of Dense and LSTM layers; this was done in order to increase the size of the network gradually until an optimal performance architecture was found, aiming to find a good trade off between model size and performance; this trade off is a key part to embedding the model into a micro-controller in the future. During the hyper-parameter tuning, a manual search was used to set the learning rate $\alpha$, the batch size and the number of training epochs. In this first stage, the trained models were evaluated with the metrics proposed in Section 2.5. For a model to be considered for the second phase (real time validation), it was required that it had a training accuracy ≥98% as well as a training F1-Score ≥98%; the model must also hve a difference ≤1% between training and validation for both the accuracy and F1-Score in order for it to not suffer from high variance.

In a second phase, a real time validation was carried out. This validation is a fundamental step, because it allows to identify flaws in the gesture recognition system and measure the performance of the model dynamically. To tackle this, the candidate LSTM-RNN models selected in the previous phase were evaluated in four sessions using real time EMG acquisition using the EMG armband, in this sessions each gesture was performed for 20 s. Taking into consideration the possible reduction in accuracy due to changes in the EMG armband position, thus, changes in the data distribution, the candidate models (pre-trained) were re-trained after the initial training phase using data of another two EMG recording sessions, where the EMG armband was taken off and put back on the arm between sessions. Each session the EMG armband was put approximately in the same position; this was done in order to take into account the variations in data distribution due to the small changes in the EMG armband position.

For this last phase, the re-trained models were evaluated using the metrics proposed in Section 2.5. In this phase, one of the candidate models was selected as the final LSTM-RNN gesture classifier. The conditions for the selection of this model were that it had an accuracy and F1-Score ≥ 80%. It is important to note that the real time target metrics are lower compared to the training/validation targets, because in this phase the sensitivity of the EMG acquisition armband and its position on the arm, which varies across different real time recording sessions, are variables that have an impact on the performance of the algorithm [40,77,78,79].

3. Results and Discussion

This section presents the research results based on the methodology presented in Section 2. The results are presented in three parts; Section 3.1 presents the training and hyper-parameter tuning of the gesture classifier, the performance across testing and validation sets; Section 3.2, presents the performance of the candidate models during real-time testing, and Section 3.3 presents the selected model, its architecture and detail performance during training and testing.

3.1. Recurrent Neural Network (RNN) Model Training

As mentioned in Section 2.4.1, multiple network configurations were proposed in order to find an optimal LSTM-RNN architecture for the classifier. A total of six LSTM-RNN models were trained, Figure 7 shows the learning curves of the six LSTM-RNN models, presenting their loss during the training process. It is important to note, that during the manual search in the hyper-parameter tuning process, it was found that setting the learning rate $\alpha$ to $0.0001$, the batch size to 64 and the number of training epochs to 1000, allowed a better performance and stability during training and validation.

Analyzing Figure 7 it can be seen how models 1, 2 and 3 under-perform when compared to the rest of the LSTM-RNN models. It can also be noted how models 2 and 3 present high instability during training, thus making it difficult for them to converge to an optimal minimum; this instability can be due to their inability to generalize and learn the patterns in the data. In contrast, models 4, 5 and 6 presented stable convergence to a minimum in the loss function, especially model 6, which had the most stable training process and its loss value was the lowest of all models.

The final performance of the LSTM-RNN models across the training and validation sets can be seen in

Table 1. LSTM-RNN models performance in training and validation sets during the hyperparameter tuning.

Model	Architecture	Training		Validation
		Accuracy	F1-Score	Accuracy	F1-Score
1	LSTM (8)	0.6575	0.6173	0.6000	0.5392
2	LSTM (16)	0.9075	0.9068	0.8500	0.8360
3	LSTM (32)	0.9700	0.9700	0.9600	0.9589
4	Dense (8) LSTM (16) Dense (8)	0.9850	0.9849	0.9900	0.9894
5	Dense (16) LSTM (16) Dense (16)	0.9875	0.9874	0.9800	0.9798
6	Dense (32) LSTM (16) Dense (32)	0.9925	0.9925	0.9900	0.9896

. This table contains in its first column the LSTM-RNN model number; the second column presents the architecture of the LSTM-RNN model indicating its layers and number of units following the format <$layername\left(numberofunits\right)$>; the third and fourth column presents the final accuracy and macro F1-Score on the training set, and the fifth and sixth column presents the final accuracy and macro F1-Score on the validation set.

As Table 1 shows, models 1 and 2 under-performed (have high bias or Under-fitting) when compared to the rest of the LSTM-RNN models, they also suffered high variance (Over-fitting), thus, making them not suitable for the classifier. In the case of model 3, even though it presented an accuracy superior to $95\%$, it was not close to the accuracy presented by models 4, 5 and 6. This can also be validated looking at the macro F1-Score. Conversely, models 4, 5 and 6 presented an accuracy superior to $98\%$ across both training and validation sets. It was the same case for their macro F1-Score; making them suitable for testing the classifier in real time.

3.2. Real Time Testing

Once models 4, 5 and 6 were selected as candidates for the gesture classifier, real time testing was performed according to the proposed validation in Section 2.6.

Table 2. Accuracy of LSTM-RNN candidate models during real time testing across four different acquisition sessions.

Model	Accuracy in Session				Average	SD ($\mathit{\sigma }$)
	1	2	3	4
4	0.7500	0.8443	0.8521	0.7456	0.7980	0.0581
5	0.8724	0.7455	0.7895	0.8406	0.8120	0.0559
6	0.8640	0.9660	0.7979	0.8640	0.8729	0.0694

and

Table 3. F1-Score of LSTM-RNN candidate models during real time testing across four different acquisition sessions.

Model	F1-Score in Session				Average	SD ($\mathit{\sigma }$)
	1	2	3	4
4	0.6555	0.7548	0.8023	0.7054	0.7295	0.0632
5	0.8720	0.7465	0.7854	0.8401	0.8110	0.0559
6	0.8526	0.9660	0.7377	0.8473	0.8509	0.0932

show the accuracy and F1-Score of the selected models obtained in different real time testing sessions along with their standard deviation ($\sigma$),

Table 4. LSTM-RNN candidate models performance summary during real time classification.

Model	Accuracy	F1-Score
4	0.7980 ± 0.0581	0.7295 ± 0.0632
5	0.8120 ± 0.0559	0.8110 ± 0.0559
6	0.8729 ± 0.0694	0.8509 ± 0.0932

shows the summary of Table 2 and Table 3.

Looking at Table 4 it can be seen that LSTM-RNN model 6 performed the best among the candidates during the real time application of the classifier. It is important to note that this model ended up being extremely sensitive. Looking at Table 2 and Table 3 it can be seen that the models vary their performance across different acquisition sessions. This sensitivity was caused by the variation in the data distribution across training/validation and real time testing sessions, this change in data distribution is mainly caused by variations in the EMG armband location; this model sensitivity to position variations of the acquisition device is one of the current challenges when working with EMG signals and real time EMG classifiers [40,77,78,79]. In order to try to solve this, the three candidate models were fed with more data and trained again for 150 epochs, aiming to make the models more robust and more capable of generalizing to new data; however, this approach failed and the three models presented similar accuracy to the one shown in Table 4.

3.3. Selected Model Training/Validation and Real Time Testing Details

According to the results showed in Section 3.1 and Section 3.2, the LSTM-RNN model that achieved the best performance was model 6. This is because of all models, it presented the best performance across training, validation and real-time testing. It also proved to be more stable during training and the more capable generalizing new data. Figure 8 shows the training process of this particular LSTM-RNN model. This figure includes both the training and validation loss across the 1000 training epochs. As can be seen in the figure, both curves converge to approximately to the same value, meaning that the model does nt suffer from high variance (over-fitting) on training and validation data. It also does not suffer from high bias (Under-fiting) because of its $99\%$ accuracy; however, taking in consideration the results and conclusions found in Section 3.2, it is important to note that the model does suffer from high variance on training/validation and real time testing data. This high variance is caused by the difficulty in reproducing the same armband location when acquiring the EMG signals across different testing sessions.

The selected LSTM-RNN architecture can be seen in Figure 9. As proposed in the methodology the model is based on recurrent neural networks and is composed of two dense layers, one recurrent LSTM layer and an output layer with SOFTMAX activation function; the model has a total of 4005 parameters and ended up with a file size of 89 KB.

Figure 10 shows the prediction of the LSTM-RNN model on the training/validation recording. Figure 10a presents the network output probability of a class $C_j$ given an input X. Figure 10b shows the actual predicted class and the actual label of the example. The recording was set to have the gestures in order.

Figure 11 shows the predictions of the LSTM-RNN model on the four real time testing sessions. This figure presents the network output probability of a class $C_j$ given an input X and shows the actual predicted class and the actual label of the example in the four real time testing sessions. The recording was set to have the gestures in order.

Figure 10 and Figure 11 allow to get a better intuition about the results shown in Table 2 and Table 3. Figure 10 shows how the output probability of the LSTM-RNN model is stable when predicting each class, except for some wrong predictions. This was expected because the LSTM-RNN model achieved a 0.99 accuracy in the training/validation data-sets. In contrast, Figure 11 shows a more erratic behavior in the output probability of the LSTM-RNN model during real time classification, especially in classes with more wrong predictions. It can be seen how the LSTM-RNN model tends to fail in classifying one of the classes even tho it generalized and predicts correctly the other four, this figure shows how the model tends to fail either in class 1 or class 4. Figure 11 depicts the sensitivity of the model when the acquisition system is taken off and put back on, thus its varying accuracy across different acquisition sessions.

Based on the above, an ideal model should exhibit a behavior in the output probability as the one seen in Figure 10, however, when implementing the system in real-time, there are variables that change the model performance across different acquisition sessions. As mentioned earlier, small changes in the EMG acquisition device and noise in the EMG signal are the ones with grater impact and cause the model to start confusing classes as Figure 11 shows.

3.4. Model Comparison

This work presented a LSTM-RNN model composed of an LSTM layer wrapped by dense layers, capable of performing in real-time.

Table 5. Comparison of the proposed LSTM-RNN model with state-of-the-art RNN models found in literature.

Work	Channels	Classifier	Test Accuracy
Samadani [36]	12	Stacked LSTM	89.50%
Jabbari et al. [35]	8	BILSTM-AT	$91.40\pm 3.00\%$
Zhang et al. [35]	8	Stacked GRU	89.60%
Jiang et al. [39]	8	Stacked LSTM	97.10%
Xie et al. [41]	16	LSTM	78.13%
He et al. [42]	12	LSTM-Dense	75.50%
Hu et al. [37]	10	CNN-RNN	87.00%
Ours	4	LSTM-Dense	$\mathbf{87}.\mathbf{29}\pm \mathbf{6}.\mathbf{94}\%$

shows the comparison of the proposed LSTM-RNN model with other state-of-the-art RNN models found in the literature; the comparison included the number of EMG channels used, which is an important parameter when performing real-time operation, the classifier type and the test accuracy achieved by the classifier. It is important to note that most of these studies tested the model on static data and not in real-time.

As can be seen in Table 5 the proposed LSTM-RNN model was capable of achieving state-of-the-art performance while using a much simpler model and less number of EMG channels (only four EMG channels); reducing both the complexity of the model and the number of EMG channels is key when working on a classifier that must be embedded in a portable device such as a hand prosthesis or a wearable Human–Machine Interface. Another key takeaway from the comparison is that most of the mentioned state-of-the-art RNN models where tested only on static data and online datasets, which does not guarantee the performance in real-time operation; in contrast, the proposed LSTM-RNN model was tested in real-time and achieved a comparable accuracy score with other state-of-the-art RNN models.

4. Conclusions and Future Work

The proposed LSTM-RNN model for gesture classification proved to be a suitable approach for gesture recognition using EMG signals. The LSTM-RNN model was able to obtain high accuracy during training and validation phase; indicating that the model was able to learn the patterns in the EMG signals and generalize on new data. Furthermore, it is important to highlight that the model is small enough to be embedded in a micro-controller, which is fundamental to implement it for a commercial or industrial application. The proposed LSTM-RNN model was able to perform at a similar level of accuracy than other state-of-the-art RNN models while reducing the model complexity and the number of EMG channels, increasing its potential to be embedded and its commercial scalability. With this in mind, the proposed model shows promising results for its industrial or commercial application, even though further improvements must be made to achieve this development state, particularly in the EMG acquisition system. This research allowed to go beyond static data classification, giving valuable insights regarding the challenges of real time classification.

It was found that the reduced performance during real-time testing was due to the high sensitivity of the model to small changes in the EMG armband position and noise present in the signal when acquiring the EMG. However, although the model accuracy varies with different postures of the acquisition system, it is still able to generalize to new data. With this in mind, having a better EMG acquisition device and a better acquisition process, where noise and variations in position across different sessions are minimized, will probably improve the model performance considerably, without the need of changing the model.

In general, the proposed approach using deep neural models its effective for finding patterns in stochastic signals like EMG signals, making deep learning models a great tool for gesture recognition tasks based on this type of signals. It is important to note that in order to build an effective and robust gesture recognition system, not only is important to build and optimize a good model, it is also fundamental to acquire quality data and perform a good pre-processing of it. It is only when an optimal point between data and model is found that the overall system can find the best performance.

In future work the EMG acquisition armband will be upgraded with the objective of acquiring more data and improving its quality; new pre-processing and feature extraction techniques will be tried, aiming to reduce the system sensitivity to noise in the EMG signal and variations in the acquisition devise; moreover, the gesture recognition model will be optimized with the new data and it will be embedded in a microcontroller in order to control an active hand prosthesis using different hand gestures.