Estimation of Hand Motion from Piezoelectric Soft Sensor Using Deep Recurrent Network

Abstract

Soft sensors are attracting significant attention in human–machine interaction due to their high flexibility and adaptability. However, estimating motion state from these sensors is difficult due to their nonlinearity and noise. In this paper, we propose a deep learning network for a smart glove system to predict the moving state of a piezoelectric soft sensor. We implemented the network using Long-Short Term Memory (LSTM) units and demonstrated its performance in a real-time system based on two experiments. The sensor’s moving state was estimated and the joint angles were calculated. Since we use moving state in the sensor offset calculation and the offset value is used to estimate the angle value, the accurate moving state estimation results in good performance for angle value estimation. The proposed network performed better than the conventional heuristic method in estimating the moving state. It was also confirmed that the calculated values successfully mimic the joint angles measured using a leap motion controller.

1. Introduction

In recent years, human–machine interaction techniques have been spotlighted in robotics research. For human–machine interaction, accurate representation of the human body state is important. In studies on human motion recognition, various sensors are used to measure human body states [1,2]. In particular, soft sensors have received significant attention with respect to human–machine interaction and are applied to many wearable applications due to their high flexibility and adaptability [3,4,5].

The raw data acquired from various sensors need to be appropriately processed to be used in a motion state estimation. However, if the sensor has a nonlinear characteristic, it is hard to analyze or utilize. To solve this problem, recent studies have proposed the use of networks using deep learning to predict the nonlinear correlation [6,7,8,9]. Given that the soft sensor has higher nonlinear characteristics compared to traditional sensors [10], deep learning has been applied to characterize soft sensors and motion recognition in [11,12,13].

Among the various soft sensors, those made of piezoelectric materials are well suited for human–machine interactions in wearable devices [14,15] due to their flexibility associated with micro-scale thickness, low power, and convenience in installation [16,17]. In our previous studies, we demonstrated the performance of smart gloves using flexible piezoelectric sensors and demonstrated the interaction ability of these gloves in virtual reality applications [18,19]. Apart from these advantages, piezoelectric sensors exhibit high nonlinearity and noise similar to other soft sensors. In this study, we adopted deep learning as a method to address this problem. Specifically, the offsets of piezoelectric sensors were randomly changed in addition to the nonlinearly in a real-time system. Since we follow the offset value only in the case that the sensor does not move, the accuracy of the sensor offset value is related to the estimation of the moving state. At the same time, the angle value is calculated using the sensor offset value. This means that a good estimation of the moving state leads to a good result for angle estimation. Thus, we propose a network to predict the moving state for stabilization of the offset value. Moreover, we incorporated the trained network into the interface system for the smart glove to recognize hand joint angles.

Given that our goal is to apply the proposed network to a the real-time virtual reality interaction system, we designed it to be suitable for processing sequential data at high speed. In the proposed network, sequential sensor data, which are analog voltages, are used as the input data. We introduce the structure of the network using the Long-Short Term Memory (LSTM) units for predicting outputs and utilized the estimated result in our glove system. Several previous studies have already used the LSTM unit for processing of sequential sensor data [11,12,20,21] and the performance of the network was demonstrated.

Training of the network was conducted using a leap motion controller as the ground-truth data. The moving state, the target data for training, was defined with the measured angle value, which means that if the value is 1, the sensor is in moving state and static when value is 0. The performance of the network was demonstrated by two different experiments, with pre-acquired datasets and using a real-time interface system with a smart glove. We evaluated the accuracy of network predictions using our training and test dataset. In addition, the performance of the network for real-time application was evaluated using two criteria, an identical ratio of moving states and the similarity of the joint angles. An overview of the proposed system is shown in Figure 1.

Our major contributions are as follows: (i) proposing a deep recurrent network that is trained by the novel input data, piezoelectric soft sensor; (ii) applying the proposed network into the real-time VR interface system; and (iii) showing the improvement of performance by applying deep learning into sensor data processing compared to our prior works.

This paper is organized into the following sections. In Section 2, we introduce the interface system used for the smart glove. In Section 3, we propose and evaluate the deep learning network for sensor data processing. We demonstrate the performance of the real-time application of network in Section 4. Section 5 is a brief summary of the main conclusions.

2. Interface System

Herein, we introduce the interface system used to receive signals from the smart glove and to calculate the joint angles of the hand motion using a piezoelectric voltage.

2.1. Glove Structure

In our previous studies [18,19], data from the fabricated sensor were received by an interface board. In this study, we use the same interface board and piezoelectric sensors proposed in previous works. Especially, the sensor used in this study is proposed in [18]. A PVDF film was used to generate a piezoelectric effect, and the sensor was fabricated with a uniform beam structure with thin PVDF and Mylar layers. Detailed information on the structure of the interface board and sensor is outlined in [18]. Different from previous works, we changed the structure of the glove and interface program. The entire glove, including the main component and ring component, was fabricated using Ecoflex 00-30 silicone rubber. The new design improves the flexibility of the glove and wear sensation. Specifically, we fabricated the ring part using several lines that were combined with the straight-finger section of the glove. Small neodymium magnets were included at the end of the ring component and they were linked to each other to form a closed loop. The structure of the glove and the position of the sensors are shown in Figure 2a,b and an image of the glove being worn with the caption of each part is shown in Figure 2c–e.

2.2. System Setup

In this study, we need to simultaneously acquire and process data from two different sensors: a piezoelectric sensor and the leap motion controller. The piezoelectric sensor data are our main interest to be trained, and the data from the leap motion controller are used as the ground-truth data for training. Thus, we programmed the interface to implement two virtual hands, as shown in Figure 3. The virtual hand located on the left side was implemented based on finger joint angles estimated by the piezoelectric sensors, and the right hand shows joint angles measured directly using a leap motion controller. The detail process of angle calculation is reviewed in Section 3. Given that the proposed program received two kinds of sensor data, it was necessary to synchronize the data. The data acquisition rate of the piezoelectric sensor is 100 Hz, while the rate of the leap motion controller is 60 Hz. Therefore, the leap motion values were synchronized to the piezoelectric sensor outputs using the following approximation.

$$\begin{array}{c}\hfill {\theta }_l\left(t_n\right)={\theta }_l(t_{k-1}^{\left(l\right)})+({\theta }_l(t_k^{\left(l\right)})-{\theta }_l(t_{k-1}^{\left(l\right)}))\times \frac{t_n-t_{k-1}^{\left(l\right)}}{t_k^{\left(l\right)}-t_{k-1}^{\left(l\right)}}\end{array}$$

(1)

where ${\theta }_l\left(t_n\right)$ is the angle at the time of interest $t_n$, ${\theta }_l(t_k^{\left(l\right)})$ is the raw data from the leap motion controller, and $t_k^{\left(l\right)}$ is time of leap motion controller data at kth order, for the condition $t_{k-1}^{\left(l\right)}\le t_n<t_k^{\left(l\right)}$.

In addition, the offset voltage applied to the sensor from the interface board is 2.5 V. This voltage is changed when the piezoelectric voltage is generated from the piezoelectric sensor. The range of the analog voltage transferred to the system from the interface board is 0–5 V.

3. Application of Recurrent Network

In this section, we propose the deep recurrent network to process the piezoelectric sensor data. To make a network that works well with the input sensor data, we need to know about the characterization of sensor data. In addition, to train the network with good performance, input data and the ground-truth data should be well defined. Therefore, we explain our motion recognition method and discuss the characterization of the piezoelectric sensor. Afterwards, we discuss how to obtain sensor data for the training. Finally, the structure of the proposed network using two different recurrent units, RNN and LSTM, is introduced, and the performance between two units is compared.

3.1. Conventional Angle Recognition Method

In our previous study, we used the following relationship between the curvature and the voltage of the piezoelectric sensor [22]:

$$\begin{array}{cc}\hfill q\left(t\right)& =Cv\left(t\right)+ϵ\overline{k}\left(t\right),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill \dot{q}\left(t\right)& =-\frac{v\left(t\right)}{R_{load}},\hfill \end{array}$$

(3)

where q is the charge stored in the piezoelectric sensor, C is capacitance, v is the piezoelectric voltage, $ϵ$ is electromechanical coupling coefficient, $\overline{k}$ is the mean curvature, and $R_{load}$ is the load resistance.

Applying the Laplace transformation, Equations (2) and (3) can be written as:

$$\begin{array}{cc}\hfill Q\left(s\right)& =CV\left(s\right)+ϵ\overline{K}\left(s\right),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill sQ\left(s\right)& =-\frac{V\left(s\right)}{R_{load}},\hfill \end{array}$$

(5)

in the s domain.

Inserting Equation (4) into Equation (5) by substituting for $Q\left(s\right)$, we have:

$$s·(CV\left(s\right)+ϵ\overline{K}\left(s\right))=-\frac{V\left(s\right)}{R_{load}}.$$

(6)

By reorganizing Equation (6) as the relation between $K\left(s\right)$ and $V\left(s\right)$, we have:

$$ϵ\overline{K}\left(s\right)=-\frac{1+s·C·R_{load}}{s·R_{load}}V\left(s\right).$$

(7)

When the piezoelectric sensor has low capacitance value and load resistance value ($1>>C·R_{load}$), Equation (7) reduces to:

$$ϵ\overline{K}\left(s\right)\simeq -\frac{1}{s·R_{load}}V\left(s\right).$$

(8)

Equation (8) can be expressed as follows in the time domain after implementing the inverse-Laplace transform:

$$ϵ\overline{k}\left(t\right)\simeq -\frac{1}{R_{load}}·{\int }_0^{t}v\left(\tau \right)d\tau .$$

(9)

In the beam structure, the value of the mean curvature is related to the change of the slope between the start and end of the beam, and the slope can be expressed as the angle of the sensor [22]. From these relations, we derive the following equation using the angular change of the sensor and piezoelectric voltage [18]:

$$\begin{array}{cc}\hfill \theta \left(t_n\right)& =a\times \sum _{m=1}^n\left[\left(v\left(t_m\right)-O\left(t_m\right)\right)\times \mathsf{\Delta }{t}_m\right]\hfill \\ \hfill n& =1,2,...\hfill \end{array}$$

(10)

where $\theta$ is the angle variation of the piezoelectric sensor, A is the gain including all proportional terms, O is the offset, $\mathsf{\Delta }{t}_m$ is the time interval, and $t_n={\sum }_{m=1}^n\mathsf{\Delta }{t}_m$.

3.2. Estimation of Moving State and Offset Voltage

Assigning $t=0$ in Equation (9), $v\left(0\right)$ is determined with the initial value of mean curvature $\overline{k}\left(0\right)$. This means that, if $\overline{k}\left(0\right)$ is nonzero, there is a voltage offset. Ideally, the mean curvature is zero at the original position. However, it is difficult to achieve zero mean curvature in a real case. In addition, given that a finger moves continuously in real-time experiment, the initial mean value can have small variation when it comes back to the original position. The difference between original position and varied position also affects the offset value.

As shown in Figure 4, calculation of joint angle values using improper offsets results in incorrect results. If $v\left(t_m\right)$ is the same as the $O\left(t_m\right)$, as shown in Figure 4a, the joint angle is calculated normally. However, if there is a difference between the estimated offset and real offset of the raw data, as shown in Figure 4b,c, the error $(v\left(t_m\right)-O\left(t_m\right))$ is integrated as the time $\mathsf{\Delta }{t}_m$. Therefore, the selection of an appropriate offset voltage value is an important issue. In [18], we defined the offset value $O_a$ as the average of 3000 data, such that:

$$O_a\left(t_n\right)=\frac{{\sum }_{m=n-2999}^{n}v\left(t_m\right)}{3000}.$$

(11)

Using the average value is a reasonable choice for following the tendency of the signal. However, the average value can oscillate when data having a large difference are entered, and a slight offset change is applied slowly. To solve this problem, we determined the offset voltage with voltages for non-movement to update $\overline{k}\left(0\right)$ in [19]. Assuming only curvature change of the sensor results in significant change of piezoelectric voltage sensor, the heuristic moving state $m_h$ is estimated as follows:

$$m_h\left(t_n\right)=\left\{\begin{array}{cc}0,& |\mathsf{\Delta }v\left(t_n\right)|<v_{th}\\ 1,& |\mathsf{\Delta }v\left(t_n\right)|\ge v_{th}\end{array}\right.$$

(12)

$$\mathsf{\Delta }v\left(t_n\right)=v\left(t_n\right)-v\left(t_{n-1}\right).$$

(13)

where $v_{th}$ is the threshold voltage for deciding the moving state. We consider the finger as moving when $m_h\left(t_n\right)=1$ and stopped when $m_h\left(t_n\right)=0$. This method can continuously renew offset change that occurs via variation of $\overline{k}\left(0\right)$ faster than previous one. However, heuristic moving state estimation of Equation (12) confronts following problems in a real-time system:

• If noise is added to v, it is hard to distinguish noise from curvature change.
• Input noise can be random value in case that the system is operated in different environment.
• When noise is random, $v_{th}$ is hard to determine.

For these reasons, it is difficult to estimate $m_h$ in noisy environment. Therefore, we propose a network to train a moving state using recurrent neural networks as alternative.

3.3. Data Acquisition

For network training, we need the dataset including the input data and the ground-truth target data which need to be predicted. In this study, we extracted the ground-truth moving state from the finger joint angle measured by leap motion controller. Different from other vision sensors, the leap motion controller facilitates the direct acquisition of the finger joint angle without additional processing. Although the measured angle from the leap motion controller exhibits differences in the real joint angle, as compared to Chophuk et al. [23], we chose the leap motion controller as a ground-truth sensor due to its characteristic and because it can facilitate large dataset generation.

To obtain a dataset for learning, we conducted an experiment using the interface mentioned in Section 2. For data acquisition, we set up the experiment as follows:

• We used the soft sensor located on the glove, and its piezoelectric voltage was measured with the interface board.
• Joint angle for acquiring moving state was simultaneously measured with leap motion controller.
• Single movement was composed with flexion and extension.
• The range of flexion angle was determined randomly between 30 degrees and 90 degrees.
• Movements for training set were conducted 1080 times, and those for the test set were conducted 270 times.

The decision of the moving state $m_{gt}$ is given by:

$$m_{gt}\left(t_n\right)=\left\{\begin{array}{cc}0,& |\frac{{\sum }_{k=n-9}^n\theta \left(t_k\right)}{10}-\theta \left(t_n\right)|<\mathsf{\Delta }{\theta }_{th}\\ 1,& else,\end{array}\right.$$

(14)

where $\mathsf{\Delta }{\theta }_{th}$ is the threshold angle variation. We assumed that the finger is moving when the difference between the average of 10 recent values and the new value is larger than $V_{th}$. We selected $\mathsf{\Delta }{\theta }_{th}$ as 2 degrees based on preliminary tests.

Figure 5 displays the sample of the dataset acquired from the experiment. The network was trained using the input data in Figure 5a to estimate the moving state in Figure 5c.

Since our goal was to extract real-time moving state with using piezoelectric sensor outputs, we preprocessed the piezoelectric voltage into sequential input data $X\left(t_n\right)=[x\left(t_{n-9}\right),...,x\left(t_n\right)]$ using the sliding window method, where

$$x\left(t_n\right)=[v\left(t_n\right),\dot{v}\left(t_n\right)],$$

(15)

$$\dot{v}\left(t_n\right)=\frac{v\left(t_n\right)-v\left(t_{n-1}\right)}{t_n-t_{n-1}}.$$

(16)

In addition, we defines target data of the network as $y\left(t_n\right)=m_{gt}\left(t_n\right)$. Summaries of the processed training and test sets are shown in

Table 1. Summary for the dataset.

	Number of the Data	Number of the Movement
Training Set	214,775	1080
Test set	59,660	270

.

3.4. Proposed Network

To train the moving state of the sensor, we used a recurrent neural network (RNN), which is appropriate for training a sequential dataset. Using input vector $x_t$ and previous hidden state $h_t$, a simple RNN determines the hidden state as follows:

$$h_t=tanh(x_t{W}_x+h_{t-1}{W}_h+b),$$

(17)

where $W_h$, $W_x$, and b are the network parameters.

Given that hidden state $h_t$ is determined with previous hidden state $h_{t-1}$, simple RNN can contain sequential information based on the recurrent hidden state. However, when the length of the data becomes larger, the gradient vanishes and explodes. To solve the problem and transfer the hidden layer information deeper, Long Short-Term Memory is proposed in [24]. Using cell vector $c_t$ and several gates, the hidden state $h_t$ is calculated as follows:

$$\begin{array}{cc}\hfill f_t& =\sigma (x_t{W}_x^{\left(f\right)}+h_{t-1}{W}_h^{\left(f\right)}+b^{\left(f\right)}),\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill g_t& =tanh(x_t{W}_x^{\left(g\right)}+h_{t-1}{W}_h^{\left(g\right)}+b^{\left(g\right)}),\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill i_t& =\sigma (x_t{W}_x^{\left(i\right)}+h_{t-1}{W}_h^{\left(i\right)}+b^{\left(i\right)}),\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill o_t& =\sigma (x_t{W}_x^{\left(o\right)}+h_{t-1}{W}_h^{\left(o\right)}+b^{\left(o\right)}),\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill c_t& =f\odot c_{t-1}+g_t\odot i\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill h_t& =o\odot tanh\left(c_t\right)\hfill \end{array}$$

(23)

where $i_t$, $f_t$, and $o_t$ are input gate, forget gate, and output gate vector, respectively; $g_t$ and $h_t$ are cell input and output activation functions; W and b denote weight matrices and bias vector of the network; ⊙ is the element-wise product of the vectors; and tanh is output activation function. $h_t$ of LSTM unit can save more sequential information than $h_t$ of simple RNN unit. The network size of LSTM is four times larger than RNN. We applied both recurrent units into the proposed network and compared the performance.

To implement the real-time moving state estimation, we propose the network as shown in Figure 6. At each time step $t_n$, $X\left(t_n\right)$ passes two recurrent layers and produces the hidden layer $h\left(t_n\right)$, which contains the sequential information of $X\left(t_n\right)$. To convert the sequential information into a single moving state, the fully connected layer combines the information of $h\left(t_n\right)$ and determines the final output. Given that state 1 refers to moving and 0 refers to stopping, the final output of the network $p\left(t_n\right)$ states the probability of moving. Therefore, we define the predicted moving state $m_p\left(t_n\right)=\left[p\left(t_n\right)\right]$.

We trained the network for 100 epochs and used stochastic gradient descent (SGD) as the optimizer. In addition, we applied a dropout layer after the final recurrent layer at a rate of 50% rate in training to avoid overfitting.

As the recurrent unit, we used simple RNN and LSTM units and compared the result.

Table 2. Training and test accuracy for proposed network.

		1	2	3	4	5	6	7	8	9	10	Average
Training	RNN	85.51%	85.57%	85.87%	85.91%	85.88%	85.84%	85.90%	85.54%	85.54%	85.49%	85.71%
	LSTM	85.88%	85.39%	85.73%	85.86%	85.92%	85.87%	85.87%	85.93%	85.73%	85.94%	85.81%
Test	RNN	86.31%	86.86%	86.36%	85.74%	86.80%	87.50%	86.16%	86.56%	87.16%	86.94%	86.64%
	LSTM	86.95%	86.93%	87.06%	87.23%	87.24%	87.27%	87.02%	87.31%	87.06%	86.97%	87.10%

displays the result for accuracy of the training set and test set for both methods.

From the result, the model using LSTM unit exhibited slightly higher accuracy for the test and training set. In addition, the test accuracy of the model using the LSTM unit is more consistent than that of the model using simple RNN. The result indicates that only simple substitution for the recurrent unit without network structure change results in performance improvement in the proposed network. As stated in [25], there exists a modeling weakness of a simple RNN structure, and the LSTM structure is used to overcome the vanishing error [24,26]. The vanishing of the error gradient is increased when the long-term dependencies of input–output are enlarged. it means the LSTM unit can transfer the error information deeply than the RNN unit during the training when the training data includes long recurrent time steps. Since the single training data of our work include 10-time steps of sensor data, the LSTM unit can transfer the information deeply than the simple RNN unit when training the network. Therefore, we conclude that the LSTM unit is better suited than the simple RNN unit for soft sensor characterization, and chose the LSTM unit for the proposed network.

4. Result and Discussion

We evaluated our model using a real-time glove system. The experiment was conducted with an interface and the system stated in Section 2. Two experiments were conducted in this study. We initially demonstrated network performance in real-time moving state estimation and established the validity of network application for joint angle calculation. To estimate moving states $m_p$ of each joint, we utilized the network proposed in Section 3.4 and estimated the moving state with all X from different sensors received by interface board. For calculation of the joint angle, we updated the offset values using $m_p$ and calculated angle degrees for all the sensor locations.

Bending thumb and clenching fist motion except for the thumb were selected as experimental motions. The motions are shown in Figure 7. Each motion was performed five times for one test. Using the thumb bending motion, the three sensors installed in the thumb, thumb1, thumb2, and thumb3, were evaluated. The sensors installed in the index finger, index1, index2, and index3, were evaluated using the motion clenching fist except the thumb. Experiments were conducted 80 times for moving state estimation and 50 times for angle value calculation.

4.1. Moving State Estimation

To evaluate the performance of the proposed network, we compared the result $m_p$ with $m_h$ from Equation (14).

Table 3. Coincidence of moving state estimations with leap motion controller.

	thumb1	thumb2	thumb3	index1	index2	index3
proposed method ($m_p$)	80.02%	84.44%	83.25%	84.47%	81.12%	82.62%
heuristic method ($m_h$)	61.25%	63.58%	78.41%	65.79%	54.31%	53.90%

shows an identical ratio of $m_p$ and $m_h$ with $m_{gt}$. In Table 3, it is evident that the coincidence between $m_p$ and $m_{gt}$ is 5–25% higher than the coincidence between $m_h$ and $m_{gt}$. This shows that the predicted $m_p$ is more similar to the ground-truth data than $m_h$. In addition, while $m_h$ confuses curvature change with noise, as shown in Figure 8d, the result in Figure 8c shows that $m_p$ rarely confuses noise with sensor movement and recognizes only sensor movement.

Based on this result, we demonstrate that the network exhibits good performance for moving state estimation in real-time, multiple sensor data as single pre-acquired data. The result indicates that the proposed network can also predict moving state in real-time system, and chasing $\overline{k}\left(0\right)$ for offset renewing in a real-time system is also facilitated with the proposed network.

4.2. Angle Value Estimation

Based on good network performance for predicting $m_p$, we show the ability for joint angle estimation by utilizing $m_p$ in this subsection. For demonstration, we compared the result for the methods used in our earlier studies [18,19]. In [18], we assumed that the offset $O_a$ as average value of sensor data without using moving state. In [19], we estimated the moving state heuristically and determined the offset $O_h$. In this experiment, we assumed that the offset values were determined for the predicted moving state $O_p$, as the average of the recent 200 data for $m_p=0$.

For joint angle calculation, we applied offsets $O_a$, $O_h$, and $O_p$ in Equation (10). The calculated joint angles were compared to the joint angles measured by the leap motion controller, and the criteria are defined as the difference between the degrees.

Table 4. Average of angle degree difference between calculated angles and joint angles from leap motion controller.

	thumb1	thumb2	thumb3	index1	index2	index3
proposed method	6.79${}^{\circ }$	5.64${}^{\circ }$	9.40${}^{\circ }$	8.92${}^{\circ }$	12.88${}^{\circ }$	13.75${}^{\circ }$
average method	13.28${}^{\circ }$	10.72${}^{\circ }$	12.16${}^{\circ }$	9.94${}^{\circ }$	16.04${}^{\circ }$	20.91${}^{\circ }$
heuristic method	6.28${}^{\circ }$	7.07${}^{\circ }$	17.23${}^{\circ }$	11.76${}^{\circ }$	25.66${}^{\circ }$	25.91${}^{\circ }$

shows the average values of the angle difference in degrees between the joint angle from the leap motion controller and the calculated joint angles.

Table 5. Standard deviation of the degree difference.

	thumb1	thumb2	thumb3	index1	index2	index3
proposed method	5.92	4.97	7.73	7.02	10.01	10.83
average method	9.61	7.69	9.46	7.64	11.50	14.75
heuristic method	5.62	5.21	13.10	8.99	18.82	19.19

shows the standard deviation of the degree difference for each location. Figure 9 and Figure 10 show the graph for examples of angle value comparisons between the joint angles in each location.

From the results, the averages and standard deviations for degree differences facilitate the lowest value of the joint angle calculated using $O_p$. In addition, as shown in Figure 9 and Figure 10, the angle estimation based on the proposed method shows the most similar value compared to the ground-truth data. The result demonstrates that a good estimation of the moving state leads to a good result for angle estimation. One notable result is the joint angle calculated using $O_h$ is very stable occasionally but does not always exhibit good result for angle estimation compared to using $O_a$. This means that incorrect estimation of the moving state can lead to poor results for angle estimation. Therefore, we demonstrate the importance of precise estimation of moving state via two results: good estimation of joint angle using $O_p$ and some estimation of joint angle using $O_h$.

5. Conclusions

In this paper, we propose a virtual reality interface system with a smart glove system using a deep learning network to predict the moving state of piezoelectric soft sensors. To predict the nonlinear characteristic of sequential sensor data, we use LSTM as the recurrent unit of the proposed network. This network predicted the moving state of a single soft sensor with 87% accuracy on the test dataset and in excess of 80% accuracy with real-time, multiple data. In addition, the joint angle calculated using the proposed method has shown the best similarity to leap motion controller compared to the other methods. Based on the result, we concluded that deep learning can be applied to the inference of the piezoelectric sensor characteristic in a real-time system. In addition, precise estimation of the sensor moving state affects the estimation of the joint angles in real-time application. For future works, we will apply deep learning networks to other systems using piezoelectric sensors and broaden the prediction of feature of these sensors.