Development of a Wearable Glove System with Multiple Sensors for Hand Kinematics Assessment

Abstract

In traditional hand function assessment, patients and physicians always need to accomplish complex activities and rating tasks. This paper proposes a novel wearable glove system for hand function assessment. A sensing system consisting of 12 nine-axis inertial and magnetic unit (IMMU) sensors is used to obtain the acceleration, angular velocity, and geomagnetic orientation of human hand movements. A complementary filter algorithm is applied to calculate the angles of joints after sensor calibration. A virtual hand model is also developed to map with the glove system in the Unity platform. The experimental results show that this glove system can capture and reproduce human hand motions with high accuracy. This smart glove system is expected to reduce the complexity and time consumption of hand kinematics assessment.

1. Introduction

Hand function assessment plays an essential role in the recovery of stroke patients. Rehabilitation of hand function always requires long-term training after medical treatment. Traditional hand function assessment methods that have been used by physicians include Fugl-Meyer assessment (FMA), the action research arm test (ARAT), Jebsen–Taylor test (JTT), or Wolf motor function test (WMFT) [1,2,3,4,5]. With these methods, the patients must carry out some prescribed actions to showcase their hand function, and rating scores are given according to the patients’ performance. However, assessment with these methods often takes several hours to complete and does not reveal enough details during the rehabilitation process. Therefore, it is important to develop hand function assessment equipment that can help physicians make a quantitative analysis of patients’ rehabilitation status.

One obvious solution to get hand gesture information is using a visual camera system [6,7,8]. The advantage of using such a system is that it does not require the patients to wear any equipment and it can capture the gesture information of multiple patients at the same time. The disadvantage is that it can only work effectively in good light conditions and its shooting angle is highly restricted by the local environment. IR or radar sensors can be another non-contact alternative for hand gesture capture or recognition [9,10].

In recent years, data gloves have been widely used in human–computer interaction studies [11]. With the advancement of sensor technology, the cost of data gloves has been greatly reduced to suit daily medical rehabilitation needs [12]. Depending on their working mechanism, data gloves can generally be divided into these main types: mechanical, flexible sensing, optical fiber sensing, and inertial sensing. A mechanical data glove can be considered as a type of glove-like haptic device with force feedback. Blake et al. proposed a haptic glove for virtual reality applications based on magnetorheological (MR) brakes in 2009 [13]. Ma et al. developed a five-finger haptic glove using a worm-geared motor in 2015 [14]. Chiri et al. developed a multi-phalange finger module for post-stroke rehabilitation in 2012 [15]. Gu et al. presented a low-cost exoskeleton glove with passive force feedback in 2016 [16]. Unlike that of mechanical data gloves, the performance of flexible sensing data gloves largely depends on the repeatability and linearity of the sensor itself. Kostas et al. presented a data glove with flexible sensors early in 2003 [17]. Togetti et al. presented a sensing glove made of conductive elastomer materials in 2006 [18]. Shen et al. also developed a soft stretchable bending data glove that incorporates a sensor based on ethylene propylene rubber (EPR) in 2016 [19]. However, the most successful commercial data glove systems today are mainly based on optical fiber sensors, such as CyberGlove and 5DT data gloves. Innovations have also occurred in the research community for the design of optical fiber data gloves. Fujiwara et al. discussed a low-cost and flexible optical fiber glove to measure joint angles in 2014 [20]. Da Silva et al. developed a sensing glove based on optical fiber Bragg grating (FBG) sensors in 2011 [21]. Although optical fiber-based data gloves such as CyberGlove and 5DT data gloves can provide sufficiently precise joint angle data for rehabilitation applications, they are still too expensive for daily rehabilitation at home. Low-cost MEMS-based inertial sensors have been investigated in a number of studies on data gloves or human interactions due to their ability to provide stable attitudes in a three-dimensional (3D) space and ease of integration with PCBs. Lin et al. presented a data glove system with six-axis inertial sensors for stroke evaluation [22,23]. Choi et al. developed a low-cost inertial measurement unit (IMU) wearable sensing glove [24]. Liu et al. designed a novel inertial and magnetic unit (IMMU)-based data glove with an optimized sensor layout [25].

The data glove system discussed in this paper aims to help physicians evaluate patients’ hand function during rehabilitation. To get the joint angles of hand attitudes, 12 nine-axis inertial sensors are integrated with the Micro Controller Unit (MCU) and Universal Asynchronous Receiver/Transmitter (UART) modules on the glove. Several calibration algorithms are implemented to avoid errors caused by the accelerometer, gyroscope, and magnetometer. In addition, a human–computer interaction module is developed in the Unity platform. A grasping ball experiment is conducted to demonstrate how the glove system can be used to evaluate dynamic hand functions.

2. System Architecture

The workflow of our data glove system is demonstrated in Figure 1. First, the sensor units installed on the data glove obtain the raw information of hand movements. Then, the MCU controller initializes the data acquisition process, in which the sensor data are received with a Serial Peripheral Interface (SPI) bus and transmitted to the computer through a USB connection line or a Bluetooth transmitter. Finally, the data processing module verifies the data frames and calculates the angles of joints for hand function evaluation. Moreover, the human–computer interaction module uses the joint angle data to drive hand modeling in the Unity platform.

2.1. Hardware Design

The hardware of our data glove system includes several parts: (1) some IMMUs for capturing the angles of hand joints; (2) a microcontrol unit for data collection and transmission; (3) a USB connection wire or wireless Bluetooth transmitter; and (4) a PC or laptop for receiving sensor data and reconstructing hand motions. The MEMS-based sensors are uniquely suited for wearable applications due to their advantages of low power consumption, compactness, and high precision. MPU-9250 units from InvenSense, Inc. are used to capture the angles of joints when hand exercises are done. The nine-axis sensor MPU-9250 can be considered as a combination of a six-axis IMU and a three-axis compass, one is the six-axis IMU MPU6515, the other is the AK8963 3-axis magnetometer from Asahi Kasei Microdevices (AKM). The angular velocity range of the gyroscope is set to ±1000°/s and the acceleration range of the accelerometer is set to ±8 g. The sample rate is set to 100 Hz for all the sensors, which is high enough to capture most human hand motions. To achieve better fusion performance, a multi-sensor fusion algorithm is developed to work with the onboard motion fusion processor that comes with MPU-9250. A Leonardo microprocessor chip is used to collect data from the IMMUs. The microprocessor controller has 21 in–out digital ports, 7 Pulse Width Modulation (PWM) ports and 6 analog input ports to connect with 12 sensors via an SPI bus, as shown in Figure 2. Both the wired UART connection and Bluetooth wireless connection are tested in our glove system. The results show that the wired UART connection is more efficient and reliable than Bluetooth wireless connection, especially at a high transmission rate.

2.2. Analysis of Hand Joints and Layout of Sensors

Human hand motions can be represented by the rotation of hand joints, so the number and position of IMMU sensors can be determined according to the anatomical structures of the hand. A human hand includes several types of bones and tissues such as muscles and ligaments. As shown in Figure 3, the joints of the index finger, middle finger, ring finger, and little finger of a human hand can be divided into three main categories: distal interphalangeal joint (DIP), proximal interphalangeal joint (PIP), and metacarpophalangeal joint (MP). As for the thumb, it only has two joints: interphalangeal joint (IP) and MP. The MP is connected to the carpometacarpal joint (CM) for hand rotation. IMMU sensors should be placed near the abovementioned joints to measure the bending status of fingers. An understanding of the parent–child relationship between joints is a prerequisite for analyzing finger motions. The movement of the parent joints has a knock-on effect on all the child joints. The local x–y–z coordinates should be established to describe the rotation of each joint, as shown in Figure 4. The bending of a finger can be defined as a rotation around the x-axis. Empirically, the bending angle of the DIP joint can be approximately considered as 1/3 of the bending angle of the PIP joint, which has no obvious influence on the reconstruction of hand gestures.

Therefore, 12 IMMU sensors are placed on hand phalanges and one IMMU sensor is placed on the back of the hand, as shown in Figure 4. Joint angles and hand gestures can be estimated by the orientation of 11 IMMU sensors. Two generations of data gloves have been developed in our lab. For the 1st generation glove, all the sensors are connected by soft wires; it is easy to fabricate but may suffer from sealing off and bad contact problems. For the 2nd generation data glove, a flexible PCB design is adopted to enhance the system’s stability and reliability, as shown in Figure 5. The IMMU sensors are directly soldered on the flexible PCB without using any wire. Moreover, the flexible PCB is coated with a silicone layer to protect the electronic components from being corroded by moisture and sweat. The thickness of the silicone layer is set to 2 mm so as not to impair the flexibility of hand movements. The plug-in connector of the flexible PCB can be easily linked to a controller board. In this paper, the 1st generation data glove is used to demonstrate the feasibility of our algorithm and software.

3. Sensor Data Processing

The data processing is as follows: The microcontroller of the data glove sends the data from IMMU sensors to the computer frame by frame, and then the received data are preprocessed by parity frame check and frame decoding. The raw sensor data must be calibrated before they can be fused. Since the environmental influence on accelerometers and gyroscopes is mostly negligible, the calibration parameters of the accelerometer and gyroscope can be used for a long time after initial calibration. Unlike the accelerometer and gyroscope, the magnetometer needs to be recalibrated because it is subjected to the disturbance from environmental ferromagnetic materials. The attitude of each IMMU sensor can be estimated with a multi-sensor fusion algorithm using calibrated acceleration, angular velocity, and geomagnetism data. The orientation of the IMMU sensors on the glove of left hand is as defined in Figure 6 with the International Society of Biomechanics (ISB) recommendations on definitions of joint coordinates [26].

3.1. Sensor Calibration

Gyroscopes can be used to measure the rotation of hand joints with angular velocity. The angle of joint rotation can be calculated by integrating the angular velocity data over time. However, the error of angular velocity will accumulate after a long measurement process, resulting in a drift in the angle. Theoretically, the output value of a gyroscope should be zero after calibration without any motions. The zero deviation caused by the random error of the gyroscope can be estimated using the average filtering method. The corrected value of each axis can be obtained by subtracting the zero deviation error from the measured value. The error model of the gyroscope can be represented as follows:

$$\left[\begin{array}{c}{\mathit{g}}_{x\_c}\\ {\mathit{g}}_{y\_c}\\ {\mathit{g}}_{z\_c}\end{array}\right]=\left[\begin{array}{c}{\mathit{g}}_x\\ {\mathit{g}}_y\\ {\mathit{g}}_z\end{array}\right]-\left[\begin{array}{c}{\mathit{g}}_{x\_e}\\ {\mathit{g}}_{y\_e}\\ {\mathit{g}}_{z\_e}\end{array}\right]$$

(1)

where ${\mathit{g}}_x$, ${\mathit{g}}_y,$ and ${\mathit{g}}_z$ denote the raw data of the gyroscope; ${\mathit{g}}_{x\_c}$, ${\mathit{g}}_{y\_c}$, and ${\mathit{g}}_{z\_c}$ the corrected angular velocity data; ${\mathit{g}}_{x\_e}$, ${\mathit{g}}_{y\_e},\mathrm{and}{\mathit{g}}_{z\_e}$ the zero bias of three axes.

Possible errors of the accelerometer include zero deviation error, scale factor error, and non-orthogonal error. The accelerometer error model can be represented as follows [27]:

$$\left[\begin{array}{c}{\mathit{a}}_{x\_c}\\ {\mathit{a}}_{y\_c}\\ {\mathit{a}}_{z\_c}\end{array}\right]=\left[\begin{array}{ccc}{\mathit{m}}_{xx}& {\mathit{m}}_{xy}& {\mathit{m}}_{xz}\\ {\mathit{m}}_{xy}& {\mathit{m}}_{yy}& {\mathit{m}}_{yz}\\ {\mathit{m}}_{xz}& {\mathit{m}}_{yz}& {\mathit{m}}_{zz}\end{array}\right]·\left[\begin{array}{c}{\mathit{a}}_x-{\mathit{a}}_{x\_e}\\ {\mathit{a}}_y-{\mathit{a}}_{y\_e}\\ {\mathit{a}}_z-{\mathit{a}}_{z\_e}\end{array}\right]$$

(2)

where ${\mathit{a}}_i\left(i=x,y,z\right)$ denotes the raw data of the accelerometer; ${\mathit{a}}_{i\_c}\left(i=x,y,z\right)$ the corrected acceleration data; ${\mathit{b}}_i\left(i=x,y,z\right)$ the zero bias; ${\mathit{m}}_{ij}\left(i=x,y,z;i=j\right)$ the scale error factor; ${\mathit{m}}_{ij}\left(i=x,y,z;j=x,y,z;i\ne j\right)$ the non-orthogonal error factor. The calibration error $\mathit{e}\left({\mathit{m}}_{xx},\dots ,{\mathit{m}}_{zz},{\mathit{a}}_{x_e},\dots ,{\mathit{a}}_{z\_e}\right)$ is defined as the residual between the squared sum of corrected acceleration and squared gravitational acceleration:

$$\mathit{e}\left({\mathit{m}}_{xx},\dots ,{\mathit{m}}_{zz},{\mathit{a}}_{x\_e},\dots ,{\mathit{a}}_{z\_e}\right)={\mathit{a}}_{x\_c}^2+{\mathit{a}}_{y\_c}^2+{\mathit{a}}_{z\_c}^2-g^2$$

(3)

Gauss–Newton’s method is applied to estimate the unknown calibration parameters (${\mathit{m}}_{xx},\dots ,{\mathit{m}}_{zz},{\mathit{a}}_{x\_e},\dots ,{\mathit{a}}_{z\_e}$) in this part. The iteration of calibration error can be represented by the following equation:

$$e_{k+1}=e_k+{\beta }_k{d}_k$$

(4)

where ${\beta }_k$ is the damping control factor, which is used to control the convergence speed of the iteration algorithm. A larger ${\beta }_k$ value means a faster convergence speed but lower accuracy. $d_k$ is defined as the iterative direction:

$$d_k={\left(J^{T}J\right)}^{-1}\left(J^T\left(-e\right)\right)$$

(5)

where matrix J is defined as the Jacobian matrix of calibration error:

$$J\left(e\right)=\frac{\partial e}{\partial x}=\left[\begin{array}{ccc}\frac{\partial e}{\partial m_{xx}}& \cdots & \frac{\partial e}{\partial b_z}\\ ⋮& \ddots & ⋮\\ \frac{\partial e}{\partial m_{xx}}& \cdots & \frac{\partial e}{\partial b_z}\end{array}\right]$$

(6)

When the iteration time reaches its maximum or the convergence condition is satisfied, the unknown calibration parameters can be estimated. With $\epsilon$ defined as the convergence threshold, the convergence condition can be represented as follows:

$$\left|e\left(m_{xx},\dots ,m_{zz},a_{x\_e},\dots ,a_{z\_e}\right)\right|<\epsilon$$

(7)

The external interference of a magnetic field can be divided into hard magnetic interference and soft magnetic interference. The hard magnetic interference would cause the measured data to deviate from the sphere origin as a whole. The soft magnetic interference would cause the shape of data distribution to change from a standard sphere to an ellipsoid. The error model of the magnetometer can be represented as follows:

$$\left[\begin{array}{c}{\mathit{m}}_{x\_c}\\ {\mathit{m}}_{y\_c}\\ {\mathit{m}}_{z\_c}\end{array}\right]=\left[\begin{array}{ccc}{\mathit{k}}_x& {\mathit{k}}_y& {\mathit{k}}_z\end{array}\right]·\left[\begin{array}{c}{\mathit{m}}_x\\ {\mathit{m}}_y\\ {\mathit{m}}_z\end{array}\right]+\left[\begin{array}{c}{\mathit{c}}_x\\ {\mathit{c}}_y\\ {\mathit{c}}_z\end{array}\right]$$

(8)

where ${\mathit{m}}_i\left(i=x,y,z\right)$ denotes the raw data of the magnetometer; ${\mathit{m}}_{i\_c}\left(i=x,y,z\right)$ the corrected magnetic field data; ${\mathit{c}}_i\left(i=x,y,z\right)$ the hard magnetic calibration parameters; ${\mathit{k}}_i\left(i=x,y,z\right)$ the soft magnetic calibration parameters. The magnetometer should be rotated in all directions to find out the maximum and minimum values of each axis. The geomagnetic calibration parameters can be calculated as follows:

$$\left\{\begin{array}{c}{\mathit{c}}_x=\frac{{\mathit{X}}_{max}+{\mathit{X}}_{min}}{2}\\ {\mathit{c}}_y=\frac{{\mathit{Y}}_{max}+{\mathit{Y}}_{min}}{2}\\ {\mathit{c}}_z=\frac{{\mathit{Z}}_{max}+{\mathit{Z}}_{min}}{2}\end{array}\right.$$

(9)

$$\left\{\begin{array}{c}{\mathit{k}}_x=\frac{{\mathit{X}}_{max}-{\mathit{X}}_{min}}{{\mathit{X}}_{max}-{\mathit{X}}_{min}}\\ {\mathit{k}}_y=\frac{{\mathit{X}}_{max}-{\mathit{X}}_{min}}{{\mathit{Y}}_{max}-{\mathit{Y}}_{min}}\\ {\mathit{k}}_z=\frac{{\mathit{X}}_{max}-{\mathit{X}}_{min}}{{\mathit{Z}}_{max}-{\mathit{Z}}_{min}}\end{array}\right.$$

(10)

where ${\mathit{X}}_{max}$, ${\mathit{X}}_{min}$, ${\mathit{Y}}_{max}$, ${\mathit{Y}}_{min}$, ${\mathit{Z}}_{max},\mathrm{and}{\mathit{Z}}_{min}$ are the maximum and minimum values of three axes.

3.2. Sensor Fusion

Several methods are available for representing attitudes in space, such as Euler angle, direction cosine, and quaternion. The direction cosine method can be used to solve matrix transformations, but it requires extensive calculations. The Euler angle method is easy to understand, but it may cause the gimbal lock problem. The quaternion method is more efficient and requires fewer calculations compared to the other two methods. Since the update of the quaternion depends on the previous state, the acceleration and geomagnetism data in the static state is used to calculate the initial quaternion attitude. The initial pitch angle ${\theta }_0$ and roll angle ${\gamma }_0$ can be calculated using the following equation:

$$\left\{\begin{array}{c}{\theta }_0=arcsin\left(a_y^n\right)\\ {\gamma }_0=-arctan\left(\frac{a_x^n}{a_z^n}\right)\end{array}\right.$$

(11)

Specifically, ${\left[\begin{array}{ccc}{a}_x^n& a_y^n& a_z^n\end{array}\right]}^T$ is the projection of gravitational acceleration from reference frame to carrier frame. Then, the initial yaw angle can be calculated with ${\theta }_0$, ${\gamma }_0$, and the projection of magnetic field data ${\left[\begin{array}{ccc}{m}_x^n& m_y^n& m_z^n\end{array}\right]}^T$ as follows:

$$\left\{\begin{array}{c}{m}_y^n=m_x^{b}cos{\gamma }_0+m_y^{b}sin\gamma sin{\theta }_0+m_z^{b}sin\gamma cos{\theta }_0\\ m_x^n=m_y^{b}cos{\theta }_0-m_z^{b}sin{\theta }_0\\ {\phi }_0= arctan\left(\frac{m_y^n}{m_x^n}\right)\end{array}\right.$$

(12)

When ${\left[\begin{array}{ccc}{\theta }_0& {\gamma }_0& {\phi }_0\end{array}\right]}^T$ is confirmed, the initial quaternion can be calculated using the following equations:

$$\left\{\begin{array}{c}i\text{\_}{q}_0=\mathit{cos}\left(\frac{{\gamma }_0}{2}\right)\mathit{cos}\left(\frac{{\theta }_0}{2}\right)\mathit{cos}\left(\frac{{\phi }_0}{2}\right)-\mathit{sin}\left(\frac{{\gamma }_0}{2}\right)\mathit{sin}\left(\frac{{\theta }_0}{2}\right)\mathit{sin}\left(\frac{{\phi }_0}{2}\right)\\ i\text{\_}{q}_1=\mathit{cos}\left(\frac{{\gamma }_0}{2}\right)\mathit{sin}\left(\frac{{\theta }_0}{2}\right)\mathit{cos}\left(\frac{{\phi }_0}{2}\right)-\mathit{sin}\left(\frac{{\gamma }_0}{2}\right)\mathit{cos}\left(\frac{{\theta }_0}{2}\right)\mathit{sin}\left(\frac{{\phi }_0}{2}\right)\\ i\text{\_}{q}_2=\mathit{sin}\left(\frac{{\gamma }_0}{2}\right)\mathit{cos}\left(\frac{{\theta }_0}{2}\right)\mathit{cos}\left(\frac{{\phi }_0}{2}\right)+\mathit{cos}\left(\frac{{\gamma }_0}{2}\right)\mathit{sin}\left(\frac{{\theta }_0}{2}\right)\mathit{sin}\left(\frac{{\phi }_0}{2}\right)\\ i\text{\_}{q}_3=\mathit{cos}\left(\frac{{\gamma }_0}{2}\right)\mathit{cos}\left(\frac{{\theta }_0}{2}\right)\mathit{sin}\left(\frac{{\phi }_0}{2}\right)+\mathit{sin}\left(\frac{{\gamma }_0}{2}\right)\mathit{sin}\left(\frac{{\theta }_0}{2}\right)\mathit{cos}\left(\frac{{\phi }_0}{2}\right)\end{array}\right.$$

(13)

As proven by the calculation process of the initial Euler angle, the yaw, pitch, and roll angles can be calculated directly from the acceleration, angular velocity, and magnetic field data in the static state. However, when the sensor is in motion, extra linear acceleration is introduced, resulting in inaccurate calculation of the sensor’s attitude. In addition, the integral drift still exists when angular velocity is used to calculate the attitude after sensor calibration. In this paper, a complementary filter algorithm is applied to solve the multi-sensor fusion problem [28,29]. The essence of multi-sensor fusion is to use the proportional–integral (PI) method to calculate the error between the estimated and measured values of acceleration and geomagnetism, and apply this error to correct the updated quaternion. The sensor fusion process is as shown in Figure 7.

${\left[\begin{array}{ccc}{\mathit{e}}_{ax}& {\mathit{e}}_{ay}& {\mathit{e}}_{az}\end{array}\right]}^T$ is defined as the error caused by the acceleration data in carrier coordinates; it can be expressed as follows:

$$\left[\begin{array}{c}{\mathit{e}}_{ax}\\ {\mathit{e}}_{ay}\\ {\mathit{e}}_{az}\end{array}\right]=\mathit{a}\times \mathit{v}=\left|\begin{array}{ccc}\mathit{i}& \mathit{j}& \mathit{k}\\ {\mathit{a}}_x& {\mathit{a}}_y& {\mathit{a}}_z\\ {\mathit{v}}_x& {\mathit{v}}_y& {\mathit{v}}_z\end{array}\right|=\left[\begin{array}{c}{\mathit{a}}_y·{\mathit{v}}_z-{\mathit{a}}_z·{\mathit{v}}_y\\ {\mathit{a}}_z·{\mathit{v}}_x-{\mathit{a}}_x·{\mathit{v}}_z\\ {\mathit{a}}_x·{\mathit{v}}_y-{\mathit{a}}_z·{\mathit{v}}_x\end{array}\right]$$

(14)

where ${\left[\begin{array}{ccc}{\mathit{v}}_x& {\mathit{v}}_y& {\mathit{v}}_z\end{array}\right]}^T$ is the estimated acceleration in carrier coordinates.

${\left[\begin{array}{ccc}{\mathit{e}}_{mx}& {\mathit{e}}_{my}& {\mathit{e}}_{mz}\end{array}\right]}^T$ is defined as the error caused by the geomagnetic field data in carrier coordinates; it can be expressed as follows:

$$\left[\begin{array}{c}{\mathit{e}}_{mx}\\ {\mathit{e}}_{my}\\ {\mathit{e}}_{mz}\end{array}\right]=\mathit{m}\times \mathit{w}=\left|\begin{array}{ccc}\mathit{i}& \mathit{j}& \mathit{k}\\ {\mathit{m}}_x& {\mathit{m}}_y& {\mathit{m}}_z\\ {\mathit{w}}_x& {\mathit{w}}_y& {\mathit{w}}_z\end{array}\right|=\left[\begin{array}{c}{\mathit{m}}_y·{\mathit{n}}_z-{\mathit{m}}_z·{\mathit{n}}_y\\ {\mathit{m}}_z·{\mathit{n}}_x-{\mathit{m}}_x·{\mathit{n}}_z\\ {\mathit{m}}_x·{\mathit{n}}_y-{\mathit{m}}_z·{\mathit{n}}_x\end{array}\right]$$

(15)

where ${\left[\begin{array}{ccc}{\mathit{n}}_x& {\mathit{n}}_y& {\mathit{n}}_z\end{array}\right]}^T$ is the estimated geomagnetic field data in carrier coordinates.

Then, the total error can be defined as:

$$\left[\begin{array}{c}{\mathit{e}}_x\\ {\mathit{e}}_y\\ {\mathit{e}}_z\end{array}\right]=\left[\begin{array}{c}{\mathit{e}}_{ax}\\ {\mathit{e}}_{ay}\\ {\mathit{e}}_{az}\end{array}\right]+\left[\begin{array}{c}{\mathit{e}}_{mx}\\ {\mathit{e}}_{my}\\ {\mathit{e}}_{mz}\end{array}\right]=\left[\begin{array}{c}\left({\mathit{a}}_y·{\mathit{v}}_z-{\mathit{a}}_z·{\mathit{v}}_y\right)+\left({\mathit{m}}_y·{\mathit{n}}_z-{\mathit{m}}_z·{\mathit{n}}_y\right)\\ \left({\mathit{a}}_z·{\mathit{v}}_x-{\mathit{a}}_x·{\mathit{v}}_z\right)+\left({\mathit{m}}_z·{\mathit{n}}_x-{\mathit{m}}_x·{\mathit{n}}_z\right)\\ \left({\mathit{a}}_x·{\mathit{v}}_y-{\mathit{a}}_z·{\mathit{v}}_x\right)+\left({\mathit{m}}_x·{\mathit{n}}_y-{\mathit{m}}_z·{\mathit{n}}_x\right)\end{array}\right]$$

(16)

The angular velocity can be fixed using the following equations:

$$\left\{\begin{array}{c}\left[\begin{array}{c}{\mathit{e}}_x\mathit{i}\\ {\mathit{e}}_y\mathit{i}\\ e_{z}i\end{array}\right]=\left[\begin{array}{c}{\mathit{e}}_x\mathit{i}\\ {\mathit{e}}_y\mathit{i}\\ {\mathit{e}}_z\mathit{i}\end{array}\right]+K_i·\left[\begin{array}{c}{\mathit{e}}_x\\ {\mathit{e}}_y\\ {\mathit{e}}_z\end{array}\right]\\ \left[\begin{array}{c}{\mathit{\omega }}_x^{\prime }\\ {\mathit{\omega }}_y^{\prime }\\ {\mathit{\omega }}_z^{\prime }\end{array}\right]=\left[\begin{array}{c}{\mathit{\omega }}_x\\ {\mathit{\omega }}_y\\ {\mathit{\omega }}_z\end{array}\right]+K_p·\left[\begin{array}{c}{\mathit{e}}_x\\ {\mathit{e}}_y\\ {\mathit{e}}_z\end{array}\right]+\left[\begin{array}{c}{\mathit{e}}_{x}i\\ {\mathit{e}}_{y}i\\ {\mathit{e}}_{z}i\end{array}\right]\end{array}\right.$$

(17)

where ${\left[\begin{array}{ccc}{\mathit{\omega }}_x& {\mathit{\omega }}_y& {\mathit{\omega }}_z\end{array}\right]}^T$ is the measured angular velocity from a gyroscope, and ${\left[{\mathit{\omega }}_x^{\prime } {\mathit{\omega }}_y^{\prime } {\mathit{\omega }}_z^{\prime }\right]}^T$ is the fixed angular velocity to update the quaternion matrix. The updated quaternion attitude ${\left[\begin{array}{ccc}{\mathit{q}}_0^{\prime }& {\mathit{q}}_1^{\prime } {\mathit{q}}_2^{\prime }& {\mathit{q}}_3^{\prime }\end{array}\right]}^T$ can be represented as follows:

$$\left[\begin{array}{c}{\mathit{q}}_0^{\prime }\\ {\mathit{q}}_1^{\prime }\\ {\mathit{q}}_2^{\prime }\\ {\mathit{q}}_3^{\prime }\end{array}\right]=\left[\begin{array}{c}{\mathit{q}}_0\\ {\mathit{q}}_1\\ {\mathit{q}}_2\\ {\mathit{q}}_3\end{array}\right]+\left[\begin{array}{cccc}0& -{\mathit{q}}_1·{\mathit{\omega }}_x^{\prime }& -{\mathit{q}}_2·{\mathit{\omega }}_y^{\prime }& -{\mathit{q}}_3·{\mathit{\omega }}_z^{\prime }\\ {\mathit{q}}_0·{\mathit{\omega }}_x^{\prime }& 0& {\mathit{q}}_2·{\mathit{\omega }}_z^{\prime }& -{\mathit{q}}_3·{\mathit{\omega }}_y^{\prime }\\ {\mathit{q}}_0·{\mathit{\omega }}_y^{\prime }& -{\mathit{q}}_1·{\mathit{\omega }}_z^{\prime }& 0& {\mathit{q}}_{31}·{\mathit{\omega }}_x^{\prime }\\ {\mathit{q}}_0·{\mathit{\omega }}_z^{\prime }& {\mathit{q}}_1·{\mathit{\omega }}_y^{\prime }& -{\mathit{q}}_2·{\mathit{\omega }}_x^{\prime }& 0\end{array}\right]·\frac{T}{2}$$

(18)

Finally, the Euler angle can be obtained using the following equations:

$$\left\{\begin{array}{c}\theta =\arcsin \left(2\left(q_2^{\prime }{q}_3^{\prime }+q_0^{\prime }{q}_1^{\prime }\right)\right)\\ \gamma =-\arctan \left(\frac{2\left(q_1^{\prime }{q}_3^{\prime }-q_0^{\prime }{q}_2^{\prime }\right)}{q_0^{\prime 2}-q_1^{\prime 2}-q_2^{\prime 2}+q_3^{\prime 2}}\right)\\ \phi =\arctan \left(\frac{2\left(q_1^{\prime }{q}_2^{\prime }-q_0^{\prime }{q}_3^{\prime }\right)}{q_0^{\prime 2}-q_1^{\prime 2}+q_2^{\prime 2}+q_3^{\prime 2}}\right)\end{array}\right.$$

(19)

where $\theta , \gamma ,\mathrm{and}\phi$ are the pitch, roll, and yaw angles of a single IMMU sensor.

4. Experimental Results of Sensor Calibration and Fusion

Since all the IMMU sensors on the data glove are of the same model (MPU-9250), one of these sensors is used to demonstrate the experimental results of sensor calibration and fusion.

The comparisons of angular velocity data before and after calibration are shown in Figure 8. It can be seen that the zero bias of the gyroscope is reduced. However, the integral drift of the gyroscope still fails to be eliminated after a long period of time, which needs to be corrected by fusing the acceleration and geomagnetism data.

The comparisons of normalized acceleration data before and after calibration are shown in Figure 9. The theoretical value of normalized gravitational acceleration in the static state should be equal to 1. The accuracy of local gravitational acceleration is improved after calibration.

The geomagnetic field data should lie on the sphere equidistant from the center. The 3D scatter plots of the geomagnetic field data are shown in Figure 10. The comparisons of the geomagnetic field data on X–Y, X–Z, and Y–Z planes are shown in Figure 11. It can be seen that the data are distributed in a nearly elliptical shape before calibration, and in a nearly standard spherical shape after calibration.

As discussed earlier, the attitude angles of the inertial sensor can be calculated using a complementary filter algorithm. The attitude angles of the IMMU sensor in the static state are shown in Figure 12. The static accuracy of attitude angles is smaller than 1°, which satisfies the application requirement of our data glove system.

The commercial HCM365B E-compass is used to evaluate the dynamic performance of the IMMU sensor, as shown in Figure 13a. The output attitude angle curves of HCM365B and the IMMU sensor are very close, while the roll angle changes from 0° to 200°.

5. Human–Computer Interaction Using Data Glove

Real-time human–computer interaction is also a required function for patients while doing rehabilitation exercises at home. A series of virtual scenes needs to be established by the human–computer interaction module, including a virtual hand, movable virtual items, lights, and a background. After the virtual scenes are built, all the 12 IMMU sensors will be mapped to the joints of the virtual hand by converting sensor attitude angles into finger rotation angles. In this paper, a virtual hand is established in the Unity platform, as shown in Figure 14.

The human interaction module includes the following three parts: (1) Virtual scene design: The virtual hand should be established according to the skeletal structure of the human hand. Some virtual items are also provided for practicing hand grasping motions. (2) Sensor–joints mapping: 12 IMMU sensors need to be mapped to the corresponding joints. (3) Scripts: The scripts include finding joint objects in the hierarchical view, mapping sensors to model joints, receiving attitude angles, and displaying animations. The bending angles of the joints of the virtual hand are limited to a range that fits in with daily human hand motions. A schematic diagram showing the bending angles of different joints is provided in Figure 15.

To validate the joint angle obtained by the data glove, the digital goniometer is used to measure the angle, as shown in Figure 16. One problem of using such a steel goniometer is that the magnetometer inside the IMMU sensor may be disturbed by nearby ferromagnetic materials. Therefore, the digital goniometer should be placed on the finger after the angle data have been exported from the data glove.

The comparison of measurement results is represented in

Table 1. Measurement results of goniometer and data glove.

Angle from goniometer	112.10°	91.20°	80.52°	69.65°
Average angle from data glove	110.28°	89.55°	79.66°	68.30°
Error rate	1.6%	1.8%	1.1%	1.9%

. The angles measured by goniometer and data glove are recorded when the index finger is held in four different positions. The average error rate is less than 2% and the maximum deviation is nearly 1.4°. The accuracy of the data glove will be enough for most wearable applications.

Hand rehabilitation assessment, by the total active range of motion (TAM) method, for example, requires the bending angle data of the joints, which can be calculated with the attitude angles of sensors on the data glove. Since the DIP joint has no sensor placed on it, its bending angle is empirically set to 1/3 that of the PIP joint. The relationship between the bending angles of joints and the attitude angles of sensors is represented in

Table 2. Calculation of bending angles of joints using sensor attitudes.

Joint	ID	Bending Angle
Thumb IP joint	${\mathsf{\theta }}_{\mathrm{IP}\_\mathrm{Thumb}}$	${\mathrm{S}}_1-{\mathrm{S}}_2$
Thumb MP joint	${\mathsf{\theta }}_{\mathrm{MP}\_\mathrm{Thumb}}$	${\mathrm{S}}_3-{\mathrm{S}}_8$
Index finger PIP joint	${\mathsf{\theta }}_{\mathrm{PIP}\_\mathrm{Index}}$	${\mathrm{S}}_4-{\mathrm{S}}_5$
Index finger MP joint	${\mathsf{\theta }}_{\mathrm{MP}\_\mathrm{Index}}$	${\mathrm{S}}_5-{\mathrm{S}}_8$
Middle finger PIP joint	${\mathsf{\theta }}_{\mathrm{PIP}\_\mathrm{Middle}}$	${\mathrm{S}}_6-{\mathrm{S}}_7$
Middle finger MP joint	${\mathsf{\theta }}_{\mathrm{MP}\_\mathrm{Middle}}$	${\mathrm{S}}_7-{\mathrm{S}}_8$
Ring finger PIP joint	${\mathsf{\theta }}_{\mathrm{PIP}\_\mathrm{Ring}}$	${\mathrm{S}}_9-{\mathrm{S}}_{10}$
Ring finger MP joint	${\mathsf{\theta }}_{\mathrm{MP}\_\mathrm{Ring}}$	${\mathrm{S}}_{10}-{\mathrm{S}}_8$
Little finger PIP joint	${\mathsf{\theta }}_{\mathrm{PIP}\_\mathrm{Little}}$	${\mathrm{S}}_{11}-{\mathrm{S}}_{12}$
Little finger MP joint	${\mathsf{\theta }}_{\mathrm{MP}\_\mathrm{Little}}$	${\mathrm{S}}_{12}-{\mathrm{S}}_8$

.

Specifically, ${\mathbf{S}}_i\left(i=1,2,\cdots ,12\right)$ represents the attitude angles of IMMU sensors around the x-axis. The experimental results of three hand gestures showing the numbers ”2”, “5”, and “10” are provided in Figure 17. The hand gestures are captured and reconstructed with the virtual hand in Unity with high accuracy. The gesture data can be saved and exported by physicians for off-line diagnosis, as shown in

Table 3. Bending angles of joints with three gestures.

Gesture	Index Finger		Middle Finger		Ring Finger		Little Finger
	${\mathsf{\theta }}_{\mathit{PIPJ}}$	${\mathsf{\theta }}_{\mathit{MPJ}}$	${\mathsf{\theta }}_{\mathit{PIPJ}}$	${\mathsf{\theta }}_{\mathit{MPJ}}$	${\mathsf{\theta }}_{\mathit{PIPJ}}$	${\mathsf{\theta }}_{\mathit{MPJ}}$	${\mathsf{\theta }}_{\mathit{PIPJ}}$	${\mathsf{\theta }}_{\mathit{MPJ}}$
“2”	9.2°	7.9°	2.4°	24.8°	109.2°	49.4°	64.4°	90.7°
“5”	5.5°	2.2°	16.4°	3.9°	9.5°	10.5°	24.3°	20.6°
“10”	110.3°	70.2°	108.2°	67.5°	148.6°	62.9°	68.9°	77.3°

.

The dynamic ball grasping experiment is demonstrated in Figure 18. The virtual ball and hand should be set in such original positions that their distance is the same as that between the physical ball and hand. The subject needs to pick up the ball and flip the hand several times. The changes in hand posture can be reviewed from the data of Sensor 8 on the back of the hand, as shown in Figure 19. The recovery status can be estimated from the recorded data, for example, the time taken for an action, the accuracy of an action, and the degree of hand tremor.

Four different data glove systems are compared with our proposed IMMU data glove, as listed in

Table 4. Comparison of proposed data glove and other data glove systems.

Publications	Type of Sensor	Number of Sensors	Deviation of Joint Angle
Cha et al. [30]	flexible piezoelecric sensor	19 (one hand)	5°
da Silva et al. [21]	fiber bragg gratings sensor	1 (each finger)	2°
Li et al. [31]	optical linear encoder	3 (each finger)	1°
Kortier et al. [32]	IMMU	3 (each finger)	1.1°
Proposed glove system	IMMU	12 (one hand)	1.4°

. It can be found that both the optical fiber sensor-based glove system and IMMU sensor-based data glove system can provide a good performance while measuring the joint angle. However, the cost of IMMU sensors is much less than optical fiber sensors for massive commercial wearable applications.

6. Conclusions

This paper presents a novel data glove system to assess hand function during hand rehabilitation. 12 nine-axis inertial sensors are integrated in a data glove to obtain the angles of hand joints. The error models of the accelerometer, gyroscope, and magnetometer are analyzed. The least square principle is used to solve the correction parameters of the accelerometer. The validity of the calibration process is verified by comparing raw sensor data and calibrated data. A quaternion-based complementary filtering algorithm is applied to fuse the acceleration, angular velocity and geomagnetism data. A commercial E-compass is also used to verify the stability and dynamic performance of our fusion algorithm.

Real-time and high-precision human–computer interaction is realized with the data glove and Unity. A virtual hand model, a ball model, and some virtual scenes are established in Unity, and then the attitude angles of 12 sensors are transformed into the rotation angles of hand joints. The experimental results show that the glove system can effectively measure hand postures and joint angles in motion, and it is expected to be used to develop a variety of rehabilitation scenarios to get patients more engaged in the long-term rehabilitation process.