3D Hand Motion Generation for VR Interactions Using a Haptic Data Glove

Abstract

Recently, VR-based training applications have become popular and promising, as they can simulate real-world situations in a safe, repeatable, and cost-effective way. For immersive simulations, various input devices have been designed and proposed to increase the effectiveness of training. In this study, we developed a novel device that generates 3D hand motion data and provides haptic force feedback for VR interactions. The proposed device can track 3D hand positions using a combination of the global position estimation of ultrasonic sensors and the hand pose estimation of inertial sensors in real time. For haptic feedback, shape–memory alloy (SMA) actuators were designed to provide kinesthetic forces and an efficient power control without an overheat problem. Our device improves upon the shortcomings of existing commercial devices in tracking and haptic capabilities such that it can track global 3D positions and estimate hand poses in a VR space without using an external suit or tracker. For better flexibility in handling and feeling physical objects compared to exoskeleton-based devices, we introduced an SMA-based actuator to control haptic forces. Overall, our device was designed and implemented as a lighter and less bulky glove which provides comparable accuracy and performance in generating 3D hand motion data for a VR training application (i.e., the use of a fire extinguisher), as demonstrated in the experimental results.

1. Introduction

In virtual reality (VR), the introduction of an artificial environment that substitutes physical surroundings convincingly is a crucial element of providing a realistic experience to users. Recently, VR-based training has become popular and promising as it can simulate real-world situations in a safe, repeatable, and cost-effective way. With interactive and immersive simulations, various input devices have been designed and proposed to increase the effectiveness of training [1,2]. A comprehensive review of VR training in various application domains is discussed in [3].

Among several VR elements (i.e., visual feedback, free movement, physical interaction, narrative engagement, and others) that increase the immersion of the user experience, the use of physical tools has gained substantial attention in virtual training [4]. For example, using data gloves, a user can interact with virtual objects in a tangible way, allowing precise hand movements. Using haptic feedback, these hand motions feel like real-life ones and make the user fully engaged with the VR environment.

Studies on hand motion tracking or haptic feedback have been actively conducted to improve the effectiveness of user interaction in VR training. A virtual training system using haptic tools allows users to know the feeling, size, and weight of the tools [5]. For example, in a training model for learning the use of a fire extinguisher, a hand-held VR controller was modified by adding tactile sensations like air pressure and vibration [6]. The perceptions of this type of training were further evaluated using a VR application with a physical object [7]. Arora et al. proposed modular tools that facilitate physical operation in a VR environment [8]. Zhu et al. developed haptic tools that are comparable to real ones and demonstrated the effectiveness of tangible interactions [9]. Shigeyama et al. introduced a portable VR controller that can change the mass properties on a two-dimensional plane and provide a realistic tool feel [10]. Similarly, Zenner and Krüger proposed a device that changes the drag and rotational inertia for sensing the surface area [11]. Baek et al. proposed a car-washing system in VR using multiple sensors [12]. In the medical field, Talhan and Jeon conducted a review on the various haptic effects of pneumatic actuation [13].

From design to deployment, VR training applications are developed for specific learning purposes. Thus, previous systems modified hand-held VR controllers or built special tools for user interactions during training. Data gloves are used to track hand motions and provide user interactions with virtual objects in a tangible way [14,15,16,17,18,19]. However, these commercial devices rely on vibrotactile sensation, which only provides the textural feeling of a real object, or a mechanical system which consists of a bulky exoskeleton and motors. Moreover, they require a special suit or an external tracker to track the global positions of hands in a VR space. In contrast to the development of tracking methods with visual feedback, techniques for 3D hand motion generation with haptic force feedback for flexible and precise interactions need further improvement.

In this study, we developed a novel device that generates 3D hand motion data and provides haptic force feedback for VR interactions. The proposed device can track 3D hand positions using a combination of the global position estimation of ultrasonic sensors and the hand pose estimation of inertial sensors in real time. Multichannel ultrasonic sensors in a 2D plane localize hand positions at a relatively low speed without a drift problem, whereas a set of inertial sensors accurately estimates hand positions at a relatively high speed. For haptic feedback, the shape–memory alloy (SMA) actuators were designed to provide kinesthetic forces and efficient power control without an overheating problem. The proposed device improves upon the shortcomings of existing commercial devices by designing and implementing the tracking and haptic capabilities in an integrated device.

The proposed device makes several contributions to the interactions in VR training. First, we have designed and implemented a flexible haptic glove that can track 3D global positions and estimate hand poses in a VR space without using an external suit or tracker. Unlike existing devices, we integrated two different types of sensors (ultrasonic and inertial) into one system for hand position and pose estimation, which overcomes the magnetic disturbance and occlusion problems. Also, the proposed device provides the kinematic feeling of hand interactions, such as gripping and releasing a real object (i.e., the use of a fire extinguisher in our experiment), through haptic force feedback. For better flexibility in handling and feeling physical objects compared to exoskeleton-based devices, we introduced a SMA-based actuator to control haptic forces. Overall, the proposed device was designed and implemented as a lighter and less bulky glove while providing comparable accuracy and performance in generating 3D hand motion data for a VR training application.

This paper is organized as follows: In Section 2, we review previous research on the hand motion tracking and haptic feedback of data gloves. Section 3 describes the design and implementation of hand motion tracking and haptic force feedback in the proposed device. We present the experimental results in Section 4 and our conclusions for the proposed device and its limitations in Section 5.

2. Related Works

2.1. Hand Motion Tracking

The hand motion tracking performance of data gloves is a key factor in providing a realistic experience in a VR environment. The techniques for flexible and strain sensor-based recognition estimate the entire hand motion by using the resistance variation of the sensors based on the degree of bend in the finger joints [17,20,21]. However, it is difficult to track the accurate positions of the finger movements. Currently, commercial data gloves use a set of inertial measurement units (IMUs) attached to the target fingers to track the motion of individual joints in real time [14,15,16,18,19]. O’Flynn et al. proposed an IMU-based glove that minimizes the calibration process and provides data analytics to track joint movement [22]. In these devices, the tracking performance depends on the number of IMUs attached to the glove, which increases the complexity and cost of the device. A simplified version with fewer IMUs is available as an off-the-shelf device [16]; however, it is difficult to recognize accurate hand poses for training purposes. Moreover, these devices rely on an external tracker such as the VIVE tracker [23] to track the global positions of hands in a 3D space.

Head-mounted displays (HMDs) for VR, such as the HTC VIVE and Meta Quest series, apply a vision-based and constellation tracking technique to locate the 3D positions of their hand-held controllers [24,25]. However, particular attention must be paid to the configuration of the base stations when tracking global positions in a VR environment because body parts and physical objects often suffer from occlusion problems [26], for example, if the tracking performance is constrained by the viewing range and affected by an in-between obstacle. Moreover, hand recognition fails if the hand disappears from the cameras mounted on an HMD.

2.2. Haptic Feedback

Haptic feedback is another key factor in providing users with an immersive experience in VR training. In the experience of VR, this is particularly emphasized in handling the physical proxy for a virtual object [27]. MANUS Prime Glove provides vibrotactile sensation using flexible sensors and voice coil actuators [17]. However, it only provides the textural feeling of a real object. Teslasuit Glove and CyberGlove are haptic devices that adopt a motor-driven technique to provide finger force feedback [18,19]. These devices adopt rotation sensors to capture hand motion; however, they constrain the degrees of freedom (DOF) in finger movements due to the bulky mechanism mounted on the dorsal hand and the fingers. Hinchet et al. proposed a wearable haptic feedback device that uses electrostatic brakes to generate torque forces [28]. However, this device generates a strong magnetic field and is not suitable to use with the tracking device of IMU sensors. Bouzit et al. developed hand–palm-based force feedback using pneumatic actuators [29]. In their device, an air compressor or a continuous supply of CO₂ cans with a regulator is required to generate actuation force feedback, which is bulky and impractical in VR training. Fujimoto et al. introduced a force feedback method using shape–memory alloy (SMA) wires [30]. However, this type of device adopts natural cooling; hence, an additional cooling method should be considered for frequent use. Moreover, a user can sense the kinesthetic effect even when the actuator is not activated, owing to the frictional force of the SMA wires mounted on the fingers. Patterson et al. applied the SMA wires to locomote a soft robot and used water to boost the cooling rate [31]. However, the application of water to an external device like a glove is difficult owing to leakage and maintenance. For these reasons, most of the previously studied methods of kinesthetic haptic feedback have been difficult to use practically because they are large and bulky to wear and carry in a VR environment.

3. Design and Implementations

3.1. Hand Position Estimation

The proposed device tracks hand positions in 3D space using a combination of ultrasound-based position estimation, inertial sensor-based 3D acceleration, and pose estimation. However, there are well-known problems for these sensors: ultrasound-based devices have high computational costs, IMU sensors are sensitive to drifting issues, and it is difficult to estimate 3D positions continuously without noise. Considering these, we combined two different types of sensors into the proposed device, which consisted of four ultrasonic source generators, two ultrasonic receiver arrays, two 3D ultrasonic source direction estimators, an ultrasonic sensor-based 3D position estimator, a 9-axis IMU sensor, an attitude and heading reference system (AHRS) processor, and a 3D position estimator, as shown in Figure 1.

Initially, the ultrasonic generators attached to the back of the glove generated ultrasonic waves of 50 kHz with a radiation pattern of ±30°. To minimize the shadow area of ultrasonic reception, four ultrasonic transducers were attached to the generators, such that the two multichannel ultrasonic receiver arrays could receive signals from all directions without restricting the range of the hand rotations. The ultrasonic receiver arrays obtained ultrasonic signals in the time domain at a sampling rate of 2 MHz and sent them to the 3D direction estimators, which estimated the direction between the ultrasonic source generators and receiver arrays using the delay-and-sum beamforming (DSBF) algorithm. Using the source localization method [32], we estimated the 3D position of the ultrasonic sources by calculating the closest intersection point of the two ultrasonic direction vectors obtained from those two estimators, as shown in Figure 2.

The inertial sensor attached to the ultrasonic transducers provided 9-axis data: 3-axis acceleration, 3-axis gyro, and 3-axis geomagnetism. This data included gravitational acceleration, which should be removed to measure the accurate acceleration of hand rotations, $a_R$, from the inertial sensor, as follows:

$$a_R=\mathbf{R}(a_s-g),$$

(1)

where $a_s$ is the acceleration of the inertial sensor, $g$ is the gravitational force, and $\mathbf{R}$ is the rotation matrix of the inertial sensor, which was estimated using the AHRS algorithm [33]. Thus, the AHRS processor in the proposed device filters out the gravitational component from the acceleration data measured by the inertial sensor on the glove.

The inertial sensor-based 3D position was estimated by integrating Equation (1) twice; however, the noise errors from the drift phenomenon should be corrected for the position estimation. Assuming a linear system used for a state-space model in the Kalman filter, the state prediction model can be defined as follows:

$${\overline{\mathbf{x}}}_t=\mathbf{A}{\mathbf{x}}_{t-1}+\mathbf{B}{a}_t,$$

(2)

where $t$ is the time index, $a_t$ is $a_R$ at $t$, ${\overline{\mathbf{x}}}_t$ is the predicted state vector, ${\mathbf{x}}_t$ is the state vector, $\mathbf{A}=\left[\begin{array}{cc}1& dt\\ 0& 1\end{array}\right]$, and $\mathbf{B}=\left[\begin{array}{c}\frac{d{t}^2}{2}\\ dt\end{array}\right]$. During the correction step, ${\overline{\mathbf{x}}}_t$ and the Kalman gain matrix, ${\mathbf{K}}_t$, are used to update ${\mathbf{x}}_t$ as follows:

$$\begin{array}{c}{\mathbf{x}}_t={\overline{\mathbf{x}}}_t+{\mathbf{K}}_t\left({\mathbf{z}}_t-{\mathbf{H}\overline{\mathbf{x}}}_t\right),\\ {\mathbf{K}}_t={\overline{\mathbf{P}}}_t{\mathbf{H}}^{⊺}{(\mathbf{H}{\overline{\mathbf{P}}}_t{\mathbf{H}}^{⊺}+\mathbf{R})}^{-1},\\ {\mathbf{P}}_t=\left(\mathbf{I}-{\mathbf{K}}_t\mathbf{H}\right){\overline{\mathbf{P}}}_t,\\ {\overline{\mathbf{P}}}_t=\mathbf{A}{\mathbf{P}}_{t-1}{\mathbf{A}}^{⊺}+\mathbf{Q},\end{array}$$

(3)

where ${\mathbf{z}}_t$ is 3D position and velocity estimated by the ultrasonic estimator, $\mathbf{H}$ is the estimation matrix, ${\mathbf{P}}_t$ is the posteriori covariance matrix, ${\overline{\mathbf{P}}}_t$ is the predicted covariance matrix, $\mathbf{Q}$ is the covariance matrix for the IMU sensors, and $\mathbf{R}$ is the covariance matrix for the ultrasonic estimators. We defined the initial state variables at $t=0$ and assigned ${\overline{\mathbf{x}}}_t$ and ${\mathbf{P}}_t$ as the input arguments to estimate the prediction states at $t+1$. In the proposed system, ${\sigma }^2$ in $\mathbf{Q}$ and $\mathbf{R}$ was set to 0.01 and 1.0, respectively [34].

The 3D hand positions were roughly estimated at a relatively low speed (10 Hz) using the DSBF algorithm from the ultrasonic receiver arrays in a 2D plane and accurately estimated at a high-speed (187.5 Hz) using the data from the AHRS processor. It is noteworthy that the inertial sensor-based AHRS processor and the ultrasound-based localization module operated independently during the position estimation process. Finally, the 3D position estimator integrated the 3D acceleration and rotation data estimated by the inertial sensor and the 3D position and velocity data estimated by the ultrasonic receiver arrays into the output data using the Kalman filter.

3.2. Finger Rotation Estimation

To obtain the rotation data of finger joints, a set of 12 IMU sensors were placed on the glove, as shown in Figure 3. We omitted five sensors on the proximal interphalangeal (PIP) joints to reduce the complexity of the device. The rotations of the proximal interphalangeal (PIP) joints, ${\mathit{q}}_{\mathrm{P}\mathrm{I}\mathrm{P}}$, were estimated by adopting linear interpolation (LERP) between the distal interphalangeal (DIP) and the metacarpophalangeal (MCP) joints as follows:

$$\begin{array}{c}{\mathbf{q}}_{\mathrm{P}\mathrm{I}\mathrm{P}}=\frac{\mathrm{L}\mathrm{E}\mathrm{R}\mathrm{P}\left({\mathbf{q}}_{\mathrm{M}\mathrm{C}\mathrm{P}}, {\mathbf{q}}_{\mathrm{D}\mathrm{I}\mathrm{P}}, \alpha \right)}{\left|\mathrm{L}\mathrm{E}\mathrm{R}\mathrm{P}\left({\mathbf{q}}_{\mathrm{M}\mathrm{C}\mathrm{P}}, {\mathbf{q}}_{\mathrm{D}\mathrm{I}\mathrm{P}}, \alpha \right)\right|},\\ \mathrm{L}\mathrm{E}\mathrm{R}\mathrm{P}\left({\mathbf{q}}_{\mathrm{M}\mathrm{C}\mathrm{P}},{\mathbf{q}}_{\mathrm{D}\mathrm{I}\mathrm{P}},\mathsf{\alpha }\right)=\left\{\begin{array}{c}{\mathbf{q}}_{\mathrm{M}\mathrm{C}\mathrm{P}}\left(1-\mathsf{\alpha }\right)+{\mathbf{q}}_{\mathrm{D}\mathrm{I}\mathrm{P}}\alpha , \mathrm{if} {\mathbf{q}}_{\mathrm{M}\mathrm{C}\mathrm{P}}{\cdot \mathbf{q}}_{\mathrm{D}\mathrm{I}\mathrm{P}}\ge 0\\ {\mathbf{q}}_{\mathrm{M}\mathrm{C}\mathrm{P}}\left(1-\mathsf{\alpha }\right)-{\mathbf{q}}_{\mathrm{D}\mathrm{I}\mathrm{P}}\alpha , \mathrm{if} {\mathbf{q}}_{\mathrm{M}\mathrm{C}\mathrm{P}}{\cdot \mathbf{q}}_{\mathrm{D}\mathrm{I}\mathrm{P}}<0\end{array}\right.,\end{array}$$

(4)

where $\alpha$ is the normalized position on the straight line between ${\mathit{q}}_{\mathrm{M}\mathrm{C}\mathrm{P}}$ and ${\mathit{q}}_{\mathrm{D}\mathrm{I}\mathrm{P}}$ in the quaternion space. We set $\alpha$ to 0.4–0.6 based on the anatomical model of hand motion [35,36].

3.3. Haptic Force Feedback

In the proposed device, we introduced haptic force feedback using SMA wires that are lightweight to wear and compatible with the hand motion estimation described in the previous section. Unlike motor-based kinesthetic feedback, a major challenge in SMA-based devices is the low cooling rate of the heated SMA when kinesthetic senses are transmitted to users. Generally, seconds to several tens of seconds are required to cool down the SMA-based devices, depending on the activation temperature, diameter, and surrounding conditions. We adopted a flexible thermoelectric device [37] and thermal grease with a high thermal conductivity to resolve the cooling problem.

The limited motion direction is another problem with SMA-based devices. Because the shape is activated in only one direction when it is heated, the generation of negative force is not possible in kinesthetic feedback. Thus, the proposed device applied SMA actuators to the hand dorsal and palm to provide bidirectional feedback. Figure 4 shows an overview of the haptic force feedback used in the VR interactions.

3.3.1. SMA Actuator

We deformed the SMA wire of Dynalloy 70 °C Flexinol^® with a diameter of 375 µm and created an actuator spring, as shown in Figure 5. To memorize the initial shape of the straight SMA wire in the form of a spring, it was placed in a furnace at 475 °C for one hour. The tensile force was measured using a weight scale and the SMA spring attached to the wooden hand of a mannequin to obtain a mathematical model of the pull forces from the supplied powers, as shown in Figure 6. The actuation of a stretched spring started at approximately 0.3 W. When a power of 3 W was applied, the pulling force measured was up to approximately 1 kg. When more than 3 W was applied, it was overheated and caused a permanent strain; therefore, it could not be actuated to the initial shape of a spring. To provide a precise force feedback, we have estimated a polynomial fit from the measured data as follows:

$$F={(6.3\times {10}^{-9})w}^3+{(-6.2\times {10}^6)w}^2+0.0027w+0.0814,$$

(5)

where $F$ is the pulling force and $w$ is the electric power.

3.3.2. SMA Cooling

As noted earlier in Section II, the slow cooling rate is a critical problem for the use of SMA materials in haptic devices that generate force feedback frequently. In our experiment, several seconds were required for the SMA spring to cool down its actuating temperature in the static air. To expedite this cooling rate, we applied a flexible thermoelectric pad that was developed by adopting the Peltier effect, as shown in Figure 7. Because the spring and thermoelectric pad are electrically conductive, the spring was wrapped with a silicone tube to avoid shorting. To effectively transfer the cool temperature of the thermoelectric pad to the spring, a hardened thermal grease with a thermal conductivity of 8.5 w/mk was applied between the pad covered with the insulating film and the copper foam, and a permanently uncured silicone grease with a thermal conductivity of 4.8 w/mk was applied around the spring in the tube. Moreover, several holes were drilled into the silicone tube to increase its thermal conductivity. When an electric current flows through a thermoelectric device, one side becomes cool while the other side becomes hot. If the hot side is not cooled quickly, heat is applied to the cooled side, thereby heating both sides of the device. To solve this heating problem, heat sinks were attached to the hot side. Finally, the cooling device was closed with a silicone cap to prevent the injected grease from leaking.

4. Experimental Results

The proposed device is best understood through examples of its uses, as described in the subsequent sections. The accompanying video can be found at Supplementary Materials.

4.1. Hand Motion Generation

Figure 8 shows the prototype of the data glove for 3D hand motion generation. The proposed device was implemented in two main parts: the ultrasonic source position estimator and the hand pose estimator using a set of inertial sensors. We used Hagisonic HG-(M/L)40(T/R)B for the ultrasonic transducers [38], Knowles SPU0410LR5H-1 for ultrasonic receivers [39], and Linear Technology AD7386BCPZ-RL for the analog-to-digital (AD) converter [40], which was sampled at a rate of 2 MHz. We adopted our previous system for the DSBF analyzer and the communication component [32].

The hand pose estimation part consisted of 12 IMU sensors, micro-processing units (MPUs), an ultrasonic source generator, and a sensor data acquisition and communication module. We used InvenSense ICM-20948 for the IMU sensors [41], which acquired 9-axis data at 187.5 Hz, and STMicroelectronics STM32L432 for the MPUs [42], which processed the AHRS algorithm via serial peripheral interface (SPI) communication. An STMicroelectronics STM32F767 processor acquired 12 quaternions from small MPUs for the finger rotations and transmitted data to the PC via a Silicon Labs WGM110 Wi-Fi communication module [43].

Figure 9 compares the performance of tracking hand positions between the proposed device and the VIVE tracker [23]. Both devices were tracked approximately 30 m in an indoor room with dimensions of 4 m × 6 m × 2.5 m (width × length × height). The total distance offset of the proposed device compared to the tracker was less than 5 cm without a drift. Furthermore,

Table 1. Comparison of finger rotations: Average of angular differences (in degrees) was compared for each hand pose between the proposed device and commercial glove [17] using the marker-based motion capture system [44].

Hand Pose
Finger		MANUS	Ours	MANUS	Ours	MANUS	Ours	MANUS	Ours
Thumb	MCP	0.74	1.63	1.85	1.64	1.18	0.79	1.02	0.67
	PIP	1.02	0.56	0.70	0.34	0.92	2.37	1.63	0.63
	DIP	0.93	0.71	1.59	1.09	1.27	0.80	1.45	0.93
Index	MCP	3.09	1.32	3.29	1.24	3.08	0.16	3.50	1.00
	PIP	1.13	3.68	1.77	0.82	1.23	2.64	1.78	1.05
	DIP	3.31	0.09	1.28	1.18	4.68	5.65	3.57	1.75
Middle	MCP	4.42	0.90	2.09	1.16	3.74	0.31	2.28	0.20
	PIP	3.57	2.25	2.63	3.42	3.54	0.12	2.29	2.15
	DIP	1.82	1.63	1.75	0.19	1.75	1.89	3.62	0.74
Ring	MCP	3.58	0.57	2.39	1.79	3.52	0.42	4.21	1.61
	PIP	2.63	3.15	3.93	1.59	3.22	1.21	3.44	3.79
	DIP	2.95	0.92	3.09	1.96	4.76	3.46	3.61	3.05
Little	MCP	2.28	1.79	1.72	0.16	2.32	0.85	2.08	0.84
	PIP	3.59	3.36	1.56	1.10	1.43	1.82	4.78	1.54
	DIP	4.54	1.71	3.81	0.31	4.76	0.26	1.40	6.97
Mean	MCP	2.82	1.24	2.27	1.20	2.77	0.51	2.62	0.86
	PIP	2.39	2.60	2.12	1.46	2.07	1.63	2.78	1.83
	DIP	2.71	1.01	2.30	0.95	3.44	2.41	2.73	2.69

compares the finger rotations in various hand poses between the proposed device and the commercial glove [17] using the marker-based motion capture system [44]. In this comparison, a total of 1000 joint rotations were collected at 120 Hz from the hand poses. As shown in the table, the angular differences of the MCP, PIP, and DIP joints for the proposed device were less than the ones for the commercial one (i.e., 0.51–1.24° against 2.27–2.82°, 1.46–2.60° against 2.07–2.78°, and 0.95–2.69° against 2.3–3.44°). An interpolation estimation was used in the proposed device for the PIP joints. For the third and fourth poses in Table 1, the commercial device showed larger differences as this device relies on flexible sensors to estimate finger rotations, which is not as precise as the proposed device using multiple inertial sensors.

4.2. Haptic Feedback with Cooling Rate

To obtain the desired amount of force feedback and a constant cooling temperature from the thermoelectric pads, constant power was controlled by a temperature sensor and supplied to each device. As shown in Figure 10, the SMA springs and thermoelectric pads operated at the desired current and voltage using an 8-bit digital-to-analog (DA) converter (Texas Instruments DAC5311) [45], a low-voltage operational amplifier (OP-Amp) as a comparator (Texas Instruments LMV321) [45], and MOSFETs (Infineon Technologies IPP026N10NF2S) [46].

The desired amount of current, $I_D$, which traverses the SMA springs, MOSFET1, and a shunt resistor $R_{Shunt}$ was controlled by adjusting the voltage of the negative input of the OP-Amp. The voltage for $I_D$ was estimated as follows:

$$V_{Shunt}=I_D\times R_{Shunt},$$

(6)

where $R_{Shunt}$ is 200 mΩ. The digital input value to the DAC to supply $V_{Shunt}$ to the negative input of the OP-Amp was estimated as follows:

$$\mathrm{D}\mathrm{A}\mathrm{C}=\frac{V_{Shunt}}{V_{In}}\times {(2}^n-1),$$

(7)

where the voltage input of the system $V_{In}$ was 3.3 V, and $n$ is the bit resolution of the DAC. Here, MOSFET1 was used to supply stable power to the SMA springs, whereas R2, R3, and MOSFET2 were used to turn on and off the power of the module [46].

Figure 11 shows the prototype of the proposed device and its controller, which had five channels (two for the SMA springs on the index finger, two for those on the middle finger, and one for the thermoelectric pad). All the power control modules received current values through SPI communication from the STMicroelectronics STM32F070 microprocessor [42].

Figure 12 compares the cooling rate between the static cooling and proposed method. The static air cooled down below the actuating temperature (70 °C) for approximately 4 s, whereas the proposed method decreased the temperature within 1 s. It is noteworthy that the proposed device maintained a temperature of 15 °C at a low power of 3 W.

Table 2. Physical comparison between the proposed device and commercial gloves [14,15,16,17,18,19].

	VRgluv [14]	BeBop [15]	Home Hi5 [16]	MANUS [17]	Teslasuit [18]	CyberGlove [19]	Ours
Force feedback	Yes (motor)	No	No	No	Yes (motor)	Yes (motor)	Yes (SMA)
Weight 1 (g)	450	200	105	134	450	340	200
Size 1 (length × width × height mm)	170 × 100 × 50	170 × 90 × 8	170 × 90 × 5	170 × 90 × 5	200 × 100 × 60	195 × 125 × 70	170 × 90 × 20

¹ Approximately estimated with sensors attached.

summarizes the physical comparison between the proposed device and the commercial gloves [14,15,16,17,18,19]. It is noteworthy that the proposed device is lighter and smaller in height compared to the commercial ones with force feedback [14,18,19], which are more bulky owing to a motor-driven structure.

4.3. User Test

We applied the proposed device to a VR training application on the use of a fire extinguisher, as shown in Figure 13. In this training, a user extinguished a fire on several spots in a VR environment using a real fire extinguisher that was used and emptied. During the exercise, various kinesthetic feedback were provided from the handle and nozzle of the extinguisher. For example, the parameters of the reaction force were estimated from the collision characteristics between the glove and handle or between the glove and nozzle, and then they were transmitted to the haptic force feedback controller via Wi-Fi. The controller applied current to the SMA springs to generate a force corresponding to the parameter input. When a user squeezes or releases the handle, the controller increases or reduces the current flowing through the SMA springs, as shown in Figure 4. The controller generated the force feedback similarly for the nozzle vibration (i.e., a vibration caused by the spraying pressure) and reaction.

For a user test, 12 general people (6 males, 6 females, and ages between 25 and 35), who did not experience the application using the haptic glove, participated in the training. Prior to the training, they were instructed on how to use a real fire extinguisher in the outside environment and repeated the same instructions in the virtual environment. Each participant conducted the virtual exercise for several sessions, which lasted approximately 30 min. Figure 13 shows a survey result of evaluating the force feedback and VR immersion against the real extinguisher on the radar chart. In this survey, an average of a five-point Likert scale was used, where a larger number indicates higher similarity. Overall, the participants felt the force feedback from the handle (i.e., squeeze and release) was more similar to the real extinguisher’s than the one from the nozzle (i.e., vibration and reaction). This is mainly because excessive force feedback was generated for the nozzle reactions compared to the real extinguisher. In addition, the participants felt the squeezing force was more similar to the real extinguisher than the releasing force. Our device recognized the user’s hand poses and controlled the SMA actuators precisely to generate the right amount of squeezing force; however, a residual force acting on the springs made it uncomfortable for the releasing force. Both groups felt the overall training was highly immersive.

5. Conclusions

In this study, we introduced a novel device that can generate hand motion data and provide haptic force feedback for VR interactions. The proposed device used a combination of ultrasonic and inertial sensors and tracked 3D hand positions and poses with high accuracy without using an external tracker. By applying efficient power control to the actuators, the proposed device provided kinesthetic feedback without an overheating problem. Compared to existing commercial gloves, the overall design of the proposed device is lighter and less bulky, so it is suitable for use in various VR training applications, as shown in the user test for the virtual training of handling a fire extinguisher. Synchronizing with the visual feedback of HMDs, this immersive experience of haptic feedback will be helpful in responding to fires ranging from small scales in houses and apartments to large scales in urban infrastructure facilities (i.e., underground utility tunnels and factories) in a safe, repeatable, and cost-effective way. For other applications, our device can be applied to a training situation that requires the kinematic feeling of hand interactions using a tool in medical, manufacturing, and sporting simulations.

There are several ongoing improvements to the proposed device. The current design of the implemented device consists of various sensors and control modules attached to the back of a hand, which are not resistant to collision and shock, particularly in intensive training situations. Using SMA-based actuators for haptic force feedback, a user can feel a residual force acting on the unactuated spring even if the springs are completely cooled down. This can interfere with the user’s natural finger movement and the force feedback of the SMA being actuated. Currently, we are looking for a substituent material that completely removes the force when the wire is bent in a non-driven state.