Teleoperation Control of an Underactuated Bionic Hand: Comparison between Wearable and Vision-Tracking-Based Methods

Abstract

Bionic hands have been employed in a wide range of applications, including prosthetics, robotic grasping, and human–robot interaction. However, considering the underactuated and nonlinear characteristics, as well as the mechanical structure’s backlash, achieving natural and intuitive teleoperation control of an underactuated bionic hand remains a critical issue. In this paper, the teleoperation control of an underactuated bionic hand using wearable and vision-tracking system-based methods is investigated. Firstly, the nonlinear behaviour of the bionic hand is observed and the kinematics model is formulated. Then, the wearable-glove-based and the vision-tracking-based teleoperation control frameworks are implemented, respectively. Furthermore, experiments are conducted to demonstrate the feasibility and performance of these two methods in terms of accuracy in both static and dynamic scenarios. Finally, a user study and demonstration experiments are conducted to verify the performance of these two approaches in grasp tasks. Both developed systems proved to be exploitable in both powered and precise grasp tasks using the underactuated bionic hand, with a success rate of $98.6\%$ and $96.5\%$, respectively. The glove-based method turned out to be more accurate and better performing than the vision-based one, but also less comfortable, requiring greater effort by the user. By further incorporating a robot manipulator, the system can be utilised to perform grasp, delivery, or handover tasks in daily, risky, and infectious scenarios.

1. Introduction

Currently, bionic hands have been widely applied in many fields [1], such as grasping tasks [2,3], industrial applications [4], human–robot-interaction [5], and conducting delicate operations in dangerous situations [6]. As multi-degree-of-freedom (DoF) end-effectors, bionic hands have excellent flexibility and versatility. Meanwhile, the anthropomorphic structure empowered bionic hands to conduct some complicated, dangerous, and inaccessible tasks in a human-inspired manner. Hence, bionic hands have gained widespread attention in a variety of practical applications.

Bionic hands can be divided into fully actuated and underactuated ones, according to the mechanical structure [7]. Fully actuated bionic hands benefit from high dexterity, which allows them to independently control each degree of freedom for more complex operations [8]. The redundant DoFs allow the robot to deal with optimisation and hierarchical control problems. However, full actuation also yields bulkier and more complex mechatronic systems and control algorithms. The underactuated ones, instead, benefit from a lightweight and portable design, as well as a simple driver structure, which is easier to build and deploy [9]. Furthermore, underactuation brings a better compliance of the mechanical structure. However, due to the underactuated and nonlinear motion characteristics, developing an accurate mathematical model is difficult [10]. Learning-based methods also have been investigated to program the motion of a bionic hand [11] and robotics [12]. However, for executing tasks in dynamic or unstructured environments, the pre-defined bionic hand motion sometimes is not suitable or robust enough. Although underactuated bionic hands have numerous advantages, how to control them in a more natural and precise manner remains a challenging issue [13].

Humans are capable of dealing with complicated tasks, even in unstructured or dynamic environments. By incorporating human intelligence, teleoperation control is considered a promising solution to control robots [14,15]. For example, a novel data glove was designed and employed to achieve the teleoperation control of a robotic hand–arm system, which was stable, compact, and portable [16]. Specifically, in the teleoperation control task of an underactuated bionic hand, by sensing human hand posture and then sending it to the bionic hand, a real-time mapping from the human to the bionic hand can be realised [17]. First of all, human hand motion should be detected in real-time, and many solutions, such as gloves, exoskeletons, bending sensors, vision-tracking-based methods, etc., have been investigated [18,19,20].

Among the human-hand-tracking methods, wearable mechanical sensor systems (fabric gloves, exoskeletons) and vision-tracking-based methods (Kinect camera, Leap Motion, Vicon tracker system, etc.) are the most used ones. In wearable solutions, flexion and potentiometer sensor are the most frequently utilised to detect the bending angle of human hand joints and then map it to a bionic hand. These methods outperform vision-tracking-based methods in terms of stability and robustness and, especially, do not suffer from line-of-sight occlusion problems [21]. A typical application of the wearable-based teleoperation control framework was implemented by NASA, which utilised two CyberGloves to teleoperate the robot to perform maintenance tasks such as threading nuts into bolts and tying knots [22]. Apart from this, a wearable-glove-based method was also introduced into virtual interaction scenarios for 3D modelling and task training [23]. Although wearable sensors have such advantages, the cumbersome wearing process, size adaptability, and pretty poor comfortableness still hinder their further application. Meanwhile, the unergonomic structural design affects the intuitiveness and transparency during the operation.

Compared with the glove- or exoskeleton-based methods, vision-tracking-based methods are much more natural and can be personalised by measuring the joint position of the human hand. Vision-based tracking systems are primarily adopted to extract important human hand features and send the command to the robotic hand after the motion mapping. In [24], Artal proposed a real-time gesture recognition technique based on a Leap Motion camera to control a 5-DoF bionic hand. Similarly, in Madina’s study [25], Leap Motion was used as an input device for hand rehabilitation training in a virtual training case. In addition, some camera tracker and infrared tracker systems have also been used to realise hand posture recognition [26,27]. Nonetheless, the tracking performance of the vision-tracking-based methods greatly depends on environmental factors, such as uneven illumination, cluttered background, and items with a similar colour to the hand. In addition, the vision-tracking-based methods have limited workspace and usually suffer from the self-occlusion issue. Although the characteristics of the different control methods are obvious, there is still no systematic study of the performance of both in practical applications as a basis for how to select an appropriate method.

In this paper, the main motivation aims to achieve intuitive and natural teleoperation control of an underactuated bionic hand. To achieve this, two control frameworks based on a wearable glove and a vision-tracking system are developed, respectively. The main contributions are summarised as follows: (1) The calibration results with an external ground truth tracking system are employed to formulate the inverse kinematics model of the underactuated bionic hand, which takes the underactuated and nonlinear characteristics of the bionic hand into consideration. (2) Two novel features to describe the human hand fingers’ motion are defined, namely the flexion angle and the bending angle. Such quantities are measured by the two tracking systems, respectively, normalised for their own range and mapped to the bionic hand motion to provide an intuitive teleoperation control. (3) Furthermore, a comparison of the proposed two methods in terms of accuracy in both static and dynamic environments is performed. User study and practical grasp tasks are designed to demonstrate the effectiveness of the two methods and to further compare their performance in terms of success rate and subjective evaluation.

The remainder of this paper is organised as follows. Section 2 describes the research background and the calibration procedures of the underactuated bionic hand kinematics. Following that, Section 3 presents the wearable glove and vision-tracking-based methods to detect human hand motion in real-time. Then, the details of the teleoperation control frameworks are described in Section 4, as well as the algorithm to apply the inverse kinematic model. Section 5 depicts the experimental setup and metrics to demonstrate the performance of the proposed method. After that, the experimental results are given in Section 6. Finally, Section 7 gives the conclusions of this work.

2. Bionic Hand and Calibration

In this section, the research background and the details of the underactuated bionic hand are described first. Afterward, the inverse kinematics model of the underactuated bionic hand is derived from the results of a proper calibration method.

2.1. Research Background

Figure 1a,b illustrate the structure of the human hand and the underactuated bionic hand, respectively. Each finger of the underactuated bionic hand is independently driven by one Hiwonder LFD-01 servo motor, which is controlled by a Pulse-Width Modulation (PWM) signal, which is a periodic square wave with a fixed period and a variable width of the switch-on sub-period (pulse width). The higher the pulse width is, the higher the current value sent to the motor is, which in turn encodes the rotation of the servo motor axis. The pulse width of the signal delivered to the LFD-01 ranges from 500 $\mathsf{\mu }$s to 2500 $\mathsf{\mu }$s, corresponding to 0° and 180° rotations of the motor, respectively. Furthermore, each finger is composed of three primary links, resembling the three human finger phalanxes, and two additional transmission links, which are used to implement the underactuation paradigm.

As shown in Figure 1b, the motion state of the i-th ( i = thumb, index, middle, ring, pinkie denotes the sequence of the underactuated bionic hand fingers) bionic hand finger can be described by a bending angle ${\theta }_i$, which is defined by the vector going from the MCP joint to the fingertip with respect to the maximum extension pose.

2.2. Bionic Hand Calibration

The inverse kinematics model of the ith finger aims at determining the mathematical relationship between the input value of the corresponding motor, in terms of PWM signal width, and the resulting finger bending angle, ${\theta }_i$. In work [28], a direct linear function was used to define the inverse kinematics of the finger, and the sensor fusion strategy was investigated to control the underactuated bionic hand. However, the aforementioned method is insufficiently accurate because it overlooks some critical issues: (1) the nonlinear feature caused by motor saturation; (2) the mechanical backlash of the bionic hand contributes to the nonlinear properties of the underactuated bionic hand; (3) the nonlinear features vary depending on whether the motion is flexion or extension.

In this paper, we employed the calibration data to further fit the relationship between the motor input value and the bending angle of the finger, instead of using a simple linear mapping solution. The detailed experimental setup for collecting data is shown in Figure 2. An external optical tracking system (Optotrak Certus Motion Capture System, NDI, Canada), with an accuracy of up to 0.1 mm and a resolution of 0.01 mm, was adopted to measure the real bending angle of the bionic hand finger. The collected data were employed as the ground truth when calibrating the bionic hand.

A set of tracking markers (active near-infrared LED) was employed to mark the landmarks for angle reconstruction. As shown in Figure 2b–d, the tracking markers were stuck on the side surface of the bionic hand finger joints, and the 3D position of these markers, with respect to the NDI position sensor, were collected. However, due to the mechanical occlusion, the MCP joint position was difficult to directly measure. Hence, an auxiliary pointing probe was employed. As depicted in Figure 2d, two tracking markers, labelled as A and B, were attached to the probe, and their positions ${\mathit{P}}_{\mathit{A}}$ and ${\mathit{P}}_{\mathit{B}}$ were measured while the probe pointed at the MCP joint. Then, by knowing the distance $d_A$ from A to the probe tip, the joint position ${\mathit{P}}_{\mathit{MCP}}$ can be calculated as follows:

$${\mathit{P}}_{\mathit{M}\mathit{C}\mathit{P}}={\mathit{P}}_{\mathit{A}}+\frac{{\mathit{P}}_{\mathit{B}}-{\mathit{P}}_{\mathit{A}}}{\left|{\mathit{P}}_{\mathit{B}}-{\mathit{P}}_{\mathit{A}}\right|}·d_A$$

(1)

By knowing the MCP joint position, ${\mathit{P}}_{\mathit{MCP}}$, and the fingertip position corresponding to the kth recording, ${\mathit{P}}_{\mathit{TIP},\mathit{k}}$, the bending vector ${\mathit{b}}_k$ from the MCP joint to the fingertip can be computed. Then, given the bending vector corresponding to the maximum extension pose of the finger, ${\mathit{b}}_0$, the bending angle ${\theta }_k$ can be expressed as:

$${\mathit{b}}_k={\mathit{P}}_{\mathit{TIP},\mathit{k}}-{\mathit{P}}_{\mathit{MCP}}$$

(2)

$${\theta }_k=acos\left(\frac{{\mathit{b}}_k·{\mathit{b}}_0}{\left|{\mathit{b}}_k\right|\left|{\mathit{b}}_0\right|}\right)$$

(3)

For each finger, four complete calibration cycles of the bending angle were obtained while altering the motor input throughout a set of predetermined values homogeneously distributed across the motor input range. The resulting samples illustrated in Figure 3 show that the nonlinear characteristics change during the extension and flexion processes. As a result, two mathematical functions for each finger that fit the sample results should be defined to achieve accurate kinematics modelling. In this paper, the neural-network-based fitting method [29] was adopted to retrieve such functions.

3. Human Hand Motion Tracking

When performing teleoperation control of the bionic hand, human hand motion is first detected in real-time, then mapped and sent to the bionic hand. Therefore, both glove-based and vision-based approaches for tracking human hand motion are presented.

3.1. Wearable-Glove-Based Motion Tracking

In the wearable-glove-based human hand motion tracking method, an exoskeleton glove was utilised to collect the motion of the human operator’s fingers. The detailed system framework is shown in Figure 4. The glove is endowed with five potentiometers, each one connected to the first phalanx of one finger, which goes from the MCP to the PIP joint, as shown in Figure 1a. Before starting the tracking, each user performs a calibration procedure to associate, for each finger, two specific potentiometer rotation values to the maximum and the minimum flexion reference poses, respectively. In such a procedure, the user has simply to flex and extend the fingers through the whole ROM allowed by the glove structure. After the calibration, the potentiometer detects the flexion movement of the phalanx with respect to the hand palm, producing a dimensionless readout S ranging from 500 to 2500, corresponding to the maximum flexion angle, ${\phi }_{max}$, and to the minimum one, ${\phi }_{min}$, allowed by the glove, respectively, as shown in Figure 5a,b. By considering ${\phi }_{min}$ equal to zero, the signal readout S of these potentiometers has the following relationship with the flexion angle $\phi$:

$$S=\left(1-\frac{\phi }{{\phi }_{max}}\right)·2000+500$$

(4)

The five obtained readouts of the potentiometers are then sent to a computer, which runs Ubuntu 16.04 and the Robot Operating System (ROS) Kinetic via a USB cable. A Rosserial interface package handles the serial communication between the glove and the ROS network. For each finger, the feature extraction node samples the signal readout S at 40 Hz and computes the so-called “measured flexion level”, $F{L}_m$, which is a dimensionless quantity, ranging from zero to one, which can concisely represent the human finger motion and is easily mappable to a robot finger motion, as described in Section 4. $F{L}_m$ is calculated as follows:

$$F{L}_m=1-\frac{S-500}{2000}=\frac{\phi }{{\phi }_{max}}$$

(5)

In this paper, to smooth the value of the flexion level to improve the robustness and steadiness during the teleoperation control of the underactuated bionic hand, a real-time linear Kalman Filter [30] was utilised. The Kalman filter has been widely utilised in many wearable robotic systems, including lower limb wearable robotic devices [31] and 3D joint angle determination [32]. In this paper, the Kalman filter acts as a recursive algorithm based on a state space representation of the finger motion. At each iteration, it predicts the values of the state variables and of the next observation of the flexion level; then, when a new observation is available, the state is updated by considering the prediction error on the observation and the features of the noises that affect the state and the observations. To design the state equations, the linearised motion model of the flexion level was considered:

$$\left\{\begin{array}{c}F{L}_k=F{L}_{k-1}+T_S{\dot{FL}}_{k-1}+\frac{T_S^2}{2}{\ddot{FL}}_{k-1}\\ {\dot{FL}}_k={\dot{FL}}_{k-1}+T_s{\ddot{FL}}_{k-1}\end{array}\right.$$

(6)

where $F{L}_k$ and $F{L}_{k-1}$ are the flexion level at time k and time $k-1$, respectively, ${\dot{FL}}_k$ and ${\dot{FL}}_{k-1}$ are the first-degree time derivative of the flexion level at time k and time $k-1$, respectively, ${\ddot{FL}}_{k-1}$ is the second-degree time derivative of the flexion level at time $k-1$, and $T_s$ is the sampling period. The following state equation for the Kalman filter can be derived from the motion model:

$${\mathit{x}}_k=\mathsf{\Phi }{\mathit{x}}_{k-1}+\mathsf{\Gamma }{w}_{k-1}$$

(7)

where $w_{k-1}$ is the value at time $k-1$ of a White Gaussian Noise (WGN) applied to the state to approximate ${\ddot{FL}}_{k-1}$, ${\mathit{x}}_k$ and ${\mathit{x}}_{k-1}$ are the vectorial state variable at time k and $k-1$, respectively, and $\mathsf{\Phi }$ and $\mathsf{\Gamma }$ are the constant parameter matrices of the equations. The vectorial state $\mathit{x}$ and the parameter matrices $\mathsf{\Phi }$ and $\mathsf{\Gamma }$ are defined as follows:

$$\mathit{x}=\left[\begin{array}{c}{x}_1\hfill \\ x_2\hfill \end{array}\right]=\left[\begin{array}{c}FL\hfill \\ \dot{FL}\hfill \end{array}\right],\mathsf{\Phi }=\left[\begin{array}{cc}1& T_s\\ 0& 1\end{array}\right],\mathsf{\Gamma }=\left[\begin{array}{c}\frac{T_s^2}{2}\\ T_s\end{array}\right]$$

(8)

Furthermore, the relationship between the state variables and the input observation, which is the measured flexion level, can be defined as follows:

$$F{L}_{m,k}=x_{1,k}+v_k$$

(9)

where $F{L}_{m,k}$, $x_{1,k}$, and $v_k$ are the measured flexion level, the flexion level state variable, and the WGN noise acting on the observation, at time k, respectively. The update estimate of the flexion level state variable stands for the filtered measured flexion level, $F{L}_m^{*}$, and is published as the final output of the tracking system.

3.2. Vision-Tracking-Based Motion Tracking

In the vision-based method, a Leap Motion Controller (LMC) (Ultraleap, CA, USA) was used to collect the motion of the human operator’s fingers. The detailed system framework is shown in Figure 6. The LMC retrieves two infrared stereo images of the bare hand, which are then sent via USB to a computer that runs the Leap Motion Service (LMS), Ubuntu 16.04, and ROS Kinetic. After removing background objects and environmental light, the LMS reconstructs a 3D representation of the hand surface by applying the principles of stereo vision, and next, it extracts the 3D coordinates of the human hand MCP joints and fingertips with respect to an absolute reference system $\left\{LMC\right\}$, as illustrated in Figure 5c. Moreover, it provides the transformation parameters describing a local reference frame $\left\{HH\right\}$ attached to the hand.

The coordinate transformation node belonging to the ROS network samples LMS data at 40 Hz and transforms MCP and fingertip coordinates to the hand reference frame. Then, the feature extraction node computes a concise finger motion feature such that it is influenced only by MCP flexion, like the robot bending angle, and on the contrary, it is minimally influenced by MCP abduction. To achieve this, the node computes the bending angle $\beta$ defined by the projection of the MCP-to-fingertip vector on a plane that is fixed with respect to the hand frame, as shown in Figure 5d. Such a plane is defined in a calibration phase as the one where the fingertip, PIP joint, and MCP joint lie when the finger is at minimum bending. $\beta$ is defined as the angle between the aforementioned projection vector and another vector belonging to the plane that represents the minimum bending pose ${\beta }_{min}$. Similar to the flexion angle defined in Section 3.1, the bending angle $\beta$ is normalised for its maximum ${\beta }_{max}$ to obtain a dimensionless variable called the measured bending level $B{L}_m$:

$$B{L}_m=\frac{\beta }{{\beta }_{max}}$$

(10)

It should be mentioned that the minimum and maximum bending poses are measured during the calibration phase. A user interface node lets the user decide when to start the calibration procedure through the keyboard. Additionally, the same real-time linear Kalman filter described in Section 3.1 was utilised to smooth the value of the bending level to improve robustness and steadiness during the teleoperation control of the bionic hand.

4. Teleoperation Control Framework

In this section, it is shown how the teleoperation control methods are implemented using the wearable-glove-based and the vision-based tracking methods, respectively. As depicted in Figure 7, the actuation controller stage is the same for both teleoperation systems, while the distinctive stage is the motion tracking one.

As described in Section 3.1 and Section 3.2, each tracking system extracts for each finger one dimensionless human motion feature, ranging from 0 to 1, which represents the finger motion, namely the flexion level $F{L}_m^{*}$ and the bending level $B{L}_m^{*}$, respectively. A prior step for teleoperation is mapping such a human feature to the robot feature, namely the robot bending angle $\theta$. As the correspondence between human and robot gestures must be intuitively understandable and applicable by the user without any mental effort or long-time learning needed, the resulting movement of the robot must be highly semantically correlated to the human one, while considering the kinematic differences between human and robot structures. Benefiting from the conciseness and meaningfulness of the motion features extracted by the two tracking systems and from the underactuated mechanics of the bionic hand, a simple and intuitive mapping can be performed. The output of the utilised tracking system, whether $F{L}_m^{*}$ in the case of the wearable system or $B{L}_m^{*}$ in the case of the vision system, is multiplied for the bending angle range ${\theta }_{max}$ measured during the bionic hand calibration described in Section 2.2.

$$\theta =\left\{\begin{array}{c}F{L}_m^{*}·{\theta }_{max}\left(\mathrm{glove}\right)\hfill \\ B{L}_m^{*}·{\theta }_{max}\left(\mathrm{vision}\right)\hfill \end{array}\right.$$

(11)

Once the bending angles are computed, an ROS node applies the inverse kinematics model synthesised in Section 2.2. To avoid a sudden change from one function to the other, and thus instability, the average value of the bending angle in the last 40 frames ${\overline{\theta }}_{40}$ (moving average on 1 second) was used to improve robustness. Only if the angle $\theta$ deviates from ${\overline{\theta }}_{40}$ for more than $1\%$ of the bending angle Range Of Motion (ROM), the direction changes are detected. Then, according to the current value of the direction (extension or flexion), the corresponding inverse kinematic function ($I{K}_E$ and $I{K}_F$, respectively) is applied to $\theta$ in order to obtain the motor control signal value $PWM$. The details of the implemented inverse kinematics algorithm are summarised in Algorithm 1.

Then, the communication handler node packs the control signals in a message characterised by a proper TCP/IP format and sends it to the bionic hand through an Ethernet link. Control messages are unpacked by the bionic hand processor, and the motors are driven consequently. A user interface manages the opening and closing of the communication link according to the user’s will. The user’s input is delivered through a keyboard and a mouse.

Algorithm 1: Inverse kinematics modelling of the underactuated bionic hand.

5. Experimental Setup

In this section, three experiments are designed to estimate the ROM of the human finger angle to which the two tracking systems are respectively sensitive, evaluate the accuracy, and compare the usability of the two implemented teleoperation systems. Firstly, the system setup employed for the first and second experiments is described. Then, the protocol and evaluation metrics of the three experiments are explained.

5.1. System Setup

The first and the second experiments were designed to estimate the ROM of the human finger angle to which the two tracking systems are sensitive and to assess the accuracy of the two systems, respectively. This paper is focused on the results for the index finger motion. During the data acquisition part of the experiments, a human operator controlled the bionic hand through the developed systems. Meanwhile, two sets of optical markers were used to measure the resulting bending angle $\theta$ of the bionic hand (see Section 2.2 for the definition) and the ground truth of the human finger angle used to control the system, namely the real human flexion angle, ${\phi }_{real}$ (see Section 3.1 for the definition of flexion angle used in this work), for the wearable system and the real human bending angle, and ${\beta }_{real}$ (see Section 3.2 for the definition of human finger bending angle), for the vision system.

As illustrated in Figure 8a, one marker was located on the side of the bionic hand index fingertip and three markers were placed on the side of the hand rigid case to define a 3D local reference frame. The MCP joint position was measured employing a pointing probe as the one shown in Figure 2d, using the same method described in Section 2.2 for the bionic hand calibration. Positioning possibilities concerning the human hand were limited by the wide space covered by the glove. In this case, five markers were used, as shown in Figure 8b: one on the fingertip side, one on the PIP joint side, one on the MCP joint side, one on the second Metacarpal Bone (SM) in correspondence with the styloid process, and on the Radial Styloid Process (RAD). The MCP, SM, and RAD were used to define a local reference frame. The MCP and PIP were used to measure the flexion angle, while the MCP and TIP where used to compute the bending angle. In both cases, a method similar to the one used by the vision-based method to retrieve the bending angle, described in Section 3.2, was implemented. Figure 8c,d show how the markers were integrated when using the glove-based and the LMC-based methods, respectively. The NDI optical tracking system, already used in Section 2.2, was employed to measure the 3D coordinates of the markers.

5.2. Protocol and Performance Metrics

5.2.1. Human Hand Range of Motion Experiment

The human angle measured by the ground truth tracking system, whether ${\phi }_{real}$ or ${\beta }_{real}$, is not directly comparable to the bionic hand bending angle $\theta$, given the different ROM. A prior step to assess the accuracy of the teleoperation systems was the estimation of the human ROM that characterises the two human-hand-tracking systems, in order to normalise the measured angles, obtaining the dimensionless variables real human flexion level, $F{L}_h$, and real human bending level, $B{L}_h$, for the glove-based and vision-based systems, respectively. Such a variable can be compared to the robotic bending level $B{L}_r$, that is $\theta$ normalised for its own range, that was measured during the bionic hand calibration in Section 2.2. Given the difficulty of precisely recording the poses that are associated with the human fingers limited by the tracking systems, an estimation of the human range was instead needed for each system.

To estimate the human ROM, a static calibration procedure was performed on each tracking system. Several ground truth human angles, homogeneously distributed in the ROM, were measured through the markers, and at the same time, the human flexion level $F{L}_m^{*}$ and the human bending level $B{L}_m^{*}$ measured and filtered by the tracking systems were recorded. Moreover, the robotic bending level resulting from the teleoperation mapping was also measured, to be used later in the accuracy experiment.

Four complete calibration cycles were obtained for each tracking system. The linear function that best fits the cycles (excluding saturation samples) was computed through least-squares linear regression. Such a function is assumed to be the ideal characteristic of the tracking system. This approximation is the most optimistic one, given that it minimises the distance of the samples from the ideal behaviour. The ${\phi }_{real}$ and ${\beta }_{real}$ values that, according to the ideal characteristic, should correspond to the minimum and maximum $F{L}_m^{*}$ and $B{L}_m^{*}$, were considered as the human range limits.

5.2.2. Accuracy Performance Comparison Experiment

To evaluate and compare the accuracy performance of the two implemented teleoperation control frameworks, three kinds of error were investigated during the second teleoperation experiment, in both static and dynamic scenarios, namely the ones listed and computed as follows and shown in Figure 9:

• Overall teleoperation control accuracy error:

  $$\left\{\begin{array}{c}\hfill {\epsilon }_{tele,glove}=F{L}_h-B{L}_r\\ \hfill {\epsilon }_{tele,vision}=B{L}_h-B{L}_r\end{array}\right.$$

  (12)
• Human hand motion tracking accuracy error:

  $$\left\{\begin{array}{c}\hfill {\epsilon }_{track,glove}=F{L}_h-F{L}_m^{*}\\ \hfill {\epsilon }_{track,vision}=B{L}_h-B{L}_m^{*}\end{array}\right.$$

  (13)
• Bionic hand actuation control accuracy error:

  $$\left\{\begin{array}{c}\hfill {\epsilon }_{act,glove}=F{L}_m^{*}-B{L}_r\\ \hfill {\epsilon }_{act,vision}=B{L}_m^{*}-B{L}_r\end{array}\right.$$

  (14)

The static accuracy assessment was performed on the data acquired for the human ROM estimation experiment described in Section 5.2.1. Furthermore, both human and robotic finger angles were normalised for the respective ROM to obtain the comparable dimensionless features. For each kind of error, the couple of variables determining the error were compared assessing their calibration cycles. For each couple, the real static characteristic was found by fitting polynomials or rational functions on the samples through least-squares regression. The adjusted R-squared ($R_{adj}^2$) metric was used to set the polynomials’ degree to choose the best fitting with the minimum function complexity. Then, for each couple, two error metrics were computed as follows:

• Nonlinearity: illustrates the maximum distance between the real characteristic and the ideal one (output equal to input);
• Hysteresis: represents the maximum distance between the two curves that compose the hysteresis cycle.

In the dynamic experiment, the human operator performed six different sinusoidal movements. In each movement, the frequency was qualitatively set by the operator to low (<0.4 Hz) or high (>0.4 Hz), while the amplitude was set to small (40% of $B{L}_r$ ROM explored), medium (60% of $B{L}_r$ ROM explored), or large ($B{L}_r$ reaches saturation). For each movement, the variables that determine the aforementioned errors were recorded at 40 Hz, which is equal to the sampling frequency of the teleoperation systems. For each couple of variables, the signals corresponding to the same recording were synchronised by compensating the frame lag underlined by the cross-correlation. The distance between the signals was extracted as a metric of dynamic accuracy, computed as the Root-Mean-Squared Error (RMSE):

$$\mathrm{RMSE}=\sqrt{\frac{1}{N}\sum _{i=1}^N{\left(x_i-y_i\right)}^2}$$

(15)

where N is the number of samples and $x_i$ and $y_i$ are the ith samples of the two signals, respectively. Moreover, the computed lag in the case of the overall teleoperation systems was used as a metric of the mean overall delay.

5.2.3. User Demonstration Experiment

To further verify and compare the feasibility and performance of the two frameworks in performing both power and precision grasp tasks, a user demonstration experiment was designed and implemented. In this experiment, 6 users were invited to perform the grasping task, and all of them were right-handed, in good health condition, and had no motor deficiency. The detailed experimental setup and steps are given as follows.

Material: As shown in Figure 10a, 4 objects were selected for grasping and divided into two groups: power grasp and precision grasp [2,33]. Each group required a different grasping posture: Object 1 (a parallelepiped, large-diameter power grasp), Object 2 (a cork stopper, tripod precision grasp), Object 3 (a cylinder, medium wrap power grasp), and Object 4 (a soft cube, quadpod precision grasp). Note that Object 2 and Object 4 belong to the precision grasp group and had to be grasped with only the bionic hand fingertips, without touching the palm. Following the completion of each experiment, each user was asked to complete a NASA TLX [34] questionnaire to assess their perceived workload.

Tasks: As shown in Figure 10b, the bionic hand was fixed to a table, and the user sat in front of it. The user approached the bionic hand with an object in his or her left hand and then performed the bionic hand grasp. Then, once the user was confident enough, he or she left the object in the robotic hand, which in turn held the object for 5 s, indicating that the grasp was successful. If the object fell or touched the palm (only for the precision grasp), the trial was deemed a failure. Each user was asked to perform 6 trials with the glove and vision-tracking methods for each object. Hence, 144 trials were performed for each method.

6. Results

This section gives the results of the designed experiments in the previous section, including the human hand range of motion estimation, the accuracy performance comparison, and the user demonstration results.

6.1. Human Hand Range of Motion Estimation Results

Four complete calibration cycles were measured for each system. There were 152 static poses obtained while using the wearable system, 56 of which related to extension, 52 to flexion, and 44 to saturation. As regards the vision-based system, one calibration cycle was excluded because it was affected by markers’ occlusion, thus obtaining 96 static poses, 33 of which related to extension, 34 to flexion, and 27 to saturation.

Figure 11 shows for each teleoperation system the fitting result of linear regression and the fitted samples. The resulting ROM was $[1.22°;38.8°]$ for ${\phi }_{real}$ (Figure 11a) when the operator used the glove and $[-5.09°;52.7°]$ for ${\beta }_{real}$ (Figure 11b) when the operator used the vision-based tracking method. As illustrated in the experiments of the estimation results, the human hand ROM varied when using the glove-based and vision-tracking-based methods, respectively. Hence, the normalisation for the human ROM is necessary to achieve intuitive and transparent teleoperation control.

6.2. Accuracy Performance Comparison Results

6.2.1. Overall Teleoperation Control Performance

Polynomial functions were adopted to find the static characteristics of the two overall teleoperation systems, which are the expressions that give $B{L}_r$ as a function of $F{L}_h$ or $B{L}_h$. The fitting curves according to the adopted criteria are represented in Figure 12, together with the fitted samples.

Table 1. Fitting results and error metrics of the two implemented teleoperation systems.

System	Direction	Polynomial Degree	RMSE	${\mathit{R}}_{\mathit{a}\mathit{d}\mathit{j}}^2$	Nonlinearity	Hysteresis
Wearable	Flexion	5th	0.0225	0.993	11.1%	14.6%
	Extension	2nd	0.0273	0.989
Vision	Flexion	1st	0.0555	0.964	20.0%	27.0%
	Extension	2nd	0.0609	0.934

summarises the fitting results and the resulting metrics about nonlinearity and hysteresis.

The wearable system is characterised by a linear region from 25% to 60% of the flexion level range, in accordance with the ideal characteristic. Nonlinearity is significantly present only outside this region, and it peaks at 89% of the range, which also corresponds to the peak of hysteresis. Fitting results in the case of the vision-based system were poorer because the dispersion of the samples was higher, hinting at lower behaviour predictability, mapping repeatability, and precision. The ideal characteristic runs from end to end of the hysteretic cycle and is completely detached from the two curves, implying that nonlinearity exists throughout the range. Nonlinearity and hysteresis peak at 20% and 26% of the range, respectively. As reported by Table 1, such behaviour gives worse error metrics.

The dynamic trials of the glove-based system shown in Figure 13a reported well-overlapped human and robot signals, except for a few missed peaks and a systematic vertical shift for the small-amplitude, high-frequency trial. The general trend, for this system, appears to be that movements with a lower frequency and amplitude are better transferred to the robot.

As regards the dynamic trials of the vision-based system, although the signals are highly correlated, they are less superimposed than in the wearable system case, and missing peaks are more visible. The small-amplitude, high-frequency trial was the worst case as for the glove system. In this case, the general trend appears to be that movements with both high frequency and high amplitude are more easily transferred to the robot. Based on the experimental results, the RMSE values reported in

Table 2. Mean delay in ms and RMSE of the two teleoperation systems for each motion condition. The terms large, medium, and small refer to the signal amplitude, while low and high refer to the signal frequency.

	System	Large-Low	Medium-Low	Small-Low	Large-High	Medium-High	Small-High
Delay (ms)	Wearable	400	225	125	175	175	150
	Vision	350	325	0	250	225	150
RMSE	Wearable	4.01%	3.40%	2.80%	5.67%	4.44%	7.33%
	Vision	8.72%	6.56%	8.51%	6.42%	7.19%	12.6%

are higher in the vision-based system in all six conditions, indicating a greater teleoperation error than in the glove case. Moreover, the vision-based system performed worse for slow and small movements, indicating that it is less suitable for fine and delicate operations.

Both static and dynamic trials showed that the accuracy of the vision-based teleoperation method was worse with respect to the glove-based one. On the contrary, there was no discernible systematic difference between the time delays that characterised the two systems, as reported by Table 2. In both cases, it was always under 0.5 s, with a mean value across the six trials of about 0.2 s, which is acceptable and does not compromise the teleoperation activity.

6.2.2. Human Hand Motion Tracking Performance

Polynomial functions were explored to find the static characteristics of the motion tracking stages of the two teleoperation systems, which are the expressions that give $F{L}_m^{*}$ or $B{L}_m^{*}$ as a function of $F{L}_h$ or $B{L}_h$, respectively. The two datasets the functions were fitted on are the same employed in Section 6.1 to find the human angle range. According to the adopted criteria, the best-fitting curves had the same polynomial degrees as those used in Section 6.2.1 to model the characteristics of the two overall teleoperation systems. Indeed, a comparison of Figure 11a,b with Figure 12a,b reveals that the data distribution in the two tracking system stages was very similar to that in the two overall teleoperation systems.

Table 3. Fitting results and error metrics of the two implemented tracking stages.

System	Direction	Polynomial Degree	RMSE	${\mathit{R}}_{\mathit{a}\mathit{d}\mathit{j}}^2$	Nonlinearity	Hysteresis
Wearable	Flexion	5th	0.0184	0.995	12.5%	16.2%
	Extension	2nd	0.0252	0.992
Vision	Flexion	1st	0.0548	0.9631	18.0%	25.1%
	Extension	2nd	0.0590	0.946

summarises the fitting results and the resulting metrics about nonlinearity and hysteresis.

In both systems, the main differences from the overall teleoperation system case were a lower dispersion of the samples around the curves and a lower hysteresis below 20% of the human angle range. These distinctions resulted in better fitting and different metrics of nonlinearity and hysteresis, which were higher in the wearable system case and lower in the vision system case. Once again, samples’ dispersion, nonlinearity, and hysteresis in the vision system case were significantly worse.

Results from the dynamic part of the experiment agree with the static one.

Table 4. Tracking systems’ dynamic performance comparison: for each motion condition, the RMSE is reported as a percentage of the human finger ROM. The terms large, medium, and small refer to the signal amplitude, while low and high refer to the signal frequency.

System	Large-Low	Medium-Low	Small-Low	Large-High	Medium-High	Small-High
Wearable	3.20%	3.05%	2.23%	3.89%	4.71%	7.50%
Vision	8.14%	6.62%	7.44%	5.70%	7.01%	12.5%

illustrates the RMSE of the two teleoperation systems for each motion condition. In most of the conditions, the RMSE was slightly lower than the one computed for the overall teleoperation system. Once again, the vision-based system was affected by more evident missing peaks and greater RMSE values in all six conditions.

Both static and dynamic trials suggest that the vision-based tracking method is less accurate than the glove-based tracking method. Since the actuation control stage is the same for both systems, the lower accuracy of the vision-based method, underlined in Section 6.2.1, is solely due to the worse accuracy performance of the vision tracking stage.

Furthermore, given that in both developed teleoperation systems, the processing on the sensor data and the mapping complexity were reduced to the minimum, such worse performance can be related to the lower suitability of the LMC to be applied, with simple processing, as the controller for the teleoperation of a bionic hand. The reason may lie in the insufficient accuracy of the image processing procedure performed by the LMS while reconstructing the joint’s position. In the case of robot teleoperation in real scenarios, higher accuracy is advisable to grant safety and effectiveness to the control; thus, further research is needed to develop more robust algorithms for joint position estimation.

6.2.3. Bionic Hand Actuation Control Performance

Rational functions were explored to find the static characteristic of the actuation control stage, common to both teleoperation systems, that is the expressions that give $B{L}_r$ as a function of $F{L}_m^{*}$ or $B{L}_m^{*}$.

Table 5. Fitting results and error metrics of the implemented actuation control system.

Direction	Polynomial Degree	RMSE	${\mathit{R}}_{\mathit{a}\mathit{d}\mathit{j}}^2$	Nonlinearity	Hysteresis
Flexion	3rd/4th	0.0139	0.998	4.11%	7.37%
Extension	3rd/1st	0.0132	0.999

summarises the degrees of rational function polynomials, the goodness-of-fit metrics, and the accuracy metrics. The RMSE is lower than the tracking stage one. Above 20% of the range, the curves are superimposed to the ideal behaviour. Nonlinearity is present only under 20% of the range. Samples’ dispersion is also higher in that region.

Then,

Table 6. Actuation control stage dynamic performance: for each motion condition, the RMSE is reported as a percentage of the bionic finger ROM. The terms large, medium, and small refer to the signal amplitude, while low and high refer to the signal frequency.

Large-Low	Medium-Low	Small-Low	Large-High	Medium-High	Small-High
1.52%	1.27%	0.78%	2.03%	1.68%	3.00

summarises the RMSE of the actuation control for each motion condition. In all conditions, the resulting robot path followed precisely the desired one defined by the measured human pose. The RMSE was always under 3% of the range.

Both static and dynamic results showed that the accuracy error of the actuation control system is not relevant for finger teleoperation. This was confirmed also by the high similarity between the results of the overall teleoperation and the ones related to motion tracking alone, underlined in Section 6.2.2. Therefore, the inverse kinematic model obtained through finger motion calibration, applied through a proper algorithm, proved to grant a generally accurate and reliable actuation control system, demonstrating improvements compared with the linear motion mapping approach.

6.3. User Demonstration Result

The screenshots in Figure 14a–d show the grasp scenario when using the vision-tracking-based method to grasp the objects, while Figure 14e–h are the screenshots when grasping the four objects utilising the glove-based one. As illustrated in these figures, the user can grasp different types of objects, using both the wearable glove and the vision-tracking-based methods, regardless of the power or precision grasp.

Results from the usability experiment in

Table 7. Comparison result from the user demonstration experiment. The success rate and the average adjusted ratings of workload indexes considered by the NASA TLX questionnaire are reported. The total workload is the sum of the adjusted ratings and can range from 0 to 100.

Evaluation Index	Wearable System	Vision System
Mental demand	8.4	7.9
Physical demand	7.1	7.2
Temporal demand	3.1	3.8
Performance	3.5	5.9
Effort	12	9.5
Frustration	4.7	3.5
Total workload	38.8/100	37.8/100
Success rate	$98.6\%$	$96.5\%$

show that the mean perceived workload was practically the same in the two systems. However, in the case of the glove, on average, the workload component due to global effort was perceived as higher, and the performance was perceived as higher as well. The success rate was high in both cases and slightly higher for the glove. Free user feedback was in line with questionnaire results: users felt the glove was better performing, but at the same time, less comfortable, causing greater effort.

The user demonstration results showed that the inaccuracy issue of the vision-based method does not compromise the teleoperation control; nevertheless, it has an impact on the teleoperation control performance. As regards the glove-based method, the user’s comfort is a critical issue. Although wearable devices are more accurate than vision-based ones, their applications may be limited if the user’s comfort is not prioritised, thus making vision-based methods become promising solutions. Therefore, to make wearable systems more competitive, mechanical hardware should be improved, taking user comfort into account.

7. Conclusions

This paper primarily compared two methods for teleoperation control of an underactuated bionic hand, namely wearable-glove-based and vision-tracking-based, respectively. First of all, the kinematics modelling of the bionic hand is a critical issue considering the underactuated and nonlinear characteristics of the bionic hand. In this paper, the calibration data from an external NDI tracking system were used to generate the kinematics model of the underactuated bionic hand, which was considered the ground truth. Then, how to achieve an intuitive motion mapping from the human hand to the bionic hand is another challenging issue. To solve this, the flexion and the bending angles of the human hand fingers were defined and measured by the two tracking systems, respectively. Besides, they were normalised for their own range and mapped to the bionic hand motion to achieve an intuitive teleoperation control. Furthermore, the accuracies of the two systems were compared, suggesting a higher performance of the glove-based method. Finally, the results of the accomplished precision and power grasp tasks in the user experiment demonstrated the feasibility of the proposed methods. Such an experiment revealed that the glove-based method grants a higher performance, but is also less comfortable and requires greater effort by the user. Future work will focus on integrating the developed teleoperation control frameworks and bionic hands into practical applications and scenarios.