Grasping Assisting Algorithm in Tele-Operated Robotic Gripper

Abstract

The involvement of Robots and automated machines in different industries has increased drastically in recent years. Part of this revolution is accomplishing tasks previously performed by humans with advanced robots, which would replace the entire human workforce in the future. In some industries the workers are required to complete different operations in hazardous or difficult environments. Operations like these could be replaced with the use of tele-operated systems that have the capability of grasping objects in their surroundings, thus abandoning the need for the physical presence of the human operator at the area while still allowing control. In this research our goal is to create an assisting system that would improve the grasping of a human operator using a tele-operated robotic gripper and arm, while advising the operator but not forcing a solution. For a given set of objects we computed the optimal grasp to be achieved by the gripper, based on two grasp quality measures of our choosing (namely power grasp and precision grasp). We then tested the performance of different human subjects who tried to grasp the different objects with the tele-operated system, while comparing their success to unassisted and assisted grasping. Our goal is to create an assisting algorithm that would compute optimal grasps and might be integrated into a complete, state-of-the-art tele-operated system.

1. Introduction

In recent years, robots, and specifically robotic grippers and arms, have been integrated into a wide variety of industries, working side by side with humans or replacing human activity entirely. Tele-operated systems are robotic systems, operated and controlled by a human operator from a long distance, designed to fulfill specific tasks, such as grasping an object.

The act of grasping an object with a tele-operated gripper is a challenging task, mainly due to the lack of sensory information our body receives while touching an object. One of the ways to compensate for the absence of sensory feedback is by assisting the human operator with choosing a stable grasp that would ensure firm handling of the object.

Our research goal is to assist operators using tele-operated systems in grasping different objects. The goal is achieved by suggesting a stable grasp solution, which is provided by an algorithm we created. The algorithm’s purpose is to find the optimal force-closure grasp for a specific object.

To prove that our proposed solution indeed assists the operator and improves his grasp, we tested the operator’s grasp performance while mimicking our proposed solution against an unassisted case.

The main contribution of this paper is the proposed algorithm, which can be implemented with a vast number of robotic grippers and applied to many objects.

The first researches on grasping theory began by classifying human grasps and creating a human hand taxonomy.

Cutkosky and Howe [1] were among the first to differentiate between two types of grasps: power grasps that rely on security and stability, and precision grasps that rely on dexterity and sensitivity. Feix et al. [2] extended that work and classified 33 different grasps into various subgroups. Their classification divided grasps into power, intermediate and precision grasps, the type of opposition (pad, palm or side opposition), and the index of the “virtual finger” used [3,4].

Due to the complex nature of the human hand, consisting of an overall 30 different degrees of freedom, mimicking the dynamics of the hand with robots is a complex task, so researchers turned to a simpler description of the complex hand movement, based on synergies, as described in [5,6,7,8]. The advantage of using hand postural synergies can be seen in the work of Palli et al. [9], who reconstructed several synergies based on measurements performed on humans grasping different objects, in order to use them in robotic movements.

In his paper, Bicchi [10] summarized and investigated the properties of form-closure and force-closure and showed the need for a quantitative quality measure of the force-closure property, so as to choose between different grasps. Suarez, Roa and Cornella [11] reviewed many quality measures, based on the geometry of the grasp or limitations on the contact forces, whereas Kim et al. [12] focused on dimensionless measures. Borst, Fischer and Hirzinger [13] suggested a way to implant the task wrench space of the problem to create a more uniform quality measure, and Ohev-Zion and Shapiro defined a grasp quality measure based on the stability of compliant contacts [14,15].There have been various attempts to develop algorithms to find an optimal grasp. Ficuciello et al. [9] developed an algorithm based on neural network and reinforcement learning to find the optimal path to grasp an object with a robotic arm based on some quality criterions. Lenz, Lee and Saxena [16] found a solution based on deep learning and visual scanning of the objects to be grasped. Sinatov and Shapiro [17,18] tried to deduce the optimal common grasp that would fit a number of different objects. Probably the most common of the different quality measures is the “minimal inscribed ball in the grasp wrench space” criterion as proposed by Kirkpatrick et al. [19] and Ferrari and Canny [20].

Grasping an object with a tele-operated system requires communication between a “master” operator and a “slave” robot, based on a varied level of control and information provided. Various uses of system like this can be examined from different angles: Ni et al. [21] predicted the movement of an object being manipulated with a tele-operated manipulator, and corrected the prediction in the virtual reality based on data from the “slave” unit. Bernardino et al. [22] used a tele-operated arm and gripper to mimic humans in creating precision grasps, thus building a set of synergies to the gripper. An ability to control the “slave” gripper using visual input command from the human “master” was introduced by Li et al. [23] with the help of a deep neural network. Nisky et al. [24,25,26,27] presented differences in the performances of novice and expert surgeons when using a tele-operated system to preform surgical cuts.

In this study we tried to create an assisting algorithm for tele-operated gripper users, advising them how to grasp an object and checking if their grasp was improved in relation to the unassisted case. Although the algorithm is applicable in generic robotic grasping, the purpose of the research is to advise the operator on grasping, and not force a specific solution. The suggested grasp was calculated through an optimal grasp searching algorithm.

In Section 2 we will analyze the calculations required for finding an optimal grasp, and in Section 3 we elaborate about our searching algorithm. Section 4 deals with the series of tests we conducted to examine the improvement of the users grasping performance, and in Section 5 a discussion on the results is presented.

2. Grasp Quality Computation

2.1. General Grasp Formulation

The interaction between a given object and the robotic gripper in ${\mathcal{R}}^3$ space is analyzed through six-dimensional grasp wrench space. The object $\mathcal{B}$’s center of mass is located at ${\underline{r}}_B\in {\mathcal{R}}^3$ in relation to the world frame coordinates and rotates around three different axes, defined by $\underline{\theta }\in {\mathcal{R}}^3$, resulting in a total of six degrees of freedom. For our purposes we define a gripper reference frame with the origin at the gripper’s wrist, and an object reference frame originating at the center of mass. These coordinate frames are noted by the indexes g and o, respectively. We denote the contact points between the body and the gripper as ${\underline{p}}_i\in \mathcal{S}$, where $\mathcal{S}$ is the object’s surface and is given in the object’s coordinates. We could also define the contact points in relation to the gripper, as seen in Equation (1). The distance between the object origin and the gripper origin is donated by $\overrightarrow{O_g{O}_o}$.

$${\underline{p}}_{g,i}=\overrightarrow{O_g{O}_o}+{\underline{p}}_i$$

(1)

To calculate the grasp wrench space we need the contact forces between the gripper and the object. Using Coulomb’s model of friction, our frictional force at the contact point is bounded by a cone centered around the normal to the contact point ${\underline{p}}_i$. Thus, we define the Friction Cone in Equation (2).

$$F{C}_i=\left\{\begin{array}{c}{f}_{n,i}\hfill \\ f_{t,i}\hfill \\ f_{s,i}\hfill \end{array}:\sqrt{f_{t,i}^2+f_{s,i}^2}\le {\mu }_{s,i}{f}_{n,i},\forall f_{n,i}>0\right\}$$

(2)

in Equation (2), $f_{n,i},f_{t,i}$ and $f_{s,i}$ are the normal and tangential force components of the ith contact on the object, and ${\mu }_{s,i}$ is the static friction coefficient. The normal to surface $\mathcal{S}$ points inwards into $\mathcal{B}$, so we demand that the normal component of the force would be definite and positive, meaning the finger pushes the object and cannot pull. Describing the forces inside the friction cone with an approximated linear combination results in the Linearized Friction Cone described in Equation (3).

$$LF{C}_i=\left\{{\underline{F}}_i:{\underline{F}}_i=\sum _{k=1}^s{a}_{ki}{\widehat{f}}_{ki},\forall a_{ki}\ge 0\right\}$$

(3)

The vectors ${\widehat{f}}_{ki}$ represent the base vectors of the polytope, while ${\underline{F}}_i$ is the total contact force at contact point ${\underline{p}}_i$.

Lastly, we define the wrenches exerted on the object in Equation (4).

$${\underline{w}}_i=\left(\begin{array}{c}{\underline{F}}_i\\ \frac{1}{\lambda }{\underline{p}}_i\times {\underline{F}}_i\end{array}\right)$$

(4)

The Wrench is a six-dimensional vector, containing information regarding both the force at a contact point on the object $\mathcal{B}$ and the moment exerted by this force in relation to the center of mass. The parameter $\lambda$ represents a relative length that converts the grasp wrench space to uniform units. Using the above Linearized Friction Cone, we define the primitive wrenches at point i, as described in Equation (5).

$${\underline{w}}_i=\left(\begin{array}{c}{\underline{F}}_i\\ \frac{1}{\lambda }{\underline{p}}_i\times {\underline{F}}_i\end{array}\right)=\sum _{k=1}^s{a}_{ki}\left(\begin{array}{c}{\widehat{f}}_{ki}\\ \frac{1}{\lambda }{\underline{p}}_i\times {\widehat{f}}_{ki}\end{array}\right)=\sum _{k=1}^s{a}_{ki}{\widehat{w}}_{ki}$$

(5)

2.2. Stable Grasps

Before examining the stability of grasps, an important assumption needs to be taken concerning the rigidness of the object. Under certain loads a soft object could deform and the wrenches acted on it would change, and on the other hand, when the object is extremely hard the grasp will become unstable due to massive loads (a phenomena called “coin snapping”). In the following physical treatment we assume rigid objects with small loads.

We illustrate the stability of our grasp with the force-closure property. A grasp is said to be a force-closure if we can counterbalance any external loads and torques exerted on the body using the contact forces. This way the body can be maintained in static equilibrium.

The Grasp Wrench Space (GWS) is defined as the set containing all the primitive wrenches in a given grasp in Equation (6).

$$GWS=\left\{\begin{array}{ccccc}{\widehat{w}}_{11}& {\widehat{w}}_{21}& {\widehat{w}}_{31}& \dots & {\widehat{w}}_{sn}\end{array}\right\}$$

(6)

In our case the conditions for a grasp to be a force-closure are [28]:

• The origin of the wrench space is in the interior of the convex hull of the GWS. This is a necessary condition for any grasp to be a force-closure.
• Using a positive linear combination of the wrenches, the external wrench on the body (due to external force) is counterbalanced. In practice, it means we sustain static equilibrium.

The first condition can be satisfied using hypergeometry. Because  the GWS is a ${\mathcal{R}}^6$ shape, The facets of its convex hull rest on five-dimensional hyperplanes. We describe these facets by the hyperplane equation in Equation (7).

$$\partial CH=\left\{{\mathcal{F}}_j|{\mathcal{F}}_j\equiv \sum _{k=1}^6{h}_{jk}{x}_k+b=0,j=1\dots m\right\}$$

(7)

Each facet is made up of six vertices (or six wrench vectors) of CH(GWS). ${\underline{h}}_j$ is the normal vector to ${\mathcal{F}}_j$, b is the offset from the origin, and x describes the coordinates of wrenches on the facet. We wish to calculate the distance between the origin to a hyperplane using the formula in Equation (8).

$$d_0=\frac{\left|b\right|}{\parallel \underline{h}\parallel }$$

(8)

To check that the origin lies in the same half-space divided by the hyperplanes of $\partial \mathit{CH}$ we calculate the signed distance between each facet and a known inside point, and then compare their signed distances. Let r be a point in space that is the “center of mass” to the primitive wrenches of the grasp wrench space, as described in Equation (9).

$$\underline{r}=\frac{1}{ns}\sum _{i=0}^n\sum _{j=0}^s{\widehat{w}}_{ij}$$

(9)

We check the signed distance using the following formula for both $\underline{r}$ and the origin in Equation (10).

$$d=\frac{\underline{h}·\underline{x}+b}{\parallel \underline{h}\parallel }\to \left\{\begin{array}{c}\hfill d_r=\frac{\underline{h}·\underline{r}+b}{\parallel \underline{h}\parallel }\\ \hfill d_o=\frac{b}{\parallel \underline{h}\parallel }\end{array}\right.$$

(10)

If the equality $sign\left(d_r\right)=sign\left(d_o\right)$ is correct, then both the origin and $\underline{r}$ lie in the same half-space divided by the hyperplane. To satisfy our first condition for force-closure we need to check the above equality for each facet in the convex hull. Even if there is only one facet where $sign\left(d_r\right)\ne sign\left(d_o\right)$ our origin is on the wrong side of the hyperplane and our grasp is not a force-closure.

The second condition is easily satisfied by Equation (11):

$${\underline{w}}_{\mathrm{external}}+\sum _{i=1}^n{\beta }_i{\underline{w}}_i=0,\forall M_i\ge {\beta }_i>0$$

(11)

The coefficients ${\beta }_i$ of our linear combination are bounded. The lower bound satisfies our demand for the fingers to push the object and not pull it. The upper bounds $M_i$ comes from the physical realization that our gripper is limited by the amount of grasp force it can exert. Consequently, the grasp wrench set is bounded by a maximal convex hull. A practical solution to check the existence of the second condition is to assume a new, shifted origin of the grasp wrench space. If we define the origin as ${\underline{w}}_o=-{\underline{w}}_{external}$ and check again that $sign\left(d_r\right)=sign\left(d_o\right)$ for each facet we can satisfy the second condition.

If both conditions are satisfied, we claim that object $\mathcal{B}$ is firmly secured.

2.3. Grasp Quality Measure

Many grasps can be used to reach force-closure on the body, and so we wish to find the best grasps. For this purpose, we define the grasp quality measure.

The grasp quality measure is calculated based on the properties of the grasps, and it rates a specific grasp relatively to other grasps. Using the process described by Roa and Suarez [11] and a similar concept in [29], we found the quality measures that best fit for our purposes. The connection between the robot motor torques and the end-point force (fingertip force) is described by the Jacobian matrix $J_i$. Our gripper has several fingers, so we describe the connection between all the motor’s torques and all end-effectors’ forces with the hand Jacobian matrix $J_h=diag$ $\left(\begin{array}{cccc}{J}_1& J_2& \dots & J_n\end{array}\right)$, and the total result is presented in Equation (12).

$$\underline{T}=J_h^T{\underline{f}}_T$$

(12)

The vector ${\underline{f}}_T$ is given by the xyz components of all the finger forces we described earlier, namely ${\underline{f}}_T=$${\left(\begin{array}{ccccc}{f}_{(}n,1)& f_{(}t,1)& f_{(}s,1)& \dots & f_{(}s,n)\end{array}\right)}^T$ equally, and the connection between the motor velocities $\underline{\dot{\varphi }}$ and end-effector velocity $\underline{v}$ is given in Equation (13).

$$\underline{v}=J_h\underline{\dot{\varphi }}$$

(13)

The connection between the forces acting at the contact points between the gripper and the object, to the net wrench acting on the object, is given by the grasp matrix G presented in Equation (14).

$$\underline{w}=G{\underline{f}}_T$$

(14)

Additionally, the connection between the fingers’ velocities and the object velocity is given in Equation (15).

$$\underline{v}=G^T\underline{\dot{x}}$$

(15)

From these definitions we receive the connections in Equation (16).

$$\left\{\begin{array}{c}\underline{T}=J_h^T{G}^{-1}\underline{w}\hfill \\ \underline{\dot{x}}=G^{-T}{J}_h\underline{\dot{\varphi }}\hfill \end{array}\to \left\{\begin{array}{c}\underline{T}=H^T\underline{w}\hfill \\ \underline{\dot{x}}=H\underline{\dot{\varphi }}\hfill \end{array}\right.\right.$$

(16)

The matrix H is called the hand object Jacobian, and its dimensions are $H\in {\mathcal{R}}^{d\times nm}$, where $d=6$ is the dimension of the grasp wrench space, n is the number of contact points and m is the number of motors in each finger.

Under the assumption that a boundary to the size of torques exists and is a function of the power P, we write Equation (17).

$$\parallel \underline{T}\parallel \le f\left(P\right)$$

(17)

Substitution of $\underline{T}$ with $H^T\underline{w}$ results in limitation in the wrench space, as described in Equation (18).

$${\underline{w}}^T\left(H{H}^T\right)\underline{w}\le f\left(P\right)$$

(18)

The above transformation matrix transforms a sphere in $nm\times nm$ torque space with radius $r=f\left(P\right)$ to an ellipsoid in $d\times d$ wrench space. The same process can be applied to object $\mathcal{B}$ velocity space resulting in the connection described in Equation (19).

$${\underline{\dot{x}}}^T{\left(H{H}^T\right)}^{-1}\underline{\dot{x}}\le g\left(P\right)$$

(19)

These two transformations have the same eigenvalues and same eigenvectors but in inverse proportion; when we apply the maximal wrenches on the body, we get the minimal velocity and vice versa. While the primitive wrenches are not bound by the ellipsoid we have obtained, and consequently our convex hull, $\mathit{CH}\left(\mathit{GWS}\right)$ might intersect the boundaries of the ellipsoid, the net wrench exerted on the body is bounded by the ellipsoid, and that is because we wish to maintain static equilibrium in the force-closure grasp.

Following the above conclusions, we wish to define a grasp quality measure for two cases. The definitions from [1,2] will be used to define these types of grasps:

• Power grasp—the goal is to hold the object as firmly as possible, preventing movement, resisting the biggest external loads and exerting the maximal contact forces.
• Precision grasp—the goal is to control the movement of the body as much as we can, with little to no force exerted on the body.

In the case of power grasp, we wish to increase the smallest possible wrench that can be resisted (the minor axis of the wrench ellipsoid), and so we use the most common grasp quality measure, the inscribed sphere method, presented in Equation (20).

$$q=\underset{\rho \in \partial CH\left(GWS\right)}{max}\parallel \rho \parallel$$

(20)

where $\rho$ is the radius of the sphere and q is the received score for the grasp. The maximal radius of the inscribed hypersphere that touches the closest facet represents the biggest external wrench that can be applied on $\mathcal{B}$ in any direction.

In the case of the precision grasp, we wish to increase the minimal range of movement with little to no force (the minor axis of the velocity ellipsoid). Because the velocity ellipsoid and the wrench ellipsoid are connected to each other, and have the same axes with inverse proportions, in the wrench space we wish to decrease the maximal wrench we can resist (the major axis of the wrench ellipsoid) and the minimal radius of the circumscribed hypersphere of the convex hull. Thus, we find a simple quality measure for the precision grasp in the form of the minimal between the maximal wrenches, shown in Equation (21).

$$q=\underset{\mathbf{p}\in \partial CH\left(GWS\right)}{min}\parallel \mathrm{P}\parallel$$

(21)

3. Optimal Grasp Search Algorithm

In order to suggest an optimal grasp for each of the objects we wish to grasp with the tele-operated system, an algorithm composed of two parts was created. Our proposed method to find the optimal grasp of an object consists in creating a large number of force-closure grasps and optimizing the optimal solution by genetic optimization.

The first part of the algorithm includes an offline part, compromised of a series of simulations using MATLAB. The simulations’ purpose is to find the optimal grasp, as described thoroughly in Section 3.1. Each simulation includes randomly chosen grasps of the object’s surface, testing if they satisfy the two conditions for force-closure, ranking of the grasps by the chosen quality measure, and lastly optimizing the grasp population using a genetic algorithm.

The second part includes computing the grasp quality measure while the operator tries to grasp the object with the gripper. The analysis is executed in the same manner that the simulation in the first part was done, and then comparison between the two received values (optimal and test) is carried out.

While grasping, the online part of the algorithm receives sensory information of the active forces at the contact points (the fingertips of the gripper) and computes their position in space. Using this information, improvement to the grasp is suggested in order to achieve a better grasp, based on the defined grasp quality measure.

3.1. Finding the Optimal Grasp

The algorithm of the optimal grasp is given in Figure 1.

The algorithm receives the entire data on the object $\mathcal{B}$ and other relevant data (finger force assumption, number of contact points, external forces if exists).

First, we randomly chose a specified number of contact points from surface $\mathcal{S}$ and checked for force-closure. We used the chosen set of contact points ${\underline{p}}_i$ to build the grasp wrench space and pre-defined finger force directed normally to the surface $\mathcal{S}$ at the points ${\underline{p}}_i$. Once we computed our grasp wrench space, we could check for force-closure using Algorithm 1.

Algorithm 1  FC grasp.

Input  $G,{\underline{w}}_{external}$ Output  $FC,{\underline{a}}_{reg}$       Initialize FC = true, stating that we have a force-closure grasp.       Define, $\underline{o}=\underline{0}$ as the origin.       Create the 6D convex hull of G, $K=\mathit{CH}\left(\mathit{G}\right)$.       Define $\underline{r}$, a point inside K as the average of all primitive wrenches.       for number of hyper facets in K           Find the points that lie on the facet. Organize them in A.           Compute the signed distance between the facet to $\underline{o}$,$\underline{r}$.           Put both results in $s\left(i\right)$       end for       for length of s           if the sign of both results in s is not equal               then FC = false               break           end if        end for        repeat from 5 with $\underline{o}=-{\underline{w}}_{external}$

A success in the force-closure test will give us a quality measure to rank our current grasp. The quality measure is computed as described in Section 2 and will be noted as $q_i$. A failure in the test will result in the algorithm searching for another suitable grasp, choosing another random points in $\mathcal{S}$.

At this stage a set of specific grasps that maintain force-closure on object $\mathcal{B}$ is sorted in a table called a Grasp Generation List, or GGL for short. The sorting is based on the quality measure $q_i$, where $q_1>q_2>\dots >q_{end}$ and the index represents the number of the row (in the case of precision grasp, $q_i$ is ordered in ascending order). Each row includes information about the contact points’ locations and contact forces.

Using a genetic algorithm, we optimize the table to receive the optimal grasp for the object. The algorithm is inspired by the evolution of species in biology, where desirable genes are inherited by the next generation.

For each generation we define a set of parent grasps, $P_1,P_2$, that contain the information from the best two grasps from the GGL table; these are the first two rows. The “genome” of the grasp contains the set of contact points of the grasp, $\underline{p_1}$ to $\underline{p_n}$. For each child grasp we mix the “genomes” of $P_1,P_2$ to create a new genome strain. The mixing occurs at a random cross section and is called a “crossover”.

The next phase would be the “mutation” phase. A mutation can occur in a genetic sequence describing our new child grasp, with low probability. The mutation is performed by switching one of our contact points $\underline{p_i}$ with a randomly selected point from the surface $\mathcal{S}$.

Once we created a new grasp we check if it satisfies a force-closure on the object and rank it among the grasps in GGL. After creating a child grasp, and reorganizing the table, we move forward to the next generation of grasps, using the currant parent grasps. By carrying out this iterative process we improve the received grasps. After 40 generations we find our optimal grasp, with the quality measure noted as $Q_{opt}$.

The initial population of the genetic algorithm is fixed in size, in our case six grasps at the start of each generation. after the “crossover” and “mutation” phases we add two new child grasps to the table (assuming they sustain force-closure), and delete the two least fitting grasps from the population. The mutation probability is calculated using the condition in Equation (22), and results are calculated with probability of $\approx 0.05$. Using trial and error we determined that 40 generations suffice for an improvement between the initial best grasp and final best grasp.

$$Cond=\{x:(xmod17)<1,\forall x<100,x\in \mathcal{N}\}$$

(22)

3.2. Comparing the Received Grasp and the Optimal Solution

We defined a value function to compare between the grasp achieved by the operator and the optimal grasp we found. The value functions are as follows. For the power grasp case we have the value function in Equation (22).

$$V=\frac{\parallel Q-Q_{opt}\parallel }{Q_{opt}}·100\%$$

(23)

For the precision grasp case we have the value function in Equation (23).

$$V=\frac{Q_{opt}}{Q}·100\%$$

(24)

To compute the received grasp, we track and send commands to the gripper, using a tele-operated apparatus (in our case, a control glove). Once the operator achieved the desired grasp with the robotic gripper, we compute the grasp quality measure as described in Section 2. The computation demands that we collect data from the gripper, mainly the forces at the contact points (the finger tips) and the location of the contact points in the object’s coordinate frame. The forces acting at the fingertips are extracted from pressure sensors located at the contact points and directed at the relevant normal vector from the contact surface.

Using the contact points’ positions and forces, the primitive wrenches are computed and the GWS matrix is created. The creation of the GWS convex hull is done using our implementation of the QuickHull algorithm, as described by Barber and Dobkin [30]. Using the convex hull we easily check for our quality measure, either for power grasp or precision grasp.

4. Experimental Setup and Tests

4.1. Overview

In the following section we will describe the purpose of our tele-operated system, its structure, components and interactions. The next section describes the test area, and the last section describes the tests themselves.

The tele-operated system allows us to simulate the daily usage of different tools (hammer, screwdriver, tweezers), their manipulation in space, holding and grasping—all controlled by an operator from a long distance. In this study we used the system to grasp specified objects with a robotic gripper and arm, with the goal of assisting the action with output from our algorithm.

The experimental system was composed of an anthropomorphic gripper and long-distance control devices (tele-operated devices). Figure 2 shows the interaction between the different components in the system. The haptic control allows the human operator to preform the grasp of an object from a long distance. He/She controls the movement of both the arm and gripper with control devices, which translate the commands to movement in the physical world. While the robotic gripper grasps the object, necessary data for the calculation of the grasp quality are sent to the computer, which in turn computes and presents the achieved score. Lastly, the human receives visual output on the quality of their grasp, which helps him/her decide their next actions (improving their grasp or continuing with the specified task).

4.2. Equipment

Figure 3 depicts the control devices and robotic arm used in the experiments.

Table 1. Lab equipment used in the tests.

Device Purpose	Device Name	Manufacturer
Gripper	Schunk S5FH gripper	Schunk Gmbh & Co.
Gripper control	VMG30 glove	Virtual Motion Labs
Arm	UR10	Universal Robots
Arm control	Virtuose 6D	Haption
Visual output	52${}^{\prime \prime }$ LCD	LG

lists the devices used in the series of tests.

The most prominent part in our tests is the five-finger gripper, Schunk S5FH, seen in Figure 4. The gripper contains nine servo motors and 20 joints and thus provides complex movements similar to those of the human hand [31]. Forces are calculated with tactile pressure sensors, and the objects to be grasped are a hammer, screwdriver and tweezers.

The control of the tele-operated devices is divided into two parts. The first is the control of the robotic arm with haptic control rod. The communication between the two was programmed as described in Figure 5. For the purpose of our research a sufficient position resolution was chosen to be $\pm 0.5$ mm, and a time response of $<1$ s was chosen. The position resolution of the haptic control rod was $\pm 0.016$ mm and the robotic arm’s was $\pm 0.1$ mm. The reaction time, while not measured precisely, was estimated to be between 0.1 and 0.2 s. These mentioned values allow for 6D control of the arm with satisfactory reaction times and movement resolution.

The second tele-operation connection was between the smart glove and the gripper. The synchronization between the human’s finger movements and the gripper’s fingers was achieved using inverse kinematics, creating commands to the servo motors of the gripper. The glove calculates the user’s finger positions in relation to the wrist using thin bend sensors and orientation sensors. After filtering of the signals is done, the received positions are transformed to finger joint angles, using inverse kinematics and the geometry of the robotic gripper. Through the angles we computed the required input values to the servo motors, and lastly the gripper’s CPU commanded the robotic fingers to move accordingly.

Assessments of both the center of mass position and reference frame relations (gripper’s frame and object’s frame) were measured and calculated beforehand and inserted into the program calculations.

4.3. Test Area

The area of the tests is illustrated in Figure 6. The UR10 robotic arm and Schunk gripper were mounted on metal beams, which were contained in a rectangular space (two sided open box). Due to limited field of view from the backward part of the experimental system (where the metal beams, computer and wiring were positioned), the participants sat at the front-left side of the arm and gripper, allowing for a field of view on both the test area and TV screen mounted on the laboratory wall. The objects were positioned in the center of the rectangular space, reachable by the arm and gripper.

4.4. The Tests

A series of tests were conducted in order to see the improvement in grasp quality using our algorithm proposition. Each test subject was tested in stages as described below:

• The participant went through a training stage, composed of certain movements. The participant was required to move the robotic arm up, down, and to the sides. Additionally, he/she was required to close and open the fingers of the gripper. After successfully completing the actions the next stage was introduced.
• For each given object, the participant was asked to securely grasp the object, with the robotic gripper, choosing the contact points at will. External forces could be applied and measured by the sensors, based on the object. The grasp quality measure was calculated using the computer application. Success in this stage was defined as securing the object in the gripper’s fingers and palm, without relative movement. For each object the test was repeated three times.
• The participant was asked to repeat the tests (grasping the object with the robotic gripper), but this time being assisted by visual information on how to grasp the object. Then the grasp quality was calculated. Success in this stage was defined as securing the object in the gripper’s fingers and palm, without relative movement. For each object the test was repeated times.

5. Results Analysis and Discussion

A comparison between the performance of humans with the system against the hypothetical optimal quality measure was performed, based on 14 tested individuals and three grasped objects.

5.1. Optimal Algorithm Results

To achieve our optimal results for each object according to the classifications defined (“power grasp” vs. “precision grasp”), we required a list of relevant parameters for our computation.

Our algorithm received a point cloud collection of the object’s surface $\mathcal{S}$, a theoretical maximal normal force at each contact point $F_{N,max}$, estimated to be a different value for each type of grasp ($F_N=5.94$ [N] for power grasps and $F_N=1.1\ast M\ast g$ [N] for precision grasps), and lastly the coefficient of static friction, based on different values taken from the literature. Due to our uncertainty regarding the estimation of the coefficient of static friction and based on the values we encountered, we picked two values of ${\mu }_s$ in order to be able to also check a stricter case of the grasp presented in

Table 2. Grasp quality measure chosen friction values.

Object	Chosen ${\mathit{\mu }}_{\mathit{s}}$
Hammer	1, 0.75
Screwdriver	1, 0.65
Tweezers	0.9, 0.65

.

Our algorithm applies concepts from genetic algorithms. While running many simulations with our chosen ${\mu }_s,F_{n,max}$ we received a large diversity in our optimal quality measure for each of the objects, which suggests we have not found the true optimal grasp for the object.

A number of explanations for this result are suggested. One of them is the disadvantage of the genetic algorithms due to the possibility to converge to local optimum and not the global optimal solution of the problem [32]. Another explanation is that the set of solutions is not constrained by the kinematics of the gripper, resulting in nonphysical solutions (two contact points too far apart). Lastly, a crossover of two good grasps could still result in a worse grasp, thus increasing the diversity of results.

We conclude that the genetic algorithm improves the score of the fitness function (meaning that our final proposed grasp is better than our starting grasp) but different iterations of the same algorithm will not necessarily converge to the same solution. Under this assumption we wish to find a good score for our quality measure, understanding that it represents a local maximum of the grasp quality measure function and not a global maxima.

For each object we simulated $n=50$ iterations of optimal grasps, testing two different coefficients of static friction, where in each one of these we scanned 300–400 grasps (both force-closure and non force-closure), and then chose the highest power grasp or lowest precision grasp received score. Our “optimal” grasp quality measure was better than $O\approx {10}^4$ grasps, and therefore we are confident in our results.

Table 3. Optimal grasp quality measures.

Object	Grasp Type	${\mathit{\mu }}_{\mathit{s}}$	Best ${\mathit{Q}}_{\mathit{opt}}$ [N]	${\overline{\mathit{Q}}}_{\mathit{opt}}$	${\mathit{S}}_{{\overline{\mathit{Q}}}_{\mathit{opt}}}$
Hammer	Power	1	3.063	2.1149	0.399
		0.75	1.991	1.401	0.239
	Precision	1	37.772	62.354	11.49
		0.75	32.25	55.92	10.60
Screwdriver	Power	1	2.273	1.332	0.399
		0.65	1.739	0.905	0.288
	Precision	1	2.872	3.835	0.265
		0.65	4.277	5.606	0.742
Tweezers	Power	0.9	2.753	1.373	0.465
		0.65	1.513	0.970	0.288
	Precision	0.9	0.427	0.5787	0.07
		0.65	0.742	1.061	0.2519

shows our results and their diversity. The table displays the coefficient of static friction, the best quality measure, the average quality measure and the standard deviation of the quality measures, where the sample size is $n=50$.

In Figure 7 we show some of the visual output provided to the test subjects about the different optimal grasps. the suggested points of contact are marked by the red markers on the objects’ surfaces.

5.2. Test Results

Our test results are based on the performances of 14 participants. Figure 8 presents the age distribution in our test population. Detailed test results are provided in the Appendix A.

In the first phase, participants were introduced to operating the system through a training session. After that, each one of them was asked to grasp the object in any manner they should choose, where each participant was asked to grasp a number of times to increase the statistics on each object. Outliers were excluded from the results.

Table 4. Test subjects performance, unassisted grasping.

Object	Grasp Type	$\overline{\mathit{Q}}$	${\mathit{S}}_{\mathit{Q}}$
Hammer	Power	2.72	3.17
	Precision	206.57	12.75
Screwdriver	Power	2.48	3.68
	Precision	19.05	0.48
Tweezers	Power	1.48	1.36
	Precision	41.42	0.75

shows their performance without the assistance of the algorithm.

In the second phase participants were asked to repeat the grasping of the objects, but now trying to mimic the optimal grasps presented in the previous section.

Table 5. Test subjects performance, assisted grasping.

Object	Grasp Type	$\overline{\mathit{Q}}$	${\mathit{S}}_{\mathit{Q}}$
Hammer	Power	3.81	3.44
	Precision	214.95	3.38
Screwdriver	Power	4.76	4.67
	Precision	19.29	0.13
Tweezers	Power	3.53	3.36
	Precision	41.56	0.75

shows their performance with the assistance of the algorithm.

Figure 9 presents the distribution of the grasp quality measures, divided into power and precision grasps, and separated for each object.

In the case of power grasps, the higher the score obtained, the better the grasp. The results are divided in relation to the optimal grasp quality measures we found in our calculations.

In the case of precision grasps (seen at Figure 10), the lower the score, the better the grasp. we should note that the number of grasps examined were different in each case.

5.3. Discussion

Based on the results we obtained, we detected a general improvement in the grasping performance of the human operators.

Table 6. Grasp quality improvement with and without assisting algorithm.

Object	$\overline{\mathit{Q}}$ before [N]		$\overline{\mathit{Q}}$ after [N]		Improvement (%)
	Power	Precision	Power	Precision	Power	Precision
Hammer	2.72	206.57	3.81	214.95	42.5	−0.04
Screwdriver	2.48	19.05	4.76	19.29	125.2	−0.006
Tweezers	1.48	41.42	3.53	43.56	43.13	0.01

compares the performance before using the algorithm and after using it.

Using our algorithm, we saw a general improvement in performance of power grasping, by 42%–125%. However, for the case of precision grasping only negligible changes in performance were detected, ranging between −0.04% and 0.01% performance change (presented in Table 6).

Individual changes in performance are presented in Figure 11.

Some of the participants improved drastically while others did not. Comparing the performance against the optimal grasp obtained in Section 5.1, we see that while some of the test subjects reached a better score than the algorithm, more than half of the subjects improved their performance by more than 50% in relation to the score obtained by the algorithm.

While a general improvement in the power grasp performances was noticed, we did not see major changes in the precision grasps, even for objects that are delicate to handle (tweezers). In practice, this means that the distance from the origin of the grasp wrench space to the farthest vertex of the convex hull remained the same. A possible explanation is that there will always be one contact point where the force exerted has to be of a minimal size (to both sustain force-closure and counter the self weight of the object).

Looking at the results established the hypothesis that human operator performance is influenced by the following parameters:

• Age—older subjects had lower initial scores and lower improvement measures in comparison to younger subjects. It is possible that teaching older people how to grasp different objects is therefore a harder task than teaching young subjects.
• Working experience—unlike our initial hypothesis that experienced subjects that have many years of training with grasping tools in uncomfortable environments would preform better, we could not discern their performance in relation to other subjects’ performances. Our conclusion is that working experience has no influence on performance with tele-operated devices.
• Intuition with tele-operated system—subjects who had previous experience with the specific experimental system or with similar gear (playing video games with controllers) showed better results.
• Object/type of grasp—from the results we detected that the best performances were obtained with the screwdriver, a convenient tool, and not with the heavy hammer or the delicate tweezers.

To conclude our discussion, we would wish to compare our algorithm with a few other algorithms presented in this paper [9,16,17,18]. A comparison is presented in

Table 7. Algorithm comparison to selected algorithms.

Criteria/Algorithm	Our Algorithm	Synergy Evaluation	Deep Learning	Planar OCOG	n-Finger OCOG
Objects database	One object	One object	Several objects	Several objects	Many objects
Variety of grippers	Not relevant	Specific gripper	2 fingers grippers	3 fingers grippers	Not relevant
Simulating force and friction	Yes	No	No	Yes	Yes

.

The algorithm proposed in this paper is intended to grasp specific pre-defined objects, and thus does not require extensive initial information (although it could be implanted with an object database), in comparrison to algorithms using a large database of objects. It is not gripper-specific due to the non-relevance of the gripper’s geometry, which is relevant in other algorithms, and can be implemented with an infinite number of grippers. Another advantage is the flexibility of the algorithm in changing the force parameters (maximal contact force, coefficient of friction), allowing to simulate different grasp conditions without the need to test the situation in reality, unlike algorithms that use geometrical features only.

6. Conclusions

In this work we created an assisting algorithm for grasping objects with a tele-operated robotic gripper. The algorithm proposed in this research can be used with a vast number of grippers, and simulate grasping of different objects with different grasping scenarios (low friction, different grasping powers).

For a given set of objects, with a known surface and a predefined coefficient of static friction, our algorithm attempted different type of grasps, and found an optimal grasp based on quality measures we defined. Optimal grasps were calculated for each object, for both the power and precision types of grasps, and with two different coefficients of static friction (likely case and severe case). For each grasp we created a visual output for our test subjects. The grasp results of the algorithm, while not being the true optimal grasp for a given object, were ranked higher than random grasps of an object, which makes the genetic algorithm a valid optimization method for grasping problems.

In a series of tests we observed the performance of different operators using a tele-operated system. Each subject was asked to grasp the objects with and without the output from our algorithm, and we compared the results.

For the case of power grasping we have seen an average improvement of at least 40%. Most of the participants improved their performance, where the younger subjects improved better than most of the older subjects. While some of the subjects were able to perform better than the optimal grasp quality measure calculated, we deduced that following the visual information of the optimal grasp improved the performance of most of the test subjects.

For the case of precision grasping no noticeable changes in performance for any of the subjects were observed. Our assessment is that there is a minimal threshold of force exerted at the contact point on an object that stays the same for all grasps, and thus the circumscribing sphere of the GWS has minor changes.

The study presented in this paper does have its limitations. The algorithm is not applicable to grasping of soft objects or grasping with contact areas, due to the physical model implemented. Initial assumptions about the forces and friction have to be assumed in order to receive results. Additionally, while not part of this study, translation of the visual output provided by the algorithm into visual instructions in augmented/virtual reality could drastically increase the quality of assistance.

To conclude, the tests’ results show that the proposed algorithm indeed assists the operator in securing an object with tele-operated devices, in spite of the challenges presented by the problem.

7. Future Work

Future research and related work on the subject should include:

• Improving the search for candidate grasps, taking into consideration the geometrical constraints of the gripper, and reducing the initial amount of force-closure calculations.
• Finding stable grasps for soft or very rigid objects, and increasing the range of applications of the algorithm.
• Improving the optimization of the received optimal grasp, using a genetic algorithm to avoid local maximal quality measure or using another optimization method.
• Integrating the algorithm with object scanning or a point cloud database to increase the range of possible objects to be grasped.
• Creating an interactive graphical user interface that responds to the operator grasping and provides better information on the improvement of the grasp.