1. Introduction
In recent years, robots, and specifically robotic grippers and arms, have been integrated into a wide variety of industries, working side by side with humans or replacing human activity entirely. Tele-operated systems are robotic systems, operated and controlled by a human operator from a long distance, designed to fulfill specific tasks, such as grasping an object.
The act of grasping an object with a tele-operated gripper is a challenging task, mainly due to the lack of sensory information our body receives while touching an object. One of the ways to compensate for the absence of sensory feedback is by assisting the human operator with choosing a stable grasp that would ensure firm handling of the object.
Our research goal is to assist operators using tele-operated systems in grasping different objects. The goal is achieved by suggesting a stable grasp solution, which is provided by an algorithm we created. The algorithm’s purpose is to find the optimal force-closure grasp for a specific object.
To prove that our proposed solution indeed assists the operator and improves his grasp, we tested the operator’s grasp performance while mimicking our proposed solution against an unassisted case.
The main contribution of this paper is the proposed algorithm, which can be implemented with a vast number of robotic grippers and applied to many objects.
The first researches on grasping theory began by classifying human grasps and creating a human hand taxonomy.
Cutkosky and Howe [
1] were among the first to differentiate between two types of grasps: power grasps that rely on security and stability, and precision grasps that rely on dexterity and sensitivity. Feix et al. [
2] extended that work and classified 33 different grasps into various subgroups. Their classification divided grasps into power, intermediate and precision grasps, the type of opposition (pad, palm or side opposition), and the index of the “virtual finger” used [
3,
4].
Due to the complex nature of the human hand, consisting of an overall 30 different degrees of freedom, mimicking the dynamics of the hand with robots is a complex task, so researchers turned to a simpler description of the complex hand movement, based on synergies, as described in [
5,
6,
7,
8]. The advantage of using hand postural synergies can be seen in the work of Palli et al. [
9], who reconstructed several synergies based on measurements performed on humans grasping different objects, in order to use them in robotic movements.
In his paper, Bicchi [
10] summarized and investigated the properties of form-closure and force-closure and showed the need for a quantitative quality measure of the force-closure property, so as to choose between different grasps. Suarez, Roa and Cornella [
11] reviewed many quality measures, based on the geometry of the grasp or limitations on the contact forces, whereas Kim et al. [
12] focused on dimensionless measures. Borst, Fischer and Hirzinger [
13] suggested a way to implant the task wrench space of the problem to create a more uniform quality measure, and Ohev-Zion and Shapiro defined a grasp quality measure based on the stability of compliant contacts [
14,
15].There have been various attempts to develop algorithms to find an optimal grasp. Ficuciello et al. [
9] developed an algorithm based on neural network and reinforcement learning to find the optimal path to grasp an object with a robotic arm based on some quality criterions. Lenz, Lee and Saxena [
16] found a solution based on deep learning and visual scanning of the objects to be grasped. Sinatov and Shapiro [
17,
18] tried to deduce the optimal common grasp that would fit a number of different objects. Probably the most common of the different quality measures is the “minimal inscribed ball in the grasp wrench space” criterion as proposed by Kirkpatrick et al. [
19] and Ferrari and Canny [
20].
Grasping an object with a tele-operated system requires communication between a “master” operator and a “slave” robot, based on a varied level of control and information provided. Various uses of system like this can be examined from different angles: Ni et al. [
21] predicted the movement of an object being manipulated with a tele-operated manipulator, and corrected the prediction in the virtual reality based on data from the “slave” unit. Bernardino et al. [
22] used a tele-operated arm and gripper to mimic humans in creating precision grasps, thus building a set of synergies to the gripper. An ability to control the “slave” gripper using visual input command from the human “master” was introduced by Li et al. [
23] with the help of a deep neural network. Nisky et al. [
24,
25,
26,
27] presented differences in the performances of novice and expert surgeons when using a tele-operated system to preform surgical cuts.
In this study we tried to create an assisting algorithm for tele-operated gripper users, advising them how to grasp an object and checking if their grasp was improved in relation to the unassisted case. Although the algorithm is applicable in generic robotic grasping, the purpose of the research is to advise the operator on grasping, and not force a specific solution. The suggested grasp was calculated through an optimal grasp searching algorithm.
In
Section 2 we will analyze the calculations required for finding an optimal grasp, and in
Section 3 we elaborate about our searching algorithm.
Section 4 deals with the series of tests we conducted to examine the improvement of the users grasping performance, and in
Section 5 a discussion on the results is presented.
2. Grasp Quality Computation
2.1. General Grasp Formulation
The interaction between a given object and the robotic gripper in
space is analyzed through six-dimensional grasp wrench space. The object
’s center of mass is located at
in relation to the world frame coordinates and rotates around three different axes, defined by
, resulting in a total of six degrees of freedom. For our purposes we define a gripper reference frame with the origin at the gripper’s wrist, and an object reference frame originating at the center of mass. These coordinate frames are noted by the indexes
g and
o, respectively. We denote the contact points between the body and the gripper as
, where
is the object’s surface and is given in the object’s coordinates. We could also define the contact points in relation to the gripper, as seen in Equation (
1). The distance between the object origin and the gripper origin is donated by
.
To calculate the grasp wrench space we need the contact forces between the gripper and the object. Using Coulomb’s model of friction, our frictional force at the contact point is bounded by a cone centered around the normal to the contact point
. Thus, we define the
Friction Cone in Equation (
2).
in Equation (
2),
and
are the normal and tangential force components of the ith contact on the object, and
is the static friction coefficient. The normal to surface
points inwards into
, so we demand that the normal component of the force would be definite and positive, meaning the finger pushes the object and cannot pull. Describing the forces inside the friction cone with an approximated linear combination results in the
Linearized Friction Cone described in Equation (
3).
The vectors represent the base vectors of the polytope, while is the total contact force at contact point .
Lastly, we define the wrenches exerted on the object in Equation (
4).
The Wrench is a six-dimensional vector, containing information regarding both the force at a contact point on the object
and the moment exerted by this force in relation to the center of mass. The parameter
represents a relative length that converts the grasp wrench space to uniform units. Using the above
Linearized Friction Cone, we define the primitive wrenches at point i, as described in Equation (
5).
2.2. Stable Grasps
Before examining the stability of grasps, an important assumption needs to be taken concerning the rigidness of the object. Under certain loads a soft object could deform and the wrenches acted on it would change, and on the other hand, when the object is extremely hard the grasp will become unstable due to massive loads (a phenomena called “coin snapping”). In the following physical treatment we assume rigid objects with small loads.
We illustrate the stability of our grasp with the force-closure property. A grasp is said to be a force-closure if we can counterbalance any external loads and torques exerted on the body using the contact forces. This way the body can be maintained in static equilibrium.
The Grasp Wrench Space (GWS) is defined as the set containing all the primitive wrenches in a given grasp in Equation (
6).
In our case the conditions for a grasp to be a force-closure are [
28]:
The origin of the wrench space is in the interior of the convex hull of the GWS. This is a necessary condition for any grasp to be a force-closure.
Using a positive linear combination of the wrenches, the external wrench on the body (due to external force) is counterbalanced. In practice, it means we sustain static equilibrium.
The first condition can be satisfied using hypergeometry. Because the GWS is a
shape, The facets of its convex hull rest on five-dimensional hyperplanes. We describe these facets by the hyperplane equation in Equation (
7).
Each facet is made up of six vertices (or six wrench vectors) of
CH(GWS).
is the normal vector to
, b is the offset from the origin, and x describes the coordinates of wrenches on the facet. We wish to calculate the distance between the origin to a hyperplane using the formula in Equation (
8).
To check that the origin lies in the same half-space divided by the hyperplanes of
we calculate the signed distance between each facet and a known inside point, and then compare their signed distances. Let
r be a point in space that is the “center of mass” to the primitive wrenches of the grasp wrench space, as described in Equation (
9).
We check the signed distance using the following formula for both
and the origin in Equation (
10).
If the equality is correct, then both the origin and lie in the same half-space divided by the hyperplane. To satisfy our first condition for force-closure we need to check the above equality for each facet in the convex hull. Even if there is only one facet where our origin is on the wrong side of the hyperplane and our grasp is not a force-closure.
The second condition is easily satisfied by Equation (
11):
The coefficients of our linear combination are bounded. The lower bound satisfies our demand for the fingers to push the object and not pull it. The upper bounds comes from the physical realization that our gripper is limited by the amount of grasp force it can exert. Consequently, the grasp wrench set is bounded by a maximal convex hull. A practical solution to check the existence of the second condition is to assume a new, shifted origin of the grasp wrench space. If we define the origin as and check again that for each facet we can satisfy the second condition.
If both conditions are satisfied, we claim that object is firmly secured.
2.3. Grasp Quality Measure
Many grasps can be used to reach force-closure on the body, and so we wish to find the best grasps. For this purpose, we define the grasp quality measure.
The grasp quality measure is calculated based on the properties of the grasps, and it rates a specific grasp relatively to other grasps. Using the process described by Roa and Suarez [
11] and a similar concept in [
29], we found the quality measures that best fit for our purposes. The connection between the robot motor torques and the end-point force (fingertip force) is described by the Jacobian matrix
. Our gripper has several fingers, so we describe the connection between all the motor’s torques and all end-effectors’ forces with the hand Jacobian matrix
, and the total result is presented in Equation (
12).
The vector
is given by the xyz components of all the finger forces we described earlier, namely
equally, and the connection between the motor velocities
and end-effector velocity
is given in Equation (
13).
The connection between the forces acting at the contact points between the gripper and the object, to the net wrench acting on the object, is given by the grasp matrix G presented in Equation (
14).
Additionally, the connection between the fingers’ velocities and the object velocity is given in Equation (
15).
From these definitions we receive the connections in Equation (
16).
The matrix H is called the hand object Jacobian, and its dimensions are , where is the dimension of the grasp wrench space, n is the number of contact points and m is the number of motors in each finger.
Under the assumption that a boundary to the size of torques exists and is a function of the power
P, we write Equation (
17).
Substitution of
with
results in limitation in the wrench space, as described in Equation (
18).
The above transformation matrix transforms a sphere in
torque space with radius
to an ellipsoid in
wrench space. The same process can be applied to object
velocity space resulting in the connection described in Equation (
19).
These two transformations have the same eigenvalues and same eigenvectors but in inverse proportion; when we apply the maximal wrenches on the body, we get the minimal velocity and vice versa. While the primitive wrenches are not bound by the ellipsoid we have obtained, and consequently our convex hull, might intersect the boundaries of the ellipsoid, the net wrench exerted on the body is bounded by the ellipsoid, and that is because we wish to maintain static equilibrium in the force-closure grasp.
Following the above conclusions, we wish to define a grasp quality measure for two cases. The definitions from [
1,
2] will be used to define these types of grasps:
Power grasp—the goal is to hold the object as firmly as possible, preventing movement, resisting the biggest external loads and exerting the maximal contact forces.
Precision grasp—the goal is to control the movement of the body as much as we can, with little to no force exerted on the body.
In the case of power grasp, we wish to increase the smallest possible wrench that can be resisted (the minor axis of the wrench ellipsoid), and so we use the most common grasp quality measure, the inscribed sphere method, presented in Equation (
20).
where
is the radius of the sphere and q is the received score for the grasp. The maximal radius of the inscribed hypersphere that touches the closest facet represents the biggest external wrench that can be applied on
in any direction.
In the case of the precision grasp, we wish to increase the minimal range of movement with little to no force (the minor axis of the velocity ellipsoid). Because the velocity ellipsoid and the wrench ellipsoid are connected to each other, and have the same axes with inverse proportions, in the wrench space we wish to decrease the maximal wrench we can resist (the major axis of the wrench ellipsoid) and the minimal radius of the circumscribed hypersphere of the convex hull. Thus, we find a simple quality measure for the precision grasp in the form of the minimal between the maximal wrenches, shown in Equation (
21).
3. Optimal Grasp Search Algorithm
In order to suggest an optimal grasp for each of the objects we wish to grasp with the tele-operated system, an algorithm composed of two parts was created. Our proposed method to find the optimal grasp of an object consists in creating a large number of force-closure grasps and optimizing the optimal solution by genetic optimization.
The first part of the algorithm includes an offline part, compromised of a series of simulations using MATLAB. The simulations’ purpose is to find the optimal grasp, as described thoroughly in
Section 3.1. Each simulation includes randomly chosen grasps of the object’s surface, testing if they satisfy the two conditions for force-closure, ranking of the grasps by the chosen quality measure, and lastly optimizing the grasp population using a genetic algorithm.
The second part includes computing the grasp quality measure while the operator tries to grasp the object with the gripper. The analysis is executed in the same manner that the simulation in the first part was done, and then comparison between the two received values (optimal and test) is carried out.
While grasping, the online part of the algorithm receives sensory information of the active forces at the contact points (the fingertips of the gripper) and computes their position in space. Using this information, improvement to the grasp is suggested in order to achieve a better grasp, based on the defined grasp quality measure.
3.1. Finding the Optimal Grasp
The algorithm of the optimal grasp is given in
Figure 1.
The algorithm receives the entire data on the object and other relevant data (finger force assumption, number of contact points, external forces if exists).
First, we randomly chose a specified number of contact points from surface
and checked for force-closure. We used the chosen set of contact points
to build the grasp wrench space and pre-defined finger force directed normally to the surface
at the points
. Once we computed our grasp wrench space, we could check for force-closure using Algorithm 1.
Algorithm 1 FC grasp. |
Input
Output
Initialize FC = true, stating that we have a force-closure grasp. Define, as the origin. Create the 6D convex hull of G, . Define , a point inside K as the average of all primitive wrenches. for number of hyper facets in K Find the points that lie on the facet. Organize them in A. Compute the signed distance between the facet to ,. Put both results in
end for for length of s if the sign of both results in s is not equal then FC = false break end if
end for repeat from 5 with
|
A success in the force-closure test will give us a quality measure to rank our current grasp. The quality measure is computed as described in
Section 2 and will be noted as
. A failure in the test will result in the algorithm searching for another suitable grasp, choosing another random points in
.
At this stage a set of specific grasps that maintain force-closure on object is sorted in a table called a Grasp Generation List, or GGL for short. The sorting is based on the quality measure , where and the index represents the number of the row (in the case of precision grasp, is ordered in ascending order). Each row includes information about the contact points’ locations and contact forces.
Using a genetic algorithm, we optimize the table to receive the optimal grasp for the object. The algorithm is inspired by the evolution of species in biology, where desirable genes are inherited by the next generation.
For each generation we define a set of parent grasps, , that contain the information from the best two grasps from the GGL table; these are the first two rows. The “genome” of the grasp contains the set of contact points of the grasp, to . For each child grasp we mix the “genomes” of to create a new genome strain. The mixing occurs at a random cross section and is called a “crossover”.
The next phase would be the “mutation” phase. A mutation can occur in a genetic sequence describing our new child grasp, with low probability. The mutation is performed by switching one of our contact points with a randomly selected point from the surface .
Once we created a new grasp we check if it satisfies a force-closure on the object and rank it among the grasps in GGL. After creating a child grasp, and reorganizing the table, we move forward to the next generation of grasps, using the currant parent grasps. By carrying out this iterative process we improve the received grasps. After 40 generations we find our optimal grasp, with the quality measure noted as .
The initial population of the genetic algorithm is fixed in size, in our case six grasps at the start of each generation. after the “crossover” and “mutation” phases we add two new child grasps to the table (assuming they sustain force-closure), and delete the two least fitting grasps from the population. The mutation probability is calculated using the condition in Equation (
22), and results are calculated with probability of
. Using trial and error we determined that 40 generations suffice for an improvement between the initial best grasp and final best grasp.
3.2. Comparing the Received Grasp and the Optimal Solution
We defined a value function to compare between the grasp achieved by the operator and the optimal grasp we found. The value functions are as follows. For the power grasp case we have the value function in Equation (
22).
For the precision grasp case we have the value function in Equation (
23).
To compute the received grasp, we track and send commands to the gripper, using a tele-operated apparatus (in our case, a control glove). Once the operator achieved the desired grasp with the robotic gripper, we compute the grasp quality measure as described in
Section 2. The computation demands that we collect data from the gripper, mainly the forces at the contact points (the finger tips) and the location of the contact points in the object’s coordinate frame. The forces acting at the fingertips are extracted from pressure sensors located at the contact points and directed at the relevant normal vector from the contact surface.
Using the contact points’ positions and forces, the primitive wrenches are computed and the GWS matrix is created. The creation of the GWS convex hull is done using our implementation of the QuickHull algorithm, as described by Barber and Dobkin [
30]. Using the convex hull we easily check for our quality measure, either for power grasp or precision grasp.
5. Results Analysis and Discussion
A comparison between the performance of humans with the system against the hypothetical optimal quality measure was performed, based on 14 tested individuals and three grasped objects.
5.1. Optimal Algorithm Results
To achieve our optimal results for each object according to the classifications defined (“power grasp” vs. “precision grasp”), we required a list of relevant parameters for our computation.
Our algorithm received a point cloud collection of the object’s surface
, a theoretical maximal normal force at each contact point
, estimated to be a different value for each type of grasp (
[N] for power grasps and
[N] for precision grasps), and lastly the coefficient of static friction, based on different values taken from the literature. Due to our uncertainty regarding the estimation of the coefficient of static friction and based on the values we encountered, we picked two values of
in order to be able to also check a stricter case of the grasp presented in
Table 2.
Our algorithm applies concepts from genetic algorithms. While running many simulations with our chosen we received a large diversity in our optimal quality measure for each of the objects, which suggests we have not found the true optimal grasp for the object.
A number of explanations for this result are suggested. One of them is the disadvantage of the genetic algorithms due to the possibility to converge to local optimum and not the global optimal solution of the problem [
32]. Another explanation is that the set of solutions is not constrained by the kinematics of the gripper, resulting in nonphysical solutions (two contact points too far apart). Lastly, a crossover of two good grasps could still result in a worse grasp, thus increasing the diversity of results.
We conclude that the genetic algorithm improves the score of the fitness function (meaning that our final proposed grasp is better than our starting grasp) but different iterations of the same algorithm will not necessarily converge to the same solution. Under this assumption we wish to find a good score for our quality measure, understanding that it represents a local maximum of the grasp quality measure function and not a global maxima.
For each object we simulated iterations of optimal grasps, testing two different coefficients of static friction, where in each one of these we scanned 300–400 grasps (both force-closure and non force-closure), and then chose the highest power grasp or lowest precision grasp received score. Our “optimal” grasp quality measure was better than grasps, and therefore we are confident in our results.
Table 3 shows our results and their diversity. The table displays the coefficient of static friction, the best quality measure, the average quality measure and the standard deviation of the quality measures, where the sample size is
.
In
Figure 7 we show some of the visual output provided to the test subjects about the different optimal grasps. the suggested points of contact are marked by the red markers on the objects’ surfaces.
5.2. Test Results
Our test results are based on the performances of 14 participants.
Figure 8 presents the age distribution in our test population. Detailed test results are provided in the
Appendix A.
In the first phase, participants were introduced to operating the system through a training session. After that, each one of them was asked to grasp the object in any manner they should choose, where each participant was asked to grasp a number of times to increase the statistics on each object. Outliers were excluded from the results.
Table 4 shows their performance without the assistance of the algorithm.
In the second phase participants were asked to repeat the grasping of the objects, but now trying to mimic the optimal grasps presented in the previous section.
Table 5 shows their performance with the assistance of the algorithm.
Figure 9 presents the distribution of the grasp quality measures, divided into power and precision grasps, and separated for each object.
In the case of power grasps, the higher the score obtained, the better the grasp. The results are divided in relation to the optimal grasp quality measures we found in our calculations.
In the case of precision grasps (seen at
Figure 10), the lower the score, the better the grasp. we should note that the number of grasps examined were different in each case.
5.3. Discussion
Based on the results we obtained, we detected a general improvement in the grasping performance of the human operators.
Table 6 compares the performance before using the algorithm and after using it.
Using our algorithm, we saw a general improvement in performance of power grasping, by 42%–125%. However, for the case of precision grasping only negligible changes in performance were detected, ranging between −0.04% and 0.01% performance change (presented in
Table 6).
Individual changes in performance are presented in
Figure 11.
Some of the participants improved drastically while others did not. Comparing the performance against the optimal grasp obtained in
Section 5.1, we see that while some of the test subjects reached a better score than the algorithm, more than half of the subjects improved their performance by more than 50% in relation to the score obtained by the algorithm.
While a general improvement in the power grasp performances was noticed, we did not see major changes in the precision grasps, even for objects that are delicate to handle (tweezers). In practice, this means that the distance from the origin of the grasp wrench space to the farthest vertex of the convex hull remained the same. A possible explanation is that there will always be one contact point where the force exerted has to be of a minimal size (to both sustain force-closure and counter the self weight of the object).
Looking at the results established the hypothesis that human operator performance is influenced by the following parameters:
Age—older subjects had lower initial scores and lower improvement measures in comparison to younger subjects. It is possible that teaching older people how to grasp different objects is therefore a harder task than teaching young subjects.
Working experience—unlike our initial hypothesis that experienced subjects that have many years of training with grasping tools in uncomfortable environments would preform better, we could not discern their performance in relation to other subjects’ performances. Our conclusion is that working experience has no influence on performance with tele-operated devices.
Intuition with tele-operated system—subjects who had previous experience with the specific experimental system or with similar gear (playing video games with controllers) showed better results.
Object/type of grasp—from the results we detected that the best performances were obtained with the screwdriver, a convenient tool, and not with the heavy hammer or the delicate tweezers.
To conclude our discussion, we would wish to compare our algorithm with a few other algorithms presented in this paper [
9,
16,
17,
18]. A comparison is presented in
Table 7.
The algorithm proposed in this paper is intended to grasp specific pre-defined objects, and thus does not require extensive initial information (although it could be implanted with an object database), in comparrison to algorithms using a large database of objects. It is not gripper-specific due to the non-relevance of the gripper’s geometry, which is relevant in other algorithms, and can be implemented with an infinite number of grippers. Another advantage is the flexibility of the algorithm in changing the force parameters (maximal contact force, coefficient of friction), allowing to simulate different grasp conditions without the need to test the situation in reality, unlike algorithms that use geometrical features only.
6. Conclusions
In this work we created an assisting algorithm for grasping objects with a tele-operated robotic gripper. The algorithm proposed in this research can be used with a vast number of grippers, and simulate grasping of different objects with different grasping scenarios (low friction, different grasping powers).
For a given set of objects, with a known surface and a predefined coefficient of static friction, our algorithm attempted different type of grasps, and found an optimal grasp based on quality measures we defined. Optimal grasps were calculated for each object, for both the power and precision types of grasps, and with two different coefficients of static friction (likely case and severe case). For each grasp we created a visual output for our test subjects. The grasp results of the algorithm, while not being the true optimal grasp for a given object, were ranked higher than random grasps of an object, which makes the genetic algorithm a valid optimization method for grasping problems.
In a series of tests we observed the performance of different operators using a tele-operated system. Each subject was asked to grasp the objects with and without the output from our algorithm, and we compared the results.
For the case of power grasping we have seen an average improvement of at least 40%. Most of the participants improved their performance, where the younger subjects improved better than most of the older subjects. While some of the subjects were able to perform better than the optimal grasp quality measure calculated, we deduced that following the visual information of the optimal grasp improved the performance of most of the test subjects.
For the case of precision grasping no noticeable changes in performance for any of the subjects were observed. Our assessment is that there is a minimal threshold of force exerted at the contact point on an object that stays the same for all grasps, and thus the circumscribing sphere of the GWS has minor changes.
The study presented in this paper does have its limitations. The algorithm is not applicable to grasping of soft objects or grasping with contact areas, due to the physical model implemented. Initial assumptions about the forces and friction have to be assumed in order to receive results. Additionally, while not part of this study, translation of the visual output provided by the algorithm into visual instructions in augmented/virtual reality could drastically increase the quality of assistance.
To conclude, the tests’ results show that the proposed algorithm indeed assists the operator in securing an object with tele-operated devices, in spite of the challenges presented by the problem.