A Systematic Approach to Evaluating and Benchmarking Robotic Hands—The FFP Index

Abstract

The evaluation of robotic hands is a subjectively biased, complex process. The fields pertaining to robotic hands are human-centric in nature, making human hands a good standard for benchmark comparisons of robotic hands. To achieve this, we propose a new evaluation index, where we evaluate robotic hands on three fronts: their form, features and performance. An evaluation on how anthropomorphic robotic hands are in basic mobility, and appearance constitutes the “Form”, while features that can be read, changed and actuated for effective control of robotic hands constitutes the “Features”. We derived these key features from an extensive analysis of robotic hands in literature. Finally, the robotic hands carry out a series of tasks that evaluate their “Performance”. An individual score for each category is drawn and we carry out a three-pronged analysis. We also propose an additional feature in the form of price to provide context when analysing multiple hands.

1. Introduction

The analysis of end effectors or robotic hands has been a problem as old as robotic hands themselves. Every robotic hand ever built has been done so for a specific purpose. In most cases, the tests have often been tailored to facilitate and highlight the end user experience. This has led to the lack of a balanced vision when it comes to the design and evaluation of robotic hands. There is a distinct lack of a standard set of tests that can evaluate robotic hands in a holistic fashion. Aspects of a robotic hand’s form, its features, the controllable parameters of its sensor and actuators and their capabilities all play a key role in the evaluation of a robotic hand. The performance of a robotic hand is a sum of all the aforementioned characteristics and is an important part in the analysis of a robotic hand, but it is not the only one.

A lot of different benchmarks and evaluation indices, mostly focused on grasping capabilities of the hand, based on the work done by Cutkosky [1] and Feix et al. [2], were proposed. Most tests focus on the dexterity of robotic hands under the context of robust grasping.

Furthermore, there exist numerous methods that evaluate various aspects of robotic hands. One such famous method is in the benchmarking of anthropomorphism and dexterity in robotic hands proposed by Biagiotti et al. [3] in 2004, where they provide separate evaluation indices for both anthropomorphism and dexterity. There also exists a number of practical assessment tests for human hands such as the The Southampton Hand Assessment Procedure (SHAP) evaluation [4].

1.1. Benchmarks in Literature

An exhaustive list of all the benchmarking tests in robotic hands is explored in the paper by Quispe et al. [5]. Some of the key evaluation indices that inspired the proposed index are discussed below.

1.1.1. Anthropomorphism Index

The work from Biagiotti et al. [3] tries to quantify anthropomorphism in robotic hands and tries to answer if it is better to call a hand anthropomorphic if it fits the form better without replicating its functions or vice versa.

It takes into account the kinematics of the hand, its contact surfaces and size, with a weightage for each as shown in Figure 1. It breaks down each of these three categories into sub-components with a weight for each.

1.1.2. Dexterity Index

A separate index in the same paper defines dexterity as the amount of useful work that can be done with a presented hand. It gives weights to each kind of prehensile, non-prehensile and manipulation tasks that can be done by the robot. Since these tasks are a combination of the morphological features, the sensors, control algorithms, etc., it provides a good idea about what features make a hand “dexterous”.

1.1.3. Kapandji Test

Differently from the previously mentioned types of evaluation, the Kapandji test or Kapandji score was originally proposed by Ibrahim A. Kapandji [6] in 1985 as a tool for assessing the opposition of the thumb based on where the thumb tip (in red) can touch various parts of the hand (in blue) as shown in Figure 2. It is a self-contained test which makes use of the human hand’s natural mobility to measure thumb dexterity. This test has been used in various robot hands [7,8] as a way of evaluating the dexterity of the thumb.

1.2. Summary on Benchmarks

The degree of “hand anthropomorphism” is defined by Biagiotti et al. in [3], as a combination of the form, features and the control paradigms incorporated into a hand. However, they are considered intangible from one another, and they are still largely quantifiable parameters with a few key features.

As explored in the paper by Quispe et al. [5], most of the evaluation indices are strictly based on the performance of robotic hands. Individual tests, such as the Kapandji test, are very useful in considering the dexterity of the thumb, but do not contribute to a holistic assessment of the hand. One of the best evaluation methods for robotic hands was discussed in [3] but even that evaluation is not easily quantifiable and is broken up in parts. In the case of prosthetic hands, a simpler method was put forth by the SHAP test [4], wherein the evaluation is done by performing specific tasks which are then validated via an online tool that rates performance.

Considering all these validation tools, a new form of evaluation index is proposed that is performance oriented, while also considering the hardware and control capabilities of the hand.

This paper is divided into the following sections. In Section 2, a review of the state of the art in robotic hands is discussed and a classification of robotic hands in general is defined. In Section 3, the proposed benchmarking index is introduced and key features in successful robotic hands are identified. In Section 4, Section 5 and Section 6, the different parameters involved in the evaluation are discussed in detail. An evaluation and analysis of the proposed index is then done using the iCub hand as an example in Section 7 and Section 8, the final remarks and conclusions are given in Section 9.

2. Robotic Hands

2.1. Survey of the State of the Art

To extract key features in robotic hands, they are first classified into major categories. Classification in hands can depend on a number of parameters, but since most hands are built with an end user need in mind, this study classifies hands depending on this criterion. Robotic hands can be built to be sold as part of a package or standalone systems, either commercially or for research. They also feature predominantly in limb rehabilitation, prosthetics or simply to be functional prototypes to exhibit advances in technology. Thus, depending on the purpose they are built for, they are categorized into the following:

• Robotic research,
• Commercial,
• Prosthetic.

Each type will be discussed briefly in the following sections. For general reviews the interested reader is invited to refer to [9] and other similar review papers dedicated to exploring this. Performing an exhaustive study of all robotic hands in literature is a near impossible task and is not the scope of this study. The current article therefore focuses on the technical characteristics of the most common commercial and prosthetic hands. For research hands, we chose to include the ones currently most used for research experiments as well as the most prominent (in terms of article citations).

2.1.1. Research

Research hands are typically prototypes that are developed to realize novel concepts or part of a specific research objective. It often focuses on a single feature and does not need to adhere to the “commercial viability” of a product. It can be further classified into humanoid hands (which are part of a humanoid robot) or standalone hands.

Standalone hands: They are typically designed to be multi-purpose hands. The technology used in these hands are usually generic enough to be tested across various platforms. Hands like the Shadow hand [10], the Deutschen Zentrums für Luft- und Raumfahrt (DLR), HIT II hand [11], the Karlsruhe University (KU) Hybrid hand [12] and the University of Bologna (UB) hands [13] are examples of such hands.

Humanoid robot hands: These are developed as part of a humanoid robot. They are specific in their needs; for example, they are in accordance with the needs of the robot. Some examples of humanoid robot hands include that of the Twendy-One robot [14], the RoboRay hand [15], the iCub hand [16] (shown in Figure 3) and the R1 hand [17].

2.1.2. Commercial

They are hands that can be standalone, prosthetic or part of a humanoid robot. However, they all have similar objectives, which are: easy to produce, robust and cost-effective to manufacture. They usually have a good weight to payload ratio and employ relatively simplistic manufacturing practices. They tend to opt for commercial off the shelf (COTS) components for the sake of production in big numbers. Some examples of commercial robot hands include The Shadow Hand [10], the UNIPI hand [18], etc.

2.1.3. Prosthetic

Prosthetic hands are aimed at restoring partial or complete mobility to people who have lost hand function either through accident, paralysis, amputation or other means. Although prosthetic hands such as [18,19,20] can technically fall under both the commercial or research section, we still keep them as a separate category because they are fundamentally different in their use case and also since they have humans in the loop.

Prosthetic hands usually share a list of desired features with robotic research hands such as having a combination of high functionality, durability, and affordability whilst being lightweight and highly anthropomorphic in nature. Self-contained robotic hands are preferred in prosthetics since remote actuation is most often not an option and depends highly on the level of amputation.

All the aforementioned types of hands can be either self-contained (comprising all required actuators and electronics in a single structure) or remotely actuated (the electronics or actuators or transmission systems are all placed outside the main structure of the hand).

3. FFP Evaluation Index

From the previous sections, it can be seen that the usefulness and versatility of a robotic end-effector depend not only on the diversity of grasps it can accomplish but also in its form and complexity of the control methods required to achieve them. All the robotic hands that were studied usually had specific features that were required due to the user needs of the platform under investigation. However, an objective benchmark is necessary to provide guidelines which aid in making a particular end-effector platform better while also acting as a guideline for best practices in the design of these hands.

Another important aspect in hand evaluations is their use for human-robot interaction (HRI). Indeed, it is of the utmost importance to incorporate communication and interaction capabilities into hands to be used in HRI tasks.

Also of importance is the type and amount of sensors that need to be incorporated into the hand. Hands for humanoid robots and prosthetic hands differ in terms of sensor requirements. Hands for prosthetics tend to rely on myoelectric sensors that are, so far, the most natural interface for user commands. While in humanoid robotics, the use of force-feedback and tactile sensing is increasing in importance as ways of handling objects better, once they are grasped.

3.1. Proposal

This article proposes the Form-Features-Performance index (FFP index), wherein different aspects of the hand can be evaluated and compared. As shown in Figure 4, the FFP index is composed of multiple sub-categories.

The three main categories in this evaluation are weighted equally as a first step. This gives an idea of how a given hand performs in each of the dedicated categories of Form, Function and its Performance, and provides a relative comparison of each category to the other two.

This relative comparison is achieved by assigning dedicated weights to the sub-categories in each of the three categories as and how important it is deemed by the user. By doing this, an estimate of the amount of importance that goes into the design of each aspect of the hand can be determined. Effort has been made to identify the various aspects of the perfect human hand, and to incorporate it into this index. The main motivation behind this type of evaluation is to act as a good guideline for building effective hands and to draw a common denominator across all the robotic hands that are available in the market. By drawing a common baseline, the comparison of different types of robotic hands is made easier.

3.2. Identifying the Key Parameters

The designers of robotic end-effectors typically need to address several performance trade-offs, like limitations on size and weight versus performance. A general comparison of some of the key trade-offs needs to be done to better understand how every key parameter influences every other parameter in the study of end-effectors. A list of the typical trade-offs encountered in hand design processes is as follows:

• Number of actuators vs. Number of joints,
• Hand weight vs. Payload,
• Hand weight vs. number of actuators,
• Number of sensor inputs to number of actuators.

The following sections are based on an evaluation of the technical characteristics of research, commercial and prosthetic robotic hands. As mentioned in Section 2.1, the hands mentioned in the following analyses were selected with heuristic criteria. We recognize that these inclusion criteria might introduce biases when used for quantitative evaluations, but we nevertheless consider the designs mentioned to cover sufficiently well the state of the art of the field.

3.2.1. Number of Actuators to Number of Joints

In recent years, a considerable amount of research efforts has been devoted to the development of underactuated hands [21]. One of the biggest problems in underactuation has traditionally been the one of precise control. This issue has been addressed in recent years with the implementation of sensors which provide information needed for robust control.

Looking into literature, it is seen that the higher the number of actuated degrees of freedom (DOF), the simpler it is to control effectively. This, in turn, also makes the hands more complicated to design. Another note of interest is that most of the grasps as defined by [22] can, in theory, be performed by a single grasp action by a hand with 2 DOF. While a more dexterous hand is needed to go beyond these functions, this gives an interesting insight on how futile it potentially is to evaluate the effectiveness of a robotic hand just on a set of grasping criteria. It can be seen from the Figure 5 that completely actuated or over-actuated hands tend to be research hands, such as the DLR hand and the UB-IV hand. Some exceptions exist, like the remotely actuated Shadow hand, which is a commercial product. Prosthetic hands require the system to be lightweight, and integrating more actuators tends to increase the complexity of the hand and its weight greatly.

From Figure 5, we see the robotic hands that lie on the line of commercial hands usually have fewer actuators than the number of joints that it controls. Prosthetic hands tend to be on the lower side in the number of degrees of freedom and actuation being usually underactuated with a single actuator. This approach restores primitive prehensile functions (e.g., power grasps) while maintaining a high degree of reliability and while keeping the overall weight as low as possible. Furthermore, having a human-in-the-loop allows for compensating and adjusting for possible sensing or control shortcomings. Research hands usually have 10 to 20 joints. A great variety of actuation methods, distribution mechanisms, and degrees of actuation (fully actuated, over-actuated and underactuated mechanisms) have been proposed in the literature. Research hands are generally equipped with rich sensor suites. Because of their compactness and the low amount of available volume, they are usually remotely actuated. In this regard, underactuation allows for alleviating volume allocation issues at the design stage; indeed, this solution is adopted quite often.

Novel distribution and transmission mechanisms as used in [7,23] can be employed to distribute the actuation forces and to actuate the joints in a pre-planned fashion, thus reducing the need to over-actuate a given system.

3.2.2. Hand Weight to the Payload

The hand weight is greatly influenced by factors such as the number of actuators, number of joints, transmission system, materials used, etc. Since the weight can depend on a number of factors, a good metric could be the weight in relation to payload. The human hand can exhert significant loads mostly thanks to its tightly integrated and powerful musculo-skeletal structure. A direct comparison of robotic systems with their human counterparts, in this case, would be difficult.

A better baseline for comparison would be to consider the hands that have been explored in literature and to calculate the median range of the payload-to-weight ratio; this value ranges approximately from 1 to 1.5 (see Figure 6). This implies that a robotic hand should be able to lift more than it weighs to be considered effective. This is indeed the case for most robotic hands considered in the Figure 6 cluster in the 0.5 kg to 2 kg range for both weight and payload.

It can be observed that the weight of prosthetic hands is not significantly different from their quoted payload. This is due to the fact that prosthetic hands need to be lightweight in order to be useful to the wearer since weight would lead to user fatigue. Moreover, in most cases, prosthetic hands are not remotely actuated.

Robotic research hands do not suffer from this limitation. They can be remotely actuated and hence allow for displacement of the actuators. Interestingly, even though they have a distinct advantage in this case, they still suffer from a poor payload-to-weight ratio. Limitations in actuation strategies, motor capabilities and transmission systems are some of the notable causes for this drawback. Another important factor is cost since the complexity and performance levels of all the aforementioned factors are limited by cost.

It can be seen that the payload-to-weight ratio for robotic hands tends to be slightly higher than the one of commercial or prosthetic hands; in these cases, the average payload is between 0.2 kg to 1 kg. The commercial hands which performed above the average were hands with fluidic actuation (without the pump weight taken into consideration). From these observations, it can be concluded that a typical hand payload-to-weight ratio falls between 1 and 1.5.

3.2.3. Hand Weight to the Number of Actuators

Another interesting analysis is the comparison of the hand weight to the number of actuators being employed. For a fair comparison, when we refer to weight, we include any type of actuation system, be it remotely actuated or self-contained (as mentioned in literature).

As can be observed in Figure 7, we can see that the robotic research hands are consistently heavier than their prosthetic counterparts; this is partly due to the fact that robotic research hands are remotely actuated and the weight of the transmission system increases the overall system weight. The hands with the lowest values on this scale are the commercial self-contained hands since they focus on cost reduction and ease of manufacturing.

It can be noted that the weight of the hand and the number of actuators play no significant role in the ratio. All self-contained and remotely actuated hands cluster together in the graph, which shows that the number of actuators in robotic hands plays no real role in determining the weight-payload ratio in the system. This could also be due to the layout of the transmission systems, the type of actuation, the distribution mechanism (in underactuated hands) and the type of sensors that are employed in the hand, to name a few.

3.2.4. Number of Sensor Inputs to Number of Actuators

Although a statistical analysis of this is harder since the exact number of sensor inputs is not completely defined for every robotic hand in literature, it would be interesting to consider the number of different types of sensors that are put into the robotic system. Although adding more actuators is not necessarily always better, providing a large amount of sensory data is almost always beneficial to improve the control of the robotic hand. This holds especially true in underactuated hands that are, in general, slightly under served because of the problems that arise in controlling them. Having enough sensory data to make the state of the robotic hand fully observable gives the advantage to underactuated systems over completely or over-actuated hands. This approach is also supported by recent advances in machine learning techniques and deep neural learning systems which require more inputs to compensate for minor disadvantages in the hardware system.

3.3. Key Features

In addition, rating a hand strictly by its anthropomorphic constraints seems primitive, since function precedes the form. Some robotic hands that perform exceptionally well for a dedicated purpose might fall low on the form scale and vice versa. Hence, an exhaustive evaluation of a robotic hands’ features and form should go along with its performance characteristics.

4. Form

Form, in this case, is defined as the level of anthropomorphism present in the robotic hands. Typically, it consists of five unique digits that aim at replicating the look of the human hand. It usually has an opposable thumb and has similar size and weight ratios to that of the human hand. Biagiotti et al. [3] define anthropomorphism in robotic hands as “the capability of a robotic end-effector to mimic the human hand, partly or totally, as far as shape, size, consistency, and general aspect (including colour, temperature, and so on) are considered".

Anthropomorphism is a key factor in prosthesis since subjects wearing it have a degree of comfort for the human form. In some cases, depending on race and demographics, it is not considered the most crucial factor when it comes to deciding for or against its use in prostheses according to the work done by Biddiss et al. [24]. According to the survey [24], the most prominent factor for prosthesis rejection was that users considered themselves as having more or less the same level of functionality without the prostheses or it was too heavy or hot or that the feedback obtained from it was limited or non-existent.

Only about 70% of the total users said that the look was even a factor, and the level to which it was a factor, was around 1 (on a scale from 0 to 3, 3 being most important). This shows anthropomorphism is not the highest on the list of requirements even in prosthetics, according to this study.

Research hands installed on humanoid robots are highly function-specific. Since the anthropomorphic shape is not always chosen in humanoid robots, it is considered very low on its importance scale and almost always is preceded by its functionality. This does not mean that the form doesn’t matter whatsoever. A detailed breakdown of all the weights for the “Form” factor in each of the hand analysed is given in Figure 8 where a no score is given as an empty index.

4.1. Opposable Thumb

In the book by Napier [25], thumb opposition is defined as “probably the single most crucial adaptation in our evolutionary history …” and that loss of thumb opposition could “… put back 60 million years in evolutionary terms …”. Thumb opposition is perhaps the most important movement of the human hand and is a major underlying factor when it comes to any kind of skilled actions performable by the hand.

As seen in literature, almost all hands have an opposable digit that takes the role of the thumb or its part in opposition. Human hands have a highly articulated thumb. An articulated thumb can facilitate apprehension tasks by moving out of its default opposable position, either through rotation, abduction/adduction or a combination of the two movements.

How the hand performs this opposition is unique for each robotic hand. The best score of 25% is assigned to the presence of an opposable, articulated thumb. The presence of a single position (no rotation, but opposable) thumb is assigned a score of 10%, and a manually articulated thumb which requires human intervention to lock its position is assigned a score of 15%, while the lack of a thumb results in zero rating.

4.2. Kinematics

Hand kinematics can be broken down to three key components in any hand, namely: mobility, stability and strength. Stability during loading without compromising its mobility is essential for the hand to achieve the various digital positions for activities. Mobility is usually defined by the Denavit-Hartenberg (DH) parameters, wherein the hands’ joint properties such as the DOF and its range of motion and its Degree of actuation (DOA), etc., all play a part. A very useful metric of DOF vs. DOA is even discussed in the “Features” section. However, the DOF is the key to define the mobility aspect in the anthropomorphic shape of a robotic hand. A hand incapable of producing a range of configurations cannot be strictly called anthropomorphic. Hence, there arises a need to define how much mobility in the robotic hand can relate to the anthropomorphism of the said hand.

Kinematics through Gestures

One solution would be to evaluate mobility in robotic hands by making it undergo a series of gestures. This lets the hand to be evaluated without having it constrained under payload restrictions. Having it undergo a range of gestures also provides the hand with a set of motion primitives that relates to the Human–Robot Interaction (HRI) aspects of the robotic hand. The other two functions of stability and strength are later tested in the “Performance” section of this paper.

4.3. Fingers

The finger refers to a type of digit, an organ of manipulation and sensation found in the hands of humans and other primates [26]. Fingers can be flexed or straightened at their respective joints placed between phalanges. They can also move side to side with respect to the centre of the hand; this movement is called abduction/adduction. Fingers in a human hand refer to four individual digits and a thumb. However, it need not necessarily be the case in a robot hand. For this evaluation, a weight of 25% is assigned to a robotic hand that has all five fingers and an appropriate number of phalanges that is classical for a human hand. The breakdown of the different aspects that influence the form of the finger is discussed below.

Joints per finger: The number of joints per finger is deemed extremely important since it provides the hand with conformance to shapes. It also acts as a contact surface for gripping and a support structure for grasps. A minimum of two joints is considered important in most cases for effective grasping.

A score of 10% is given if the hand has a minimum of two active joints (physically distinct); a score of zero is given otherwise.

Number of fingers: From literature, it can be seen that hands can perform very well with just two [27] or three [28] digits. Some of them even outperform some of the anthropomorphic robotic hands. The performance also highly depends on the control and overall achievable posture of the robotic hand. In essence, robotic hands need not necessarily be anthropomorphic, but an increased number of fingers (be it in underactuated or fully actuated hands) provides a larger grasping surface and increases conformity to shapes.

Keeping this in mind, a score of 10% is assigned for the overall evaluation score for five fingers (four fingers, one thumb), and 5% for three or four-fingered (minimum of two fingers, one thumb) end effectors and zero, otherwise.

Range of motion: The range of motion is to be similar to that of the human hand and it should have individually actuated joints at each finger. It is considered an acceptable range of motion if it has: MCP joint is 75–85${}^{°}$, PIP joint is 100–110${}^{°}$ and DIP joint is 65–70${}^{°}$ or more. Some of the prosthetic hands under study were evaluated to a score of zero since they have their distal and interphalangeal links fused.

4.4. Padding

The soft padding on the palm and fingers on the front side of the human hand is called the glabrous; it is tightly stretched with flexure lines in specific areas to accommodate folding and stretching. The glabrous is covered by papillary ridges or fingerprints, which provide the necessary friction and also act as a sensor to micro-vibrations. This is also one of the most difficult features to mimic in robotic hands.

Many robotic and prosthetic hands have imitated this kind of padding [29,30,31] to cite a few. This kind of soft-padding is considered important since it allows for conformity to objects and also gives rise to compliance. Friction surfaces also provide the hand with extra support during both prehensile and non-prehensile tasks.

Therefore, it is clear that the presence of compliant paddings and friction surfaces are very useful in providing a good grasp. A weight of 5% is assigned for the presence of soft padding and 5% to a dedicated friction surface such as a commercial gripper material, or high-density neoprene, to name a few. This adds up to 10% for this sub-category of the evaluation.

4.5. Palm Properties

The size, shape and functionality of the palm also play key roles in defining the anthropomorphism in robots.

4.5.1. Closed vs. Open Palm

Another feature which has been sparingly explored in the evaluation of robotic hands is the approach of the hand to a grasp. Human hands have an offset thumb making the hand more “open” in its approach. Traditional grippers and a number of hands in literature, however, have an opposing digit in the centre of its palm. This could also be defined as the classical gripper form versus the anthropomorphic/semi-anthropomorphic hand, a sample of each is shown in Figure 9.

The human hand tightly integrates with the motion of the rest of the body to perform a wide range of actions. Actions such as opening doors can be highly complex and require the coordinated motion of the entire torso. The same holds true for humanoid robots. Hence, the approach to handling objects and its planning is a lot easier when the thumb is offset or the palm has more space to approach the object and wrap around it.

The offset thumb also provides increased, effective contact surfaces for object grasping and manipulation. Sometimes, this can be compensated with longer digits and decreased palm surface, but this makes in-hand manipulation more difficult.

Therefore, the type of palm design warrants a separate, albeit small weight in the evaluation scale. It is given 10% of the total Form evaluation score for an open palm, with an offset thumb design while a no score for closed palm, since the anthropomorphic form is favoured in humanoid robotics and prostheses.

4.5.2. Width of Opening

Another essential parameter to the approach of grasps in robotic hands is the hand width opening in robotic hands [32]. This can be defined as the maximum distance between the tip of the index or middle finger (or digit closest to the thumb in case of closed type hands) to the tip of the thumb in opposable position. To this end, the target value for the width opening was set at a value of 10 cm minimum for a full score and between 5 cm and 10 cm for half the score.

4.6. Size

For prosthetic robotic hands, self-contained or otherwise, it is only natural to have a human hand size or a proportional size to whom the prosthetic is being fit. The same holds true for humanoid robots. A hand proportional to the size of the robot, keeping with human proportions should be selected. Human hand proportions can vary due to a number of reasons as explored in Markze et al. [33]. For the purposes of this work, the size here refers to a proportional scaling for the height of the 95th percentile hand as referred to by Tilley [34]. This feature is assigned a weight of 10% in the Form part of the evaluation, if the hand is within $\pm 15\%$ of its proportional human size and no score for all other values.

4.7. Weight

Hand weight plays a smaller role in determining anthropomorphism. However, it should not differ widely from that of the human hand. The average human hand is around 0.58% of total body weight [35]. The average human weight (mass) is 70 kg (154 lbs), so the average hand weighs 406 g (0.89 lbs). This can be defined as a ratio to the human body weight. The human hand weighs about 0.6% of the human body weight.

For the purposes of this study, any hand that falls under 3% of the total body weight of the subject should be considered anthropomorphic. This is due to the fact that human arm and hand weight average around 2.5% among all humans [35] and actuation in robotic hands is often housed in the forearm.

A detailed breakdown of all the weights for the "Form" factor is given in

Table 1. The “Form” weight distribution in the FFP index.

Form		Score
Opposable thumb	Fixed	10
	Manual Multiple	15
	Actuated Multiple	25
Weight to payload	-	15
Fingers	Joints per finger	10
	Number of fingers	10
	Range of motion	5
Padding	Soft pads	5
	Friction pads	5
Palm structure	Type of palm	5
	Width	10
Size of the hand	-	10

.

5. Features

Features or control features in this context are intended to signify the parameters that would enhance the amount of data available to control the hand effectively. This means that the robotic system should have all the tools to make itself completely deterministic, fully observable and actuated at its disposal. Control features are also weighted separately, along with Form and Performance.

5.1. Actuation

Fully observable systems compensate for the drawbacks to underactuated systems to a great extent keeping this in mind, and actuation features are given a slightly lower weight than sensor systems. Actuators have a completely different set of constraints. Human hands vary greatly in their speed and grip force and are capable of reaching up to 400 N in everyday tasks. These kinds of actuation characteristics are next to impossible with the current motors in the market if they are to be housed within the hand. Hence, remote actuation seems to be the stop-gap solution for now. Finding miniaturized actuators that would fit into the finger phalanges or at least into the palm of the hand is a constraint apparently shared with nature. This necessitates the use of actuators placed in the forearm and using tendons to transmit forces to the finger joints in many cases.

Speed: Human-like speed in normal every day grasping is not impossible to achieve, as is the case with grasping force. It is, however, mandatory to ’react’ quickly and to perform grasps in an efficient manner. Since the target value for full closure can vary on a number of factors such as number of joints, range of motion etc., and since we want to keep the requirements to be as general as possible so as to not constrain the designer, it was kept at a low level target of 1 to 1.5 s for full open-closure of a finger across its entire range of motion. Hence, it is given a weight of 15% for complete flexion and extension is carried out within one second. A score of 5% is given for a speed anywhere between 1 to 1.5 s for flexion/extension which is near human speed, and a no score for anything below those prescribed speeds.

DOF vs. DOA: This is a difficult parameter to evaluate considering that the efficacy of the system depends on if the system has enough sensors and adequate controllable parameters to make the system fully observable. Keeping this in mind, a weight of 10% is assigned to underactuated hands, while a completely actuated robotic hand is given a score of 20% while over-actuated systems are given a full score of 25%. There are also some some robotic hands in which you have more actuators than the systems’ degrees-of-freedom; these are called over actuated systems. In the case of robotic hands, the advantages they provide is typically to control to a great extent the impact of a force load response of the system and also being case-specific, but the marginal advantages do warrant a slightly better score than that of fully actuated systems.

5.2. Sensors

Sensors are what robotic systems use to get useful data of its own state or the environment it is placed in. It can be partially or fully observable to play any role in the control of the hand. Robotic hands are basically puppets without the use of sensors since open loop control of robotic hands requires a human in the loop all the time. Hence, sensors play a major role in the development of a hand, and some of the key types of sensors are discussed in the following sections. The category as a whole carries an increased weight of 60% as compared to actuation in the features part of the evaluation. The breakdown of the weights in each of the sub-categories is as shown in

Table 2. The “Features” weight distribution in the FFP index.

Features		Score
Actuation	Fixed Speed/Power	5
	Variable Speed/Power	10
	DOF vs. DOA	25
Sensors	Joint/Position	20
	Force/Torque	15
	Tactile/Touch	20
	Others	10

.

Joint/Position: Joint position sensing is critical in robotic hands as they give feedback on the link and joint position in the robot’s workspace. In underactuated hands, it also provides feedback on the grasp as it conforms to the shape of the objects being handled. In the absence of force sensors (which is usually the case), it can prove vital to grasp quality.

Force/Torque: A very useful feature to have in robotic hands is information on the amount of force being applied by the hand on its environment or even the forces and torques that are present within the system itself. This can provide useful information on tendon tension in tendon actuated hands; it can provide force and torque parameters that are critical in force-closure and form-closure control. It can also be key in aiding human–robot interaction and promoting safety. In the case of prosthetic robotic hands, myoelectric sensor inputs are used for intent learning and are also considered to be part of this category and carry the same weight of 15% as that of force feedback, even though it is a sensor that gives an input for actuation response rather than being a status monitor affected by the actuation.

Tactile/Touch: An expansive review of tactile sensors was done by [36]. The minimum functional requirements for a robotic tactile sensing system mimicking human manipulation was summarised as:

• Detect the contact and release of an object,
• Detect the lift and replacement of an object,
• Detect shape and force distribution of a contact region for object recognition,
• Detect contact force magnitude and the direction for maintaining a stable grasp during manipulation,
• Detect both dynamic and static contact forces,
• Track variation of contact points during manipulation,
• Detect the difference between predicted and actual grip forces necessary for manipulation,
• Detect force and magnitude of contact forces due to the motion of the hand during manipulation,
• Detect tangential forces due to the weight and shape of the object to prevent slip.

Hence, any type of feedback sensor that fulfills most or all of the above requirements falls under the category of tactile feedback. It is essential for the above-mentioned reasons. In addition, as technology advances in this direction, there has been an increasing amount of robotic hands that have been incorporating this as part of their design [16,37,38].

Others: These could be sensors providing information on temperature, olfactory, vision, any type of depth sensing, stress, strain, twist etc. These are considered tertiary level sensors and, although they might be function specific and aid in enhancing the features of the robotic hand, they are not considered to be mandatory as a general guideline.

A detailed analysis of the “Features” in each of the hand analysed is shown in Figure 10 where a no score is given as an empty index.

6. Performance

The most common way of grading hands has been to make it grasp objects of different shapes and sizes. The first attempts to make a standard set of grasps that well define and categorize a hand grasp comes from [1], which was later improved upon by [2,39].

Biagiotti et al. [3] state that “Grasping is intended as constraining objects inside the end-effector with a configuration that is substantially invariant with time (the object is fixed with respect to the hand workspace), while internal manipulation means controlled motion of the grasped object inside the hand workspace, with constraint configuration changing with time.” By this definition, simply grasping an object is not sufficient to completely grade all of the qualities in a robotic hand. Some benchmarking techniques also suggest carrying out manipulation tasks and non-prehensile tasks.

6.1. Performance Benchmarking in Literature

When it comes to evaluation of hands, grasping of objects is the standard. However, a standardized set of objects that are grasped in these evaluations does not exist. Therefore, a uniform comparison across platforms does not exist. There exists a variety of objects’ models: some use 3D model meshes, some provide just images while some others give real-world objects in the form of commercial kits that can be bought or in the form of a shopping list that can be used for buying the objects from any commercial outlet.

These grasping benchmarks supposedly represent the set of objects that best encompasses all types of grasps or even tasks that a robotic hand should perform. The more famous ones are from Georgia Tech and Yale University. A review of all available benchmarks was carried out by [40]; the interested reader is invited to consult it for further details.

A major shortcoming in any kind of grasping dataset is that it is inherently biased or too expansive, which in turn does not give a very intuitive view on the overall performance of the robotic hand under evaluation.

Keeping all this in mind, a list of objects that are used in daily events in a domestic environment was made. This was done by sourcing research done by Georgia Tech [41] and their study with amyotrophic lateral sclerosis (ALS) patients. These are objects used daily in a home environment and objects most often used in elderly care homes.

For performance metrics, the weights are assigned as shown in

Table 3. The “Performance” weight distribution in the FFP index.

Performance		Score
Prehensile	Power	20
	Intermediate	10
	Precision	15
Non-prehensile	Simple	15
	Complex	10
	Gestures	5
Manipulation	Translation	5
	Shift	5
	Rotation	5
Speed	-	10

for all sub-categories. In addition, all the following tests that are recommended are to be done with a minimum success rate of two out of three trials, with success being recorded if the desired status is maintained for a minimum of 30 s.

6.2. Prehensile

The prehensile tasks are based on the work done by [1] while proposing a new set of objects for grasping. We came up with a list of 65 objects which use all types of grasps as enlisted by [1,39]. It also tries to balance a wide range of textures and weights on real-world objects which are also used in everyday life.

In [42], Feix et al. define a successful grasp as “A grasp is every static hand posture with which an object can be held securely with one hand, irrespective of the hand orientation”. For the sake of simplicity, the type of grasps were divided into either Power, Intermediate or Precision grasps.

Power: The power grasp encompasses grasps that require both large and intermediary grasp forces. These types of grasps focus on the stability of the grasp and are usually imparted on larger objects that need a secure clamping action. These are most often used in scenarios where the object being grasped needs to be moved from one place to the other and usually does not involve manipulation of the object.

Intermediate: These are the “in-between” state of grasps that are represented within the taxonomy since it is between power and precision in its state [43].

Precision: Precision grasps require lower force and higher accuracy in positioning and control. It is typically characterized with thumb opposition to the distal joints of the other fingers. These are used in the grasp and manipulation of smaller objects.

6.3. Non-Prehensile

Non-prehensile manipulation is generally categorised as the handling of any objects without straight grasping. This kind of manipulation might be done in a number of ways such as: pushing, squeezing, twirling, tapping, rolling, etc. A set of non-prehensile tasks are also proposed in the following sections based on everyday activities.

Simple: These kinds of tasks involve displacement of a single object placed within or outside the hand from its current state, e.g., lifting a cup.

Complex: These are manipulation tasks that involve interaction between two distinct objects excluding the hand, and changing their current state, e.g., pouring water into a glass.

Gestures: With the rise in Human–Robot and Human–Agent interaction, non-verbal communication has become an important part of the development of robotic hands. A simple set of gestures is hence recommended in the FFP index to evaluate the basic gesture performing capabilities of the hand. Since hand gestures vary according to culture, fields and regions; five distinct hand signals which are universally accepted for their respective action intents are listed.

6.4. In-Hand Manipulation

A major distinction from most performance metrics of robotic hands is the emphasis on the in-hand manipulation aspects of the hand. Since this aspect of performance analysis is heavily dependent on the control robustness of the system, we limit it to a few simple tasks which can be done using open loop position control.

Translation: It is the ability to move objects from the fingertips to the palm or vice versa.

Shift: It is the ability to move an object in a linear manner with the fingertips.

Rotation: It is the ability to turn an object around the pads of the fingers and thumb.

6.5. Tool Use

Additionally, tool use in robotic hands is a useful yet redundant task when it comes to the design of robotic hands for prehension and manipulation tasks as they involve, to a great extent, robustness in control. The tasks listed in the Appendix are recommended but not necessary in the FFP evaluation of robotic hands.

6.6. Weight to Payload

The weight of the hand compared to the payload it is capable of carrying is an important factor in the design of robotic hands. It also plays a significant role in the design of humanoid robots since the final articulation and manipulation skills depend on the end-effector of the robot.

It is considered to be a key design parameter in prosthetics according to the survey done by [24]. In humanoid robotics, a heavy hand can increase the cantilever effect along the arm and can make the robot structurally weaker. In this evaluation scheme, what is referred to as “weight” also includes the actuation method (be it remote or self-contained). On the other hand, “payload” refers to the maximum weight that can be held by the hand given only its actuators and transmission system.

An adequately well-balanced ratio of 1.5:1 is adopted from the analysis done in Section 3. Hence, if the ratio is 1.5 or lower, the hand gets a perfect score of 20%. If the ratio is 1.5 or lower excluding the actuators from the weight but higher otherwise (remotely actuated hands), the weight is cut by half to 10% and a no score for all other values.

A detailed breakdown of all the weights for the “Performance” in each of the hand analysed is given in Figure 11 where a no score is given as an empty index.

7. Price Addendum

This section is aimed at providing an additional criterion to the standard evaluation (in our sample, shown in Figure 12), wherever it is deemed necessary. This is not considered to be a necessary part of the evaluation and remains only to show how similar performances across the varied categories can be contextualised when provided with a price-to-cost ratio for the hands being rated.

In this case, all hands are compared against the most expensive robotic hand being evaluated, to provide a reference. Hence, the most expensive robotic hand gets rated at a 100% while all the other hands’ costs are scaled accordingly. A sample of the price addendum incorporated results is shown in Figure 13 where the iCub, R1 and Michelangelo hands are compared.

8. FFP Evaluation of the iCub Hand

The iCub hand (shown in Figure 3) and the Perfect Hand (the human hand) is used as a preliminary baseline to evaluate other hands and to compare against them. The standard evaluation (Figure 12) is done by means of an online questionnaire form [44] where users can fill in a questionnaire based on the hand that is being evaluated. The hand will be benchmarked if the user sends video proof in the form of pictures and videos to the authors.

This was done for the iCub hand and some sample pictures of the performance characteristics are shown where the iCub is grasping a series of objects (Figure 14) and performing a series of gestures (Figure 15).

8.1. Form

The iCub has an open form, anthropomorphic, five-fingered and underactuated hand. It has three phalanges to each of the fingers. It has an opposable, articulated thumb. The tactile skin is covered with a dielectric layer and a second conductive layer: electrically conductive Lycra for the palms and electrically conductive silicone for the fingertips. This layer is connected to the ground and enables the sensor to respond to objects irrespective of their electrical properties. This gives it a thin layer of compliant material with minimal friction properties. The iCub hand scored an $80\%$ for its Form properties, which is a good indication that the hand is highly anthropomorphic.

8.2. Features

The iCub hand has Hall effect position sensors at each of its joints, it has motor position sensing and a tactile skin on its palm and fingertips. The actuation of all these joints is obtained using nine DC motors (resulting in nine DOAs), seven of which are embedded in the forearm and two in the hand. Therefore, certain DOFs are obtained by coupling different joints (either tightly or elastically) so that they are moved by a single motor.

It has minor issues with the joint position sensing as it tends to be nonlinear and is not a fully observable system. It scored a 55% on the features part of the FFP evaluation.

8.3. Performance

The performance analysis of the iCub was done on all three sub-categories. It performed well when it came to almost all types of grasps and gestures. It was, however, not the ideal hand for manipulation and non-prehensile tasks. One of the major issues being the thumb being too long to manipulate the objects in-hand effectively, since it leads to workspace occlusion. It scored a high 80% in its performance evaluation.

8.4. Other Evaluated Robotic Hands

Some of the other hands evaluated include the Michelangelo hand and the R1 hand.

The Michelangelo prosthetic hand [45] features a highly natural design: it has a thumb, index and middle fingers that are actively driven, while the ring and pinky passively follow the other fingers. The fingertips are made of a combination of soft and hard materials to allow real precision when grasping objects. Michelangelo claims to have the first thumb in the world that can be positioned electronically. The R1 is a service robot [46] that is built for domestic applications.The R1 hand [47] uses a completely plastic structure with embedded electronics. It has two actuated degrees of freedom (DOF) with four phalanges, coupling two phalanges to each degree of actuation.

The evaluation results for the R1, Michelangelo and the iCub hand is shown in Figure 13. It shows the clear difference across the different platforms and also draws conclusions as to what the design priorities were during and after production.

All the performance tests for the prosthetic hands were done by a professional user, experienced at operating the hands: with a human in the loop, controlling the system using surface electromyography (s-EMG) sensors coupled with a teach pendant. The R1 performance tests were done on the first version of the hand by an expert user in a controlled environment.

All tests were done using the price addendum to show the trade-offs done in the different designs and how they affected the end cost of the robotic hand.

9. Conclusions and Future Work

We developed the FFP index to provide a more objective platform for evaluating robotic hands. In more detail, we analyzed three different aspects of robotic hands (form, features and performance). The quantitative evaluations allowed for comparing key characteristics of given robotic hands with respect to the state of the art, thus suggesting the rationale used to resolve design trade-offs. As illustrated in the evaluation section, it is apparent that robotic hands in prostheses aim at optimizing weight, payload and generic grasps while trying to maintain an anthropomorphic appearance. In humanoid robotics, the form and performance vary, but sensing features are usually incorporated into the design to effectively control the robotic hand. This provides a good grading and comparison platform for robotic hands across multiple use cases. Sub-category analyses also explore the nuances among robotic hands that have the same functionality. This can be helpful as a guideline for designers when developing novel robotic hands.

Future Work

We evaluated the robotic hands available at our disposal. Due to the hands-on approach of the evaluation index, we were unable to evaluate a wide range of hands. We are in the process of collaborating with experts working in the field to build a bigger repository of available robotic hands in the market and their analyses from the corresponding scores. The questionnaire is limited to providing only the scores for now. A more comprehensive site that provides an in-depth analysis and a comparative study of different robotic hands, right away after data submission, is in the works.