A Systematic Approach to Evaluating and Benchmarking Robotic Hands—The FFP Index
Round 1
Reviewer 1 Report
This paper presents a new index (dubbed FFP) to evaluate robotic hands. I consider the study quite interesting for robotic applications and for design of prosthesis.
The main lack of the paper is that a comparison of different types of robotic hands is missing. In addition the FFP index seems to be function of subjective evaluation (within section 8 authors mention a questionnaire filled by users of the hand), and of the dataset of hands considered. These two factors imply that it may be not univocally defined (for instance the price addendum is computed considering the most expensive hand, that is the Michelangelo hand in the paper but this may change if different hands are considered).
Following you can find specific questions and comments.
1. The title of the section 1.2 _Analysis_ is not clear. What does it mean Analysis in this context?
Section 2
2. It is surprising that within section 2.1 _Survey of the state of the art_ the authors did not cite any review.
3. Although a thorough review of the existing robotic hands are out of the scope of the paper, the robotic hands cited are a few. It is not clear what are the Inclusion and exclusion criteria.
Lines 104-108
4. I am not fully convinced by the claim in Section 2.1.2 that commercial robot hands must be cost effective or I would exclude the Shadow Hand since the price is quite high.
Lines 109-121
5. Within the Prosthetic section there are no examples cited. Why?
Lines 130-131
6. Another important factor that should be taken into account for hand evaluations is human-robot interaction (HRI) that …. This sentence sounds odd. I am not sure that HRI could be defined as a factor.
Lines 133-135
7. Myoelectric hands have been on the rise in prosthetics, they tend to focus on a more natural input from the user. This sentence is not clear. First Myo Hand are available since the ’60, secondly I am not sure about the influence of the hand design with the natural input from the user. Are you referring to the user-hand interface design?
Figure 3
8. I would propose to better present the different blocks in the picture. For instance, since Form, Features and Performance shared the same weights and are directly connected to the main block FFP index I would align these three blocks.
Section 3.2.1
9. Concerning the underactuation I would suggest the authors to cite the works of T Laliberté and L Birglen and to better define underactuation in the context of robotic hands.
Figure 4
10. It is not clear which hands have been considered in the graph. Each mark must be labelled with the name of the hand, otherwise these data cannot be verified.
Lines 177-180
11. Prosthetic hands …, they are usually underactuated with a single actuator since they have human in the loop to compensate and adjust for any shortcomings in sensing and control. This sentence is questionable. Prosthetic hands are usually underactuated since they are designed to restore primitive prehensile functions (at least power grasp) and must be lightweight and reliable.
Lines 180-181
12. Research hands are usually equipped with a sensor suite and are usually remotely actuated, which makes underactuation a better choice. It is not clear why underaction is a better choice if the research hands are usually remotely actuated. If I could rely on more volume for the actuation unit I would try to use the largest number of actuators, isn’t it?
Figure 4
13. As for figure 3. In addition, according to the data reported here there is a prosthetic hand that weights 6 Kg and others that have a payload<1 Kg. Are you sure these are data of prosthetic hands
Line 195
14. Simply put, a good robotic hand should be able to lift more than it weighs. This sentence sounds vague and the result is questionable. There is not a scientific evidence of such output.
Line 197
15. It can be observed that the weight of prosthetic hands is not significantly different from their quoted payload. I cannot agree with this claim. Which are the prosthetic hands considered?
Lines 207-208
16. It can be seen that the payload-to-weight ratio for robotic hands tends to slightly higher than the one of commercial or prosthetic hands. I cannot agree with this claim. Which are the hands considered?
Line 210 -211
17. From these observations it can be concluded that a good hand payload-to-weight ratio is between 1 and 1.5. Here the authors assume that all the hands considered are good designs and by computing an average consider the result as a good target. This is questionable.
Line 214
18. Even underactuated hands tend to be heavy if motors are housed within the hand. Why the weight should be different if the actuation unit is housed within the hand? Even in remotely actuated hands, the weight of the actuation unit must be considered otherwise the comparison is unequal.
Figure 6
19. As for figure 4.
20. Sections 3.2.4 and 3.3 are confusing. It is not clear what are the take-home messages from these two paragraphs. In addition, concepts are reported without citing any supporting papers (as the sentence reported in lines 236-238).
Line 303
21. There is a mismatch between the weight reported in the text (20%) and the weight reported in figure 3 (25%)
Section 4.4 _Padding_
22. Relevant papers addressing this topic are not cited in this section.
Figure 8
23. Is the perfect hand the human hand?
24. It is not clear how the range of motion has been evaluated and why the index for two of the hands considered is missing.
4.5.2. _Width of opening_
25. What is the target value?
Line 360
26. For prosthetic robotic hands… Are you considering both or just one of the two types?
Line 364
27. It is not clear why the authors referred to the 95th percentile. Is it from male of female subjects?
Lines 373-374
28. For the purposes of this study, any hand that falls under 3% of the total body weight of the subject should be considered anthropomorphic. It is not clear and questionable. First are you considering an average human male? If yes I would report the directly the weigh. Considering a human subject weight of 70 Kg, the 3% is about 2.1 Kg, that could be ok for a robotic hand but not for a prosthesis.
Line 392 _Speed_
29. What is the target value?
Lines 501-502
30. Lifting a cup is quite general and seems to be related to a prehensile action.
Section 7 _Price addendum_
31. This addendum is briefly explained and seems to be related to the dataset of hands considered. In addition, considering a price of about 35 K€ for the Michelangelo Hand (which includes a lot of extras that must be not considered such as maintenance, training, …), based on the graph in figure 15 the price of the iCUB results in about 5 K€. By looking at the two design I would expected different figures: The Michelangelo hand seems to be cheaper to manufacture respect to the iCUB hand.
Section 8 _FFP Evaluation of the iCUB hand_
32. In this section the authors mentioned an online questionnaire filled by users used to compute the evaluation. I would report within an appendix the questionnaire since it may be used as guideline for the reader to compute the FFP index.
33. The FFP index of the iCUB reported in the paper is the average result of how many questionnaire filled? Or it has been computed by the authors?
34. Figure 12 shows the results for two hands: Michelangelo and iCUB, while figure 13 and 14 report the video sequences only for the iCUB.
35. There is a mismatch between figure 11 (where is shown ICUB 2 hand) and the text within section 8.4 where the authors refer to the ICUB plastic hand. Are the two hands the same or different?
36. In addition the description of the Michelangelo and R1 hands are missing.
37. Figure 15 can be deleted since the same info are reported in figure 16.
Section 9 _Conclusion_
38. By analyzing three different aspects of robotic hands, we were able to identify the key motivation behind the development of each robotic hand considered in this study. There is no evidence of this in the paper or is not clear to the reader.
Author Response
The reviewer raised several interesting points. We are very thankful for your feedback. The responses are attached in the pdf.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The scope and contribution of this survey-like paper is good for a journal publication. The paper comprehensively surveys robotic hands with of the comparison of their form, feature and performance. For general purpose robotic hands, the new proposed FFT evaluation method is useful and meaningful. However, most industrial or commercial robots are designed for specific tasks; therefore, the proposed FFP index may not be the best way for evaluation. A discussion on the evaluation methods for general purpose robotic hands and application specific robotic hands would be helpful for the readers.
Author Response
The reviewer raised interesting points and the general comments were encouraging. We are very thankful for your feedback. The responses are attached in the pdf.
Author Response File: Author Response.pdf
Reviewer 3 Report
The paper provides a systematic approach for evaluation for robotic hands. It proposes indices based on the form, features and performance. The paper is well-written and provides useful information for performance evolution of robotic hands and can be published after minor revision.
1. How the weights for each section in Figure 3 were determined and assigned? Is there any reference to such?
2. On page 6, first line, it says: “Looking into literature, it is seen that higher the number of actuated DOF, the simpler it is to control effectively.” Which references are mentioning this?
3. DOA and DOF are reversed in the title of figure 4.
4. The performance section is a very important section, but very little explanation is provided on how the performance of different hands were evaluated and how the scores are calculated for different hands, i.e., information on experiments, number of trials, etc.
5. For most sections, the explanation of scores are provided in the form of text and sentences. It is better to provide those information also in tables to be easier to follow and use in the future for readers.
6. Very little information of the sensors and actuation systems (including transmission).
Author Response
The reviewer raised several interesting points. We are very thankful for your feedback. The responses are attached in the pdf.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Authors reply to all my questions.
