Quantitative Assessment of Drone Pilot Performance

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The paper presents a methodology for simuating, capturing performance measures and the defining design and training needs for drone pilots. There is quite a good literature review considering the breadth of the topics covered, followed by a strong description of the simulation environment. Some empirical evidence is presented along with the application of these outputs to a number of applications. There are some weaknesses of the paper (see below). Overall, the paper is quite nice, there is useful material, and the underpinning work seems thorough, but there is a lot (too much?!) in the paper from a technical perspective but little in the way of discussion. So, I think the authors have done a good job, and this should be publishable, but more work is required first. 

Overall the structure is a little odd – I would have the experimental section in a section of its own, rather than a subhead, though see comment below

Would be good to clarify scope – you are looking at single drone control or multiple drone control?

Also, somewhere to describe the stage or phases of the drone control task – you mention this later but also a general outline of the phases of control, and the level of human intervention / monitoring would be helpful. How much of this is about the physical control of the drone rather than monitoring and managing things like target acquisition etc. How much automation is involved? These are all important for you to position your context.

I realise you already have a Cummings reference in there, but this paper is also very relevant https://ieeexplore.ieee.org/document/4445761

The term human performance modelling is used in a very general sense – you don’t necessarily mean just digital modelling (as per https://www.tandfonline.com/doi/full/10.1080/00140139.2020.1723683 ), and could be specified for the context of the paper; the empirical results are presented in a very cursory way, which is ok if this is indicative use of the simulator, but not if it is meant to give a substantial contribution; and there is no discussion (though the conclusion is quite lengthy). 

L 488 – Flip, not lip, a switch

I think the section on the simulation is excellent and informative. TBH this would make a significant contribution on its own, especially in light of the next few comments.

I am a Human Factors researcher so I was looking forward to the study part of the paper, but it was really quite weak in terms of its presentation. It was not clear how many people were involved, at different levels of experience, what the method and protocol was etc etc. If you have anything like decent participant numbers you should present this as a standalone study with lots of depth (you might already have this planned). It can still be included in the current paper but you must be much clearer in the introduction and in the part of the paper where you present it, how it contributes to the paper – and given the level of detail you have, this must be as an ‘illustration’ or an ‘indicative example’ of how the simulation is used. I would also put this in its own section.

I didn’t quite understand section 5 – is this just off the simulation, or is it also generated from the data? Much more clarity pleas.

There is no discussion where you go back to the literature in the background and say how your work matches, extends or maybe finds something different from the current research. Your conclusion is quite long and most of that is really discussion (your conclusion should just be the key outcomes of the paper) so you could repurpose much of that material and link it to some of the research you have already presented.

 

Author Response

Comments 1: The paper presents a methodology for simuating, capturing performance measures and the defining design and training needs for drone pilots. There is quite a good literature review considering the breadth of the topics covered, followed by a strong description of the simulation environment. Some empirical evidence is presented along with the application of these outputs to a number of applications. There are some weaknesses of the paper (see below). Overall, the paper is quite nice, there is useful material, and the underpinning work seems thorough, but there is a lot (too much?!) in the paper from a technical perspective but little in the way of discussion. So, I think the authors have done a good job, and this should be publishable, but more work is required first.

Response 1: Thank you for your thoughtful analysis and positive remarks.

 

Comments 2:  Overall the structure is a little odd – I would have the experimental section in a section of its own, rather than a subhead, though see comment below

Response 2: Based on your suggestion, this content has now been placed in its own section.

 

Comments 3: Would be good to clarify scope – you are looking at single drone control or multiple drone control?

Response 3: We are focusing on single drone control, and this has been clarified in the revised manuscript.

 

Comments 4: Also, somewhere to describe the stage or phases of the drone control task – you mention this later but also a general outline of the phases of control, and the level of human intervention / monitoring would be helpful. How much of this is about the physical control of the drone rather than monitoring and managing things like target acquisition etc. How much automation is involved? These are all important for you to position your context.

Response 4: We appreciate this insightful comment. A section detailing the phases of drone control, including the level of human intervention and automation, has been added to better position our context.

 

Comments 5: I realise you already have a Cummings reference in there, but this paper is also very relevant https://ieeexplore.ieee.org/document/4445761

Response 5: We have added the recommended reference and expanded our discussion on multi-drone control accordingly.

 

Comments 6: The term human performance modelling is used in a very general sense – you don’t necessarily mean just digital modelling (as per https://www.tandfonline.com/doi/full/10.1080/00140139.2020.1723683 ), and could be specified for the context of the paper;

Response 6: We have added the recommended reference and expanded our discussion on multi-drone control accordingly.

 

Comments 7: the empirical results are presented in a very cursory way, which is ok if this is indicative use of the simulator, but not if it is meant to give a substantial contribution; and there is no discussion (though the conclusion is quite lengthy).

Response 7: The results section has been enhanced to provide more details.

 

Comments 8: L 488 – Flip, not lip, a switch

Response 8: Thank you for pointing that out. The typographical error has been corrected.

 

Comments 9: I think the section on the simulation is excellent and informative. TBH this would make a significant contribution on its own, especially in light of the next few comments.

Response 9: We greatly appreciate your positive feedback on the simulation section.

 

Comments 10: I am a Human Factors researcher so I was looking forward to the study part of the paper, but it was really quite weak in terms of its presentation. It was not clear how many people were involved, at different levels of experience, what the method and protocol was etc etc.

Response 10: We have revised the study part of the paper to include clearer details on participant numbers, experience levels, methodology, and protocol.

 

Comments 11: If you have anything like decent participant numbers you should present this as a standalone study with lots of depth (you might already have this planned).

Response 11: Given our current participant numbers, we have opted not to present this as a standalone study, as it might extend the paper significantly without substantial additional value.

 

Comments 12: It can still be included in the current paper but you must be much clearer in the introduction and in the part of the paper where you present it, how it contributes to the paper – and given the level of detail you have, this must be as an ‘illustration’ or an ‘indicative example’ of how the simulation is used.

Response 12: We have clarified that the results serve as an illustrative example of how the simulation can be used.

 

Comments 13: I would also put this in its own section.

Response 13: Based on your suggestion, this content has now been placed in its own section.

 

Comments 14: I didn’t quite understand section 5 – is this just off the simulation, or is it also generated from the data? Much more clarity pleas.

Response 14: This section indeed uses real data, and we have clarified this in the text.

 

Comments 15: There is no discussion where you go back to the literature in the background and say how your work matches, extends or maybe finds something different from the current research. Your conclusion is quite long and most of that is really discussion (your conclusion should just be the key outcomes of the paper) so you could repurpose much of that material and link it to some of the research you have already presented.

Response 15: We have revised the conclusion to include a discussion that relates our findings back to the literature, ensuring alignment and comparison with existing research.

Reviewer 2 Report

Comments and Suggestions for Authors

_{-  The references cited in this manuscript seems to be appropriate and relevant to this paper.}

_{- Introduction are too long but in general are also ok. But is important to do some appoiments about it:}

_{1) It was found several use of the subject pronouns 'we' in 'We chose/We will/Our'.  All of those pronouns must to be removed and replaced by something like: 'The present paper focuses on .....'. The same thinking must to be applied to anothers. e.g., 'we chose' replace by '.. was chose...', etc.} 

₂₎ The introduction need to have more references, since several assertions was placed on.

3) Item 1.2-Related Work can be placed out of the introduction scope, like a new Section. It helps to be more readable.

4) Lines 64-85, paresents two differente itemization format. One using numbers, and othe using dots. Please, uniform both shapes to have same patterns along all the paper. The itemization based on numbers is quite recommended when the numbers represents a sequence os steps or order of execution.

5) L86-93: Please rewrite thit part. It seems redundant and confuse comparing with the items above.

6)  L226: Figures, e.g. Figure 1, must to be FIRST cited, then presented. It must to be applied for all figures, tables, equations, and so on.

7) The terms: Parametric packet-layer/Parametric planning/Bitstream layer/Hybrid models, were used in the text but it weren't used anymore. It is like new terms that just appeared suddely, out of the blue. Please, fix it, or remove it.

- L390: Replace '..section 1.2', by '..Section 1.2' (UPPERCASE).

_{-  L326, L439, L441, L446: Avoid to use acronyms at begining of the sentences, e.g. PX4. Apply it along all the paper scope.}

- L296: Biofeedback mechanisms WAS ALSO USED TO MEASURE THE PILOT PERFORMANCE?

- Figure 3: 'Mavlinkrouter– mavp2p' box, got wrong shape. Please correct it.

- L232, L513: Sometimes the authors itemize with numbers, sometimes with dots. Please, take a standard format for all, since the numbers didn't represent sequence or real meaning.

- L533: Itemize in standard shape. 

- L533: Remove all, 'we want...and so on'. The text must to be unpersonal.

- L548: Remove all bold texts, e.g. 'A standardised visual acuity object./An enemy camp'

- L623: Wrong text format. A subsection/section can't be empty. If such section can't be filled, it is better to not write it on the text. Remove Subsection 3.1.11, 3.1.12 because it are useless and empty.

L727, L740: Please, cite equation BEFORE and then, show it.

L873: Remove 'WE...'.

L873: The reference used to measure stress levels, are so old. Exist other methodologies from the past 10 years, based on questionnaires to measure stress levels. Why did the authors not used it?

- Table 1, seems to be bit confuse. What is Importance Level? It wasn't cited anywhere in the text. Who computed it?

- Table 1, haven't refenrences.

- L815, L821: Please, put words 'section' like '[S]ection', if it is followed by a number. SEE IT ALONG ALL THE TEXT.

- L831:  in the subsection 'Results [&] discussion', REPLACE '&' by the correct one 'and'. 

- L831. Since all the text bings a conclusion and the Results and Discussion, should be places in a different SECTION, not SUBSECTION., .e.g. Section 5. Right?

- L696: Remove 'we'.

- L700: Use 'The figure' is wrong. Please, rewrite it. 'Figure XYZ, demonstrates...' Take care to not be redundant since before you wrote Figure 6.

- Figure 10 and Figure 12: In the plot, what is the unit of the axis-Y? Unclean.

- How the performance level was indeed reached/computed is unclear along the text. Please, be clearer to the readers on that.

FINAL REMARKS: The paper have good team of authors really inside the context of the proposed paper. In addition, the paper brings good ideas to the table, but in general, since it brings TO MUCH information and methodologies, it makes the paper confuse. It is much better try to reach less solutions and be clearer then be really heavy of content and be confused.

I think the paper is ok but can be improved and if possible, have less content and problems to solve. Paper really big and sometimes confuse, hard to conclude some. Some plots results seems to be really simples facing the size of the paper.

RESULTS: Must to be improved. Weak compared to the size of the paper.

 

Best regards.

 

 

Comments on the Quality of English Language

 It requires a minor editing of English language.

Author Response

Comment 1: The references cited in this manuscript seems to be appropriate and relevant to this paper.

Response 1: Thank you for the confirmation.

 

Comment 2: Introduction are too long but in general are also ok.

Response 2: We acknowledge your point. As suggested, we have now split the introduction to improve readability.

 

Comment 3: But is important to do some appoiments about it: It was found several use of the subject pronouns 'we' in 'We chose/We will/Our'.  All of those pronouns must to be removed and replaced by something like: 'The present paper focuses on .....'. The same thinking must to be applied to anothers. e.g., 'we chose' replace by '.. was chose...', etc.

Response 3: After careful consideration, we have decided to retain the use of the pronoun "we," following the style commonly accepted in articles published in the “Drones” journal, as it aligns with our intention to use active and clear language.

 

Comment 4: The introduction need to have more references, since several assertions was placed on.

Response 4: This seems to be contradictory to the previous comment. We chose not to add more references to the introduction to avoid further lengthening the manuscript, considering the existing comment on the paper's size.

 

Comment 5: Item 1.2-Related Work can be placed out of the introduction scope, like a new Section. It helps to be more readable.

Response 5: We have implemented this suggestion, and "Related Work" is now presented as a separate section.

 

Comment 6: Lines 64-85, paresents two differente itemization format. One using numbers, and othe using dots. Please, uniform both shapes to have same patterns along all the paper. The itemization based on numbers is quite recommended when the numbers represents a sequence os steps or order of execution.

Response 6: We have standardized the itemization format throughout the paper, using a consistent pattern for clarity.

 

Comment 7: L86-93: Please rewrite thit part. It seems redundant and confuse comparing with the items above.

Response 7: Although the section may seem redundant, we retained it to help readers understand the structure of the document, which some journals require explicitly.

 

Comment 8: L226: Figures, e.g. Figure 1, must to be FIRST cited, then presented. It must to be applied for all figures, tables, equations, and so on.

Response 8: Figure placements were managed by LaTeX, and we anticipate these will be adjusted during the final copyediting process by the journal, hence we did not enforce strict placement at this stage.

 

Comment 9: The terms: Parametric packet-layer/Parametric planning/Bitstream layer/Hybrid models, were used in the text but it weren't used anymore. It is like new terms that just appeared suddely, out of the blue. Please, fix it, or remove it.

Response 9: These terms are defined by ITU, as explained in the paper and Figure 1. They are included to illustrate how our approach fits within this taxonomy.

 

Comment 10: L390: Replace '..section 1.2', by '..Section 1.2' (UPPERCASE).

Response 10: This formatting issue has been corrected.

 

Comment 11: L326, L439, L441, L446: Avoid to use acronyms at begining of the sentences, e.g. PX4. Apply it along all the paper scope.

Response 11: We have addressed the issue of using acronyms at the beginning of sentences and corrected it throughout the paper.

 

Comment 12: L296: Biofeedback mechanisms WAS ALSO USED TO MEASURE THE PILOT PERFORMANCE?

Response 12: While biofeedback mechanisms can indeed measure pilot performance, we did not employ these methods in this study, as discussed in the text. We have also outlined this as a potential area for future research.

 

Comment 13: Figure 3: 'Mavlinkrouter– mavp2p' box, got wrong shape. Please correct it.

Response 13: We have reviewed the shape of the "Mavlinkrouter– mavp2p" box and believe it to be accurate as presented.

 

Comment 14: L232, L513: Sometimes the authors itemize with numbers, sometimes with dots. Please, take a standard format for all, since the numbers didn't represent sequence or real meaning.

Response 14: We have made the itemization format more consistent, with sequential lists used only where necessary.

 

Comment 15: L533: Itemize in standard shape.

Response 15: The itemization format has been standardized as recommended.

 

Comment 16: L533: Remove all, 'we want...and so on'. The text must to be unpersonal.

Response 16: We maintained the use of the pronoun "we," as previously discussed in response to comment 3.

 

Comment 17: L548: Remove all bold texts, e.g. 'A standardised visual acuity object./An enemy camp'

Response 17: The bold text in the manuscript has been removed.

 

Comment 18: L623: Wrong text format. A subsection/section can't be empty. If such section can't be filled, it is better to not write it on the text. Remove Subsection 3.1.11, 3.1.12 because it are useless and empty.

Response 18: The empty subsections have been removed as suggested.        

 

Comment 19: L727, L740: Please, cite equation BEFORE and then, show it.

Response 19: We have adjusted the text to ensure these equations are cited before they are presented.

 

Comment 20: L873: Remove 'WE...'.

Response 20: We have addressed the use of the pronoun "we," consistent with our earlier responses.

 

Comment 21: L873: The reference used to measure stress levels, are so old. Exist other methodologies from the past 10 years, based on questionnaires to measure stress levels. Why did the authors not used it?

Response 21: Although older references were used to measure stress levels, we acknowledge this limitation and discuss it in the context of future work. Notably, we also utilized questionnaires to assess stress levels, as mentioned in the paper.

 

Comment 22: Table 1, seems to be bit confuse. What is Importance Level? It wasn't cited anywhere in the text. Who computed it?

Response 22: We have provided a clearer definition of the "Importance Level".

 

Comment 23: Table 1, haven't refenrences.

Response 23: A citation has been added to Table 1.

 

Comment 34: L815, L821: Please, put words 'section' like '[S]ection', if it is followed by a number. SEE IT ALONG ALL THE TEXT.

Response 24: We have corrected the formatting of section references.

 

Comment 25: L831:  in the subsection 'Results [&] discussion', REPLACE '&' by the correct one 'and'.

Response 25: The ampersand has been replaced with "and" in the subsection title.

 

Comment 26: L831. Since all the text bings a conclusion and the Results and Discussion, should be places in a different SECTION, not SUBSECTION., .e.g. Section 5. Right?

Response 26: We have implemented this suggestion by creating a separate section for Results and Discussion.

 

Comment 27: L696: Remove 'we'.

Response 27: As discussed earlier, we have retained the use of the pronoun "we."

 

Comment 28: L700: Use 'The figure' is wrong. Please, rewrite it. 'Figure XYZ, demonstrates...' Take care to not be redundant since before you wrote Figure 6.

Response 28: The sentence structure has been revised to avoid redundancy and to clarify the reference to figures.

 

Comment 29: Figure 10 and Figure 12: In the plot, what is the unit of the axis-Y? Unclean.

Response 29: The normalization of plots has been clarified in the text, indicating they are dimensionless.

 

Comment 30: How the performance level was indeed reached/computed is unclear along the text. Please, be clearer to the readers on that.

Response 30: We have explicitly referenced the methodology used to compute performance levels in Section 5.1

 

Comment 31: FINAL REMARKS: The paper have good team of authors really inside the context of the proposed paper.

Response 31: Thank you for your kind remarks.

 

Comment 32: In addition, the paper brings good ideas to the table,

Response 32: We appreciate your recognition of the ideas presented in the paper.

 

Comment 33: but in general, since it brings TO MUCH information and methodologies, it makes the paper confuse. It is much better try to reach less solutions and be clearer then be really heavy of content and be confused.

Response 33: Thank you for your feedback. Our intention was to provide a comprehensive overview of the various approaches explored in our research. In our view, it is the holistic framework that the simulation environment provides (with all the presented possibilities) that makes this paper strong. We acknowledge your concern about the paper’s breadth and have worked to clarify the content in response to your feedback. We believe the comprehensive nature of our simulation environment is a strength and hope the revisions have enhanced its clarity.

 

Comment 34: I think the paper is ok but can be improved and if possible, have less content and problems to solve. Paper really big and sometimes confuse, hard to conclude some. Some plots results seems to be really simples facing the size of the paper.

Response 34: Thank you for your constructive feedback. We're glad you found the paper acceptable and appreciate your suggestions for improvement. We have made efforts to streamline the content where possible and clarify the presentation of our results, while maintaining the necessary depth. It is true that the plots are kept simple, but our objective was just to keep the presentation simple. We could provide more elaborate data, but our fear is that this would bring few added value to the reader.

 

Comment 35: RESULTS: Must to be improved. Weak compared to the size of the paper.

Response 35: The results section has been revised to provide greater clarity and emphasize that the results are intended as illustrative examples.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thankyou to the authors for addressing my comments. I hope they feel their paper is improved. I have no further comments and am happy to recommend the paper for publication. 

Author Response

Comments 1: Thankyou to the authors for addressing my comments. I hope they feel their paper is improved. I have no further comments and am happy to recommend the paper for publication. 

 

Response 1: We thank the reviewers for their valuable feedback, which has certainly contributed significantly towards increasing the quality of the paper.