ATC Human Factors Involved in RPAS Contingency Management in Non-Segregated Airspace

Round 1

Reviewer 1 Report

This paper reports on a high-fidelity simulations that were conducted to quantify the effect of two factors on experienced ATCO workload and ATCO performance: factor one was RPAS-missions with failures, consisting of two different RPAS types and failures, and factor two was the effect of a tool /assistance system for ATCOS - the intent.

This paper claims to and conducted human factors research, but editing is needed to meet the standards of a human factors paper reporting empirical research. We recommend to check with APA recommendations on reporting quantitative data.

These additional information / changes in the style of reporting are required and standard for papers in the human factors domain; as presented at that time there are important methodological information missing.

Especially - no information about the sample of ATCOs is given, how many humans were involved, what is the overall sample size? What do whiskers show - variance or standard deviations? Statistical test to check significance are mentioned but not described - which tests were conducted, which experimental factors were tested against each other. The experiemtental design and factors are not described, especially dependent and independent variables are not clearly reported in the methodological part.

The discussion section does not answer the question raised by the title but repeats mainly arguments given in the introduction.There is no section where research questions or hypothesis are presented. Formally, test for statistical signifiance requires formulation of a hypothesis. Furthermore, we recommend to choose one description of scenarios (either scenario 1 ... 4) or "a.... c....e". For the subejctive rating of workload in Figure 5 - is this the ISA? This scale is from 1 to 5, there is no 0. Acceptable workload is in the area around 3, 1 is an underutiliztation which should be avoided.

In general - a clear distinction between describing the method of measuring the human factors, reporting the results and the discussion of the meaning of the result is recommended and necessary.

There is mentioned many times that there is a lack of research on the impact of RPAS operations on (conventional) air traffic controllers work. The authors fail to show from a human factors perspective, why RPAS are special to ATCOs (when they are supposed to behave like manned a/c). Even without RPAS, ATCOs are trained (and used) to handle traffic with different performance parameters and to consider these performance constraints when advising manoeuvres to a/c. Engine failures and loss of communication links or other technical failures can happen to manned aircraft as well, what is the specific additional taskload for the ATCO in these situations?

It would be interesting to compare a non-nominal situation with an RPAS with a non-nominal situation with a manned aircraft, to understand the impact of RPAS operation on experienced workload. It is "trivial" that workload is higher in all scenarios tested than in the baseline scenario.

What is the speciality of RPAS from this perspective? Why were the specific RPAS types chosen for this study?

For comparing the results on workload and ATCO taskload, we recommend to review papers on modeling this relationship and also take into account communication frequencies etc. The novelity of the questionnaire designed for this study cannot be evaluated as the items are not reported and the reader gets no information about sensitvity of this questionnaire compared to the more standard methods used in this paper, like NASA-TLX, ISA, taskload metrics.

It would be of great value for the human factors peer group to get more insights into the raw data of this empirical research. We know that getting more knowledge on the impact of new tools and modalities of traffic on human operators is crucial; if the authors follow our recommendations we are sure, that this research can contribute to this quest.

Author Response

Dear reviewer, the authors have taken into great consideration all reviewers' criticism, comments and suggestions. As a result, great effort and care have been taken to clarify the objectives of the experiment, the operational conditions surrounding the work performed by ATC controllers, new data has been added to provide a better foundation for the analysis of the results, and the analysis of the results has been deepened.

We honestly expect to have improved our original submission.

Following this, we respond more precisely to each of the comments the review provided.

Author Response File: Author Response.pdf

Reviewer 2 Report

Please see attached file for comments and suggestions

Comments for author File: Comments.pdf

Author Response

Dear reviewer, the authors have taken into great consideration all reviewers' criticism, comments and suggestions. As a result, great effort and care have been taken to clarify the objectives of the experiment, the operational conditions surrounding the work performed by ATC controllers, new data has been added to provide a better foundation for the analysis of the results, and the analysis of the results has been deepened.

We honestly expect to have improved our original submission.

Following this, we respond more precisely to each of the comments the review provided.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Compliment to the authors. You put a lot of effort into restructuring and rewriting the paper. The paper shows that the authors have a very good and deep understanding of the operational problem and the ATCOs tasks. The conclusions are clear and understandable.

From the human factors point with the added information and details the experiments, studies and results are comprehensible, clear and allow to relate this research to the existing human factors research.

There are some minor issues, e.g. the text mentions that the questionnaire shown in Table 7 and 8 are rated on a 1-5 Likert scale, but the table indicates a rating from 0 to 5.

In Figure 7, questions 24 to 27 are mentioned but these are not described in tables 7 and 8. Furthermore, the charts show an average rating of 6 for WL-Q22, but the scale only reaches from 1 to 5. How can such a value be achieved?

Author Response

Dear reviewer, we truly appreciate your praise. Indeed, thanks to your comments, we intend to increase the quality of our submission. We have reviewed the document again, collecting minor errors, typos and cleaning up details.

Author Response File: Author Response.docx