Holographic Mixed Reality System for Air Traffic Control and Management
Round 1
Reviewer 1 Report
The authors present a novel concept to use mixed reality for air traffic control in the airport environment.
The paper describes accurately the work performed by the authors and it is original.
The clarity of the paper shall be improved based on the comments reported below.
One major comment concerning the inclusion of a few more relevant and recent publications in the literature review is reported as well.
Line 12 - explain the acronym ATC/M (Air Traffic Control and Management)
Line 17 - replace “complicated” with “complex” and apply this to the whole paper when referring to data or information
Line 22 - replace “head-mounted display (HMD) MR display” with “head-mounted see-through display”
Line 30 - ICAO is a very well known organization so I suggest to remove the reference to ICAO website and modify the reference list accordingly
Line 36 - replace “liquidity” with “flow”
Line 62 - delete “and noise” as it doesn’t influence visibility
Major comment:
Review the lines from 116 to 135 considering the following additional references:
Bagassi, Sara; De Crescenzio, Francesca; Piastra, Sergio, The use of synthetic vision tools in the control tower environment : the RETINA concept, in: 31st Congress of the International Council of the Aeronautical Sciences, ICAS 2018 : September 9-14, 2018, Belo Horizonte, Brazil, International Council of the Aeronautical Sciences, 2018, 143115, pp. 1 - 8
Ellejmi M., Bagassi S., Piastra S., Persiani C.A., Evaluation of augmented reality tools for the provision of tower air traffic control using an ecological interface design, in: 2018 Modeling and Simulation Technologies Conference, Reston, VA, American Institute of Aeronautics and Astronautics Inc, AIAA, 2018, pp. 1 - 10
Line 138 - add details about the operational scenario (i.e. Jeju airport) such as ILS category (likely is ILS CAT II), traffic density (high-medium-low), number of runways…
Line 168 - replace “we came to understand” with “it can be stated”
Line 206 - replace “squawk” with “transponder”
Line 217 - avoid the massive use of the first person “we”. For instance, the sentence “we created a 3D height map” should be turned in “a 3D height map was created”. This suggestion applies to the whole paper.
Line 249 - replace “is one of the sole mediums” with “is the main medium”.
Line 267 - replace “To give” with “to timely give”
Line 268 - delete “in a timely manner”
Line 342 - delete the sentence starting with “let” and delete equation 10 as it is pretty obvious
Line 344 - The sentence should be turned into “Later, the calculated magnitude, i.e. the control tower-airplane distance, was downscaled…”
Line 359 - delete “was”
Line 360 - delete “by us”
Line 386 - delete “very”, add “timely” between “to” and “convey”
Line 387 - delete “in a timely way”
Lines 414-420 - replace “identify” with “analyze” when referring to accident sequences
Line 433 - replace “air data” with “the proper information”
Line 445 - replace “complicated” with “complex”
Author Response
Reviewers’ Comments to Author:
Reviewer: 1
Line 12 - explain the acronym ATC/M (Air Traffic Control and Management)Line 17 - replace “complicated” with “complex” and apply this to the whole paper when referring to data or information
Line 22 - replace “head-mounted display (HMD) MR display” with “head-mounted see-through display”
Line 30 - ICAO is a very well known organization so I suggest to remove the reference to ICAO website and modify the reference list accordingly
Line 36 - replace “liquidity” with “flow”
Line 62 - delete “and noise” as it doesn’t influence visibility
We have changed all.
Major comment:
Review the lines from 116 to 135 considering the following additional references:Bagassi, Sara; De Crescenzio, Francesca; Piastra, Sergio, The use of synthetic vision tools in the control tower environment : the RETINA concept, in: 31st Congress of the International Council of the Aeronautical Sciences, ICAS 2018 : September 9-14, 2018, Belo Horizonte, Brazil, International Council of the Aeronautical Sciences, 2018, 143115, pp. 1 - 8
We have added the mentioned previous work along with the benefits of our system over existing method at Line number 126 to 140 as follows.“The previously proposed pipeline for augmented reality systems overlaid real windows with visual information through head-up displays [16]. Only a mathematical formulation was provided and no real-time or simulation-based experimental results were provided in the proposed work at that time. Later in 2018, under ‘The RETINA Concept’, they validated the concept of superposing of synthetic overlay on outside tower view with flight’s information [21]. The concept proposed in this method was not applicable to view air traffic from remote locations in the case of bad weather conditions. Whereas, our method provided entire view of airfield in holographic mixed reality along with the important information tagged with each flight. In this way, our proposed method dealt with low visibility conditions as well as monitoring the air traffic from remote locations. Moreover, method presented in [21] facilitated the controllers in low visibility conditions but did not reduce overload for air traffic controllers to interact with multiple computer screens for air traffic management. Most of the research has only been done on the visualization of air traffic, but not the interaction mechanisms, whereas in our work, along with the improvement in the visualization of air traffic, we enabled air traffic controllers to easily switch between multiple interfaces and interact with them.”
Ellejmi M., Bagassi S., Piastra S., Persiani C.A., Evaluation of augmented reality tools for the provision of tower air traffic control using an ecological interface design, in: 2018 Modeling and Simulation Technologies Conference, Reston, VA, American Institute of Aeronautics and Astronautics Inc, AIAA, 2018, pp. 1 - 10 We will investigate in the future.
Line 138 - add details about the operational scenario (i.e. Jeju airport) such as ILS category (likely is ILS CAT II), traffic density (high-medium-low), number of runways… We have added details about the operational scenario at Line number 154-156 as follows.
“This study proposes a new air traffic control system. Prior to system development, Jeju International Airport was selected to extract sample data. The Jeju International Airport is considered among the airports with high traffic density. It has for runways. But, only two of them are functional for ILS CAT Ⅰ, CAT Ⅱ operations.”
Line 168 - replace “we came to understand” with “it can be stated”
Line 206 - replace “squawk” with “transponder”
We have changed all.Line 217 - avoid the massive use of the first person “we”. For instance, the sentence “we created a 3D height map” should be turned in “a 3D height map was created”. This suggestion applies to the whole paper. We have applied to the whole paper.
Line 249 - replace “is one of the sole mediums” with “is the main medium”.
Line 267 - replace “To give” with “to timely give”
Line 268 - delete “in a timely manner”
Line 342 - delete the sentence starting with “let” and delete equation 10 as it is pretty obvious
Line 344 - The sentence should be turned into “Later, the calculated magnitude, i.e. the control tower-airplane distance, was downscaled…”
Line 359 - delete “was”
Line 360 - delete “by us”
Line 386 - delete “very”, add “timely” between “to” and “convey”
Line 387 - delete “in a timely way”
Lines 414-420 - replace “identify” with “analyze” when referring to accident sequences
Line 433 - replace “air data” with “the proper information”
Line 445 - replace “complicated” with “complex”
We have changed all.Author Response File: 
Author Response.pdf
Reviewer 2 Report
The manuscript presents an air traffic control and management system using holographic mixed reality technology. It is an interesting topic for this journal.
Overall, I could agree with the necessity of new air traffic control and management system, and well-arrangement of existing outcomes described in Chapter 1 and 2. However, to tell the truth, the proposed system described in Chapter 4 is boring. Because the developed MR applications were just used by existing components. What are the unique points as academic research? And how to verify/validate them?
Verification and validation sections described in Section 5 are also weak to prove your proposed system though there are many recommendations for the future described in the Conclusion section.
"The ultimate goal of mixed reality is to reinforce user recognition 49 and interaction with the real world [3].": The author mentioned "mixed reality" but the reference title includes "augmented reality". The differences between mixed and augmented reality should be defined in advance.
Author Response
Reviewer: 2
Overall, I could agree with the necessity of new air traffic control and management system, and well-arrangement of existing outcomes described in Chapter 1 and 2. However, to tell the truth, the proposed system described in Chapter 4 is boring. Because the developed MR applications were just used by existing components. What are the unique points as academic research? And how to verify/validate them? We assessed the interaction paradigms for ATC/M in mixed reality. Our study focused on supporting the ATC/M tasks by providing the 4D holographic view of airfield and interaction mechanisms in mixed reality. As holographic view provides the 3D spatial view of virtual objects, it is easy to interpret the traffic information and keep the user more involved. Furthermore, this study focused on how the use of latest display technologies could improve the workflow of ATC/M. Finally, with the help of conducted user test with the former air traffic controllers we tried to validate our study. Verification and validation sections described in Section 5 are also weak to prove your proposed system though there are many recommendations for the future described in the Conclusion section. We have added usability test result in Line 402-426 with Table 2, Figure 10 as follows.“In emergency situations, it is necessary to timely convey a message to the air traffic controller about the emergency with all information on the flight. The main purpose for creating emergency scenarios in flights tests was to analyze the performance of the system in emergency situations. With the help of 3D spatial sound and the interface of the proposed system presented in Section 3.6, it sent the user an alarm about the emergency situation and location of the airplane at the right time.
Due to safety reasons, it was difficult to evaluate the system’s performance in control tower with the real time air traffic. Hence, we used the pre-recorded as well as specially designed scenario based flight trajectories to test our system in controlled environment. For this testing, we invited two experienced former air traffic controllers who visualized flight trajectories and interacted with the system to assess the system's usability.
The proposed system was deployed on Hololens and tested by two former air traffic controllers with average age of 62 years. Due to their unfamiliarity with Hololens device, we demonstrated the way of using it to them and later executed the test.
The usability test mainly encompassed the four areas i.e. usefulness, ease of use, ease of learning and satisfaction. While designing the questionnaire, ease of usability of Hololens was also taken into account as shown in Table 2.
The results of the conducted user test are as follows:
As shown in Figure 10, the category of 'Usefulness' in the questionnaire, achieved higher score than the remaining three categories. This is likely to be associated with the Holographic Device's view and usability, which is scored 71.2. During the experiments, subjects pointed narrow field of view, unfamiliar interactions, and so on. Subjects were unfamiliar with Hololens device's gesture input method and found it difficult to learn. These limitations of the device may have affected the system usability results. “
"The ultimate goal of mixed reality is to reinforce user recognition 49 and interaction with the real world [3]": The author mentioned "mixed reality" but the reference title includes "augmented reality". The differences between mixed and augmented reality should be defined in advance. We have defined about mixed reality in Line 51-60 with Figure 1 as follows.“The ultimate goal of mixed reality is to reinforce user recognition and interaction with the virtual objects also termed as ‘Holograms’ using head mounted display in 3D real world space [2]. We often confuse the terms augmented reality and mixed reality, as both augments the real world with virtual objects. Mixed reality is the extended form of augmented reality. Augmented reality uses handheld flat LCD screens to augment real world with virtual objects and shows the indirect view of real world. Whereas, in mixed reality, it provides hybrid environment by combining real and virtual environment into single environment. Here, physical and virtual objects coexist in hybrid environment and provide direct view using head mounted holographic display. Continuum of reality to virtuality is shown in Figure 1.”
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
Validation parts were added. However, the testees were only two (Line 420). Are the average points valid by two testees shown in Figure 10 as a statistical approach? What is the detailed feedback from the two former air traffic controllers?
The submitted manuscript is with comments on the right side. A comment is written by a Korean word then I can not understand the sentence.
Author Response
1. Validation parts were added. However, the testees were only two (Line 420). Are the average points valid by two testees shown in Figure 10 as a statistical approach?
1. I agree with the reviewer's concern. So I have changed Figure 10. In order to reflect the overall opinion of each expert about every categorical measure in questionnaire, the results were analyzed against each individual instead of going with the average result. As shown in Table 2, a total of 17 questions were mainly divided into four different categories in order to cover maximum aspects of the system i.e. Usefulness, Ease of Use, Ease of learning, and Satisfaction.
2. What is the detailed feedback from the two former air traffic controllers
2. We have added participants’ feedback in line 428-460 as follows:
“User test came up with positive feedback about applicability of the system along with the constructive criticism to enhance the system. They also recorded their points of view about the hardware used. Overall feedback of each expert is as follows:
<Participant 1>
As a ‘Tower cab’ too, it is a good way to monitor the work of the tower team. Narrow field of view (FOV) of Hololens makes it difficult to analyze wider area which may results in difficulty to identify objects. While the flight had been in progress, detailed information i.e. altitude, speed of aircraft etc. were shown along with aircraft. However, it was suggested to show the required information when it’s needed.<Participant 2>
It would be good to use it for simulation of flight operations as it enhanced the analysis power pretty well. Narrow FOV of Hololens was a problem. Interaction was quiet difficult and extensive training about how to use Hololens is required.Overall feedback from the experts can be seen as both positive and negative. Positive comments are expressing new accessibility dimensions possible with proposed system, which are as follows:
To begin with, the Tower Cab in the control tower can be used for real-time monitoring of control tasks performed by the entire team. Through monitoring, it will be possible to detect work situations quickly and respond efficiently to emergency situations in less time. Secondly, it can also be used as an educational tool to teach and train students for operations related to air navigation. Proposed system is developed by the use of data which is actually used by real controllers. Hence, it is possible to support simulation control work through this system.
On the other hand, negative feedback was also received from the experts and it was mainly related to the limitations in ‘Hololens’ device, which we’re already worried about. On first hand, there was an opinion that the narrow FOV made the overall system inconvenient to use. The narrow viewing angle is a hardware related problem and can be easily solved by using the newly developed mixed reality head mounted device (HMD) with the recent advances, as one developed by Microsoft [27]. The new Hololens offers almost double FOV as compared to previous version of Hololens. Second opinion was the difficult and unfamiliar interaction. Initially, gesture based interactions provided by Hololens may be unfamiliar to the user. But, this problem can be partially solved by expanding speech recognition in the future. Complementing the interaction and providing selective information will be carried out in future work.”
3. The submitted manuscript is with comments on the right side. A comment is written by a Korean word then I cannot understand the sentence.
3. It was a simple note for editing, deleted and submitted.
Author Response File: Author Response.pdf
Round 3
Reviewer 2 Report
The authors have modified well corresponding to the reviewer's comments. I am looking forward to seeing the authors' further development in the near future.
