Establishing Training and Certification Criteria for Visual Observers of Unmanned Aircraft Systems

Abstract

Safe integration of Unmanned Aircraft Systems (UAS) into airspace generally occupied by manned aircraft and other aviation stakeholders is a pressing global challenge. In the United States, efforts are being made to integrate small and large UAS into the National Airspace System (NAS). Whereas regulations for the civil operation of small UAS (25 kg and lighter) have already been adopted, those for larger unmanned systems are still being crafted. Thus, a two-part mixed methods study was conducted to examine three pivotal issues in the safe operation of large UAS: (1) What kind of visual observer skills are needed to execute safe UAS operations; (2) Should visual observers involved in UAS operations receive formal training; and (3) Should visual observers be required to pass a certification exam? In the first phase, subject matter experts identified various vigilance, trajectory estimation and communication skills that were vital to performing visual observer duties successfully and elaborated on their training regimens. In the second phase, survey participants were approximately evenly split on the need for formal classroom/online and hands-on visual observer training. Furthermore, participants generally favored visual observers having to pass a classroom/online certification exam, whereas they were against a practical (hands-on) exam.

1. Introduction

The establishment of visual observer training and certification requirements for unmanned aircraft system (UAS) operations is a component in ensuring the safe and efficient integration of UAS into the U.S. National Airspace System (NAS). As stated in the Federal Aviation Administration’s (FAA) Unmanned Aircraft Systems (UAS) Operational Approval policy notice [1], visual observers are expected to be responsible for: (1) Helping UAS pilots keep the aircraft within visual line of sight and (2) Exercising see-and-avoid responsibilities by preventing the unmanned aircraft from creating a collision hazard and maintaining compliance with federal regulations, including Operating Near Other Aircraft [2], Right-of-Way Rules: Except Water Operations [3] and Right-of-Way Rules: Water Operations [4]. So, to ensure the safety of UAS operations, visual observers must be able to visually scan the airspace effectively, track aircraft and make accurate and reliable estimates of (relative) aircraft position, assess the need for a potential avoidance maneuver and communicate that need to the UAS pilot in a timely manner [5].

In 2015, the FAA released a Notice of Proposed Rulemaking for civil UAS operations [6]. The proposed language stated that flights were to be limited to small UAS (sUAS) operated within visual line of sight for the sUAS pilot during daylight hours in visual meteorological conditions. Some simple operational tasks, like flying a predefined route in dedicated airspace, allow sUAS pilots to maintain the aircraft in their line of sight throughout flight. However, more complex operational tasks, like flying an ad-hoc route in shared airspace, typically require the pilot to actively maneuver the aircraft. This can result in fewer opportunities to maintain visual contact with the sUAS and perform see-and-avoid tasks, and, consequently, reduced situation awareness and increased cognitive load. So, in the following scenarios the Notice of Proposed Rulemaking suggested that sUAS operations would necessitate a visual observer [6,7]:

• Operations conducted above 500 feet above ground level or beyond 1500 feet laterally from the pilot-in-command must have at least one dedicated visual observer, even if visual line of sight can be maintained by the pilot-in-command.
• When the pilot-in-command determines that one or more visual observers are necessary for the flight crewmember to maintain the safety of the operation.
• When the pilot-in-command is (expected to be) in a “heads-down” or any situation that precludes the ability to perform see-and-avoid duties.
• When the pilot-in-command is within an enclosure, at least two visual observers are required.

The sUAS Notice of Proposed Rulemaking also specified the following guidelines for visual observers [6,7]:

• In order to be certified, applicants for a visual observer certificate would be required to pass a practical test with either a certified sUAS pilot or instructor.
• Medical standards and operational limitations in this proposed rule ensure that the pilot and visual observer are capable of scanning the airspace of intended operations. Aids to vision, such as binoculars, must be used with care to ensure that the total overall viewing of the airspace is not inadvertently limited. An FAA second-class medical certificate is required for commercial operations.
• The pilot-in-command must be able to see or ensure that a visual observer is able to see the aircraft throughout the entire flight well enough to: Know its location, determine its attitude and direction to exercise effective control, observe the airspace for other air traffic or hazards and determine its altitude.
• The visual observer will be required to always know where the sUAS is and to discern the attitude and trajectory in relation to conflicting traffic, obstacles, or inclement weather. Because of the level of vigilance that would be required in scanning the surrounding airspace, a visual observer would be prohibited from supporting more than one aircraft operation at a time.
• The visual observer will need to be in close proximity (within 10 feet) to the pilot-in-command and should be able to communicate directly, exchange non-verbal signals and share the same relative visual references. A backup communications system is required for operations where the pilot-in-command is in an enclosure and cannot directly see at least one visual observer.
• Operations that involve the hand-off of UAS control and/or see-and-avoid duties among multiple pilots and/or visual observers, sometimes referred to as ‘daisy-chain,’ ‘relay,’ or ‘leap-frogging,’ would not be authorized under this proposed rule.

In 2016, many of these guidelines were reiterated in the FAA’s Small Unmanned Aircraft Regulations [8] for operations in the NAS. Similar to the Notice of Proposed Rulemaking, these regulations state that flights are limited to sUAS (25 kg or less) operated within visual line-of-sight in visual meteorological conditions. However, the proposed operational ceiling was lowered by 30 m and, notably, all certification requirements for visual observers were dropped. This has resulted in a potential regulatory gap.

Moreover, while systems larger that 25 kg are allowed to operate in the NAS with special certification from the FAA, general guidelines for such systems are still under development. Much care must be taken with this process because operating larger unmanned platforms inherently carries more risk due to the increased momentum of the aircraft and sturdier construction compared to their smaller counterparts. In addition, larger UAS platforms are flown with the pilot-in-command inside an enclosure much more frequently. Thus, it follows that flying such aircraft in the NAS will require visual observers to maintain visual line of sight in a variety of operational scenarios and settings. Furthermore, the added risk in such operations provides a clear impetus to standardize visual observer training and establish performance benchmarks such that visual observers can become certified to serve in the operation of large UAS (heavier than 25 kg). Although prior experimental work has examined visual observers’ performance in field settings (e.g., [5]) and unmanned crewmember aviation training studies exist (e.g., [9]), no prior research has investigated training and certification of visual observers.

Thus, a two-phase mixed methods study was conducted to address this timely and complex issue. Phase 1 was a phenomenological study that consisted of guided subject matter expert interviews. Phase 2 was a descriptive study which consisted of a broad survey of aviation stakeholders that focused on two pivotal issues: Should visual observers receive formal training and should visual observers be required to pass an exam?

2. Materials and Methods

The initial phase of this research (New Mexico State University Institutional Review Board #12314) was a phenomenological inquiry and consisted of interviewing three subject matter experts who were licensed manned aircraft pilots and also performed the roles of UAS pilot, visual observer and mission commander. These individuals were selected due to their deep understanding of all parts of manned and unmanned aircraft operations in the NAS. The interviews lasted approximately 60 min and were guided by the research questions noted in Appendix A. These interviews were transcribed and coded to examine subject matter experts’ background, training, assessment of vital skills and technologies needed to perform visual observer duties, assessment of UAS operation risks in various conditions and assessment of various current and potential UAS regulations (see

Table A1. Research Question: What roles/duties have participants carried out in the context of manned and unmanned aircraft operations?

Open Code	Properties	Example
Manned Pilot	Certified military, commercial, or private manned aircraft pilot	“Yes, I am a licensed pilot.”
UAS Pilot	Certified military, commercial, private unmanned aircraft pilot	“I have acted as an internal pilot at the computer controls for flights and ... mission commander... and, also, I act as an observer.”
UAS Mission Commander	Mission commander for commercial (university) UAS operation	See previous.
UAS Visual Observer	Safety-related visual observer in UAS Operations	See previous.
UAS Flight Instructor	Flight instructor in UAS flight school	“I was hired by {company} as a flight instructor.”
Radio Controlled Aircraft Pilot	Recreational pilot of radio-controlled aircraft	“So, recreationally, I fly RCs. I’ve been flying since I was nine on that.”

,

Table A2. Research Question: What types of training and certification did participants receive prior to engaging in aviation duties?

Open Code	Properties	Example
Manned Pilot Training Program	Training program involves both classroom and field training	“I mean the pilot training was excellent and I had previously taken ground school before I did that. … So, the pilot training and just actually working with aircraft, the flight training, the hands-on stuff.”
Manned Certification	Licensed FAA pilot, certified military pilot	Experimenter: “And are you or were you a licensed pilot?” Participant: “I am.”
UAS Pilot Training	Training program involves both classroom and field training	“Of course with the Predator I did the pilot training some of that included mission training because to fly the airplane you have to be able to work with the payload operator, move the aircraft around so the payload guy can see what he’s looking at, the communications, airspace, all the things that go into operations in a war zone.”
UAS Pilot Certification	Military, FAA and manufacturer certification	“Airforce check ride. Ground check ride and then a flight check ride.”
UAS Mission Commander Training	Training program primarily involves classroom/manual-based training	“… with the Predator, I did the pilot training some of that included mission training …. So, that carried through my CFI training which is also safety based. Then to get to the university, since I was trained at least going through the courses for each aircraft I knew what the safety hazards were …. So, I just threw all those things together in order to be able to be mission commander.”
UAS Visual Observer Training	Training program involves both classroom and field training	“We have PowerPoints then we have actual on the job training …. So, I can’t really think of anything that’s rocket science and you have got to teach a visual observer. If they’ve got decent hearing and decent eyesight and they can stay vigilant and communicate clearly that’s all there is to it.”
UAS Visual Observer Certification	No certifications were noted	“Section 333, as of right now, there’s no requirements for the visual observer. It has no age requirement, has no training requirement, has no minimum …”

,

Table A3. Research Question: What aspects of training were most beneficial?

Open Code	Properties	Example
Apprenticeship	Being trained by an experienced instructor / pilot	“… a new guy gets back from Iraq who’s been over there for 3 or 4 months and he’s kind of burned out and tired and they put him in with these students but they were the best people with all the information. Not just what the book said but what really happened what really worked.”
Hands-on Training	Practical training with real aircraft	“Hands on training for me I think is the most valuable being able to fly the aircraft safely and being able to decide whether it’s safe or not to operate an aircraft because sometimes proponents don’t know when to take a step back and think about safety.”

,

Table A4. Research Question: What aspects of training were lest beneficial?

Open Code	Properties	Example
Non-practical Testing	Classroom or computer-based testing	“Paper tests. Performance tests are useful vs. paper tests. Knowledge tests; I understand they’re important but sometimes there is too much weight put on them especially for people that don’t take test well. And there are a lot of people in high stress situations in regards to test taking that just don’t do well.” “… they required us to take weekly quizzes that was fine. An open book quiz annually that was harder than it sounds because they ask very obscure questions you had to go through a bunch of books to find it, that was fine. But then they gave us a really really difficult instrument exam every year and I’m an instrument instructor and they gave us an instrument exam that had nothing to do with reality or flying UAV’s or anything and it was a thing everybody had trouble with and never needed. So, it was very frustrating and a waste of time.”

,

Table A5. Research Question: What are essential skills/knowledge for visual observers?

Open Code	Properties	Example
Effective See-and-Avoid	Visually tracking aircraft; Disengaging to scan for other air traffic and re-engaging; Identifying potential near midair collisions; Orienting using local landmarks and cardinal directions	“If you’re just doing regular commercial operations, then yeah I mean they can be within line of sight… You look around if you see an airplane, [then you estimate] distances and position. Whether you feel it’s going to be a factor or not to the flight of the aircraft do we need to divert or not. That’s really about it.”
Effective Communication	Communicating with pilot in person or via radio; Using global and local landmarks to relay aircraft position	“How to operate the radios, what you’re looking for…’Here’s south, figure out where it’s coming from. If you’re not sure give someone a call.’” “As long as you can get the message across to the pilot to maneuver the aircraft in time then yeah that should be fine.”
Aircraft Operations Rules	Knowledge of airspace and FAA regulations	“The other day I was at the RC field and a guy was flying a parasail. Don’t need a pilot license, don’t need to have any training on airspace and he’s circling our RC field … and he’s down at like 150 feet but that’s safe according to the FAA. But flying a smaller UAS below that altitude, we have to have a pilot’s license.”

and

Table A6. Research Question: What aspects of UAS operations impact risk?

Open Code	Properties	Example
Momentum	Weight; Speed	“I thought that the 55-pound weight was too big for the speed that they’re allowed. That’s a lot of mass and can cause a lot of damage really fast and danger…. There’s no doubt that the training required to fly the Predator is a lot more extensive than what is required to fly a multi-copter. It was, it is and has to be.”
Components/Configuration	Fixed wing aircraft vs multi-rotor aircraft; Rotor guards	“I have a multi-copter and I’ve crashed it twice and in both cases I threw a propeller blade. I didn’t do anything wrong; I didn’t hit anything…. you’re just flying along and one blade came off. It didn’t hit anybody it just turned upside down … I’ve never had a wing come off an airplane, if it did that’d be catastrophic so you got to weigh that.”
Distance/Line of Sight	Ability of UAS pilots and/or visual observers to see the UAS and scan surrounding airspace; Weather; Time of Day; Proximity to Hazards	“We’re flying at low altitude and for our flights we generally have three observers. We have one on each end of the flight area and one would be external pilot. So, we have people around that are an extra set of eyes. And the external pilots found that even with the larger aircrafts they didn’t want it as far away as they could possibly see it. And then the radio controls too, the larger one is probably large enough that it could get out of range of the radio controls.” “So the size [matters], an Aerostar or Tigershark [has] got all the light systems on it with a strobe then the risk I don’t really foresee that risk being any different than during the day. But if you’re looking at something like the Phantom, it’s got lights on it, you can see it from the ground but is a pilot in the air going to see it? Nope. So large UAS, sure I don’t see any reason why a large system can’t fly beyond line of sight with light systems on it at night, as long as they’re equipped with transponders, ASB, something of that sort and well-lit but small systems to be honest there is no reason to fly them at night.”

).

The second phase (New Mexico State University Institutional Review Board #15418) of this research consisted of a brief descriptive survey of aviation stakeholders (see Appendix B) that was distributed in aviation forums, social networking site groups of NAS stakeholders and via email (snowball sampling).

3. Results

3.1. Phase 1

All subject matter experts reported that they underwent a combination of computer-based and practical hands-on training before engaging in visual observer duties. The lengths and particulars of the training programs varied but all were completed over a few days. All subject matter experts were licensed manned and unmanned aircraft pilots prior to visual observer training, thus the majority of the training was redundant with their prior knowledge.

Open-coding was performed on the interview transcripts (available in the Supplementary Materials); tables of the generated research questions and codes are noted in Appendix A. The subject matter experts unanimously stated that visual observers should be well-versed in FAA rules and regulations regarding manned and unmanned aircraft operations in the NAS. As noted in Table A5, subject matter experts stated that visual observers must establish proficiency in the following skills that are needed to successfully accomplish see-and-avoid duties:

• Tracking unmanned and manned aircraft in various lighting and meteorological conditions:

  1.

  Must be able to maintain visual-line-of-sight to the aircraft

  2.

  Must be able to re-engage visual contact after loss and/or distraction

• Scanning airspace for approaching air traffic:

  3.

  Must be able to shift visual depth of field

• Informing pilot of impending near midair collision or some other danger with enough time for the pilot to take appropriate action:

  4.

  Must maintain cockpit discipline

  5.

  Must use appropriate verbiage when communicating with the pilot

  6.

  Must be able to use global bearings and local landmarks to identify positions of UAS and other air traffic

  7.

  Must be able to estimate aircraft flight paths, altitudes and closure rates in order to determine the likelihood of a near midair collision

  8.

  Must be able to determine and communicate correct course of action and a safe deviation from the flight path to avoid a potential near midair collision

3.2. Phase 2

Phase 2 consisted of a broad survey of aviation stakeholders (see Figure 1 for participants’ roles in aviation). As shown in

Table 1. Yes/No counts for survey responses and results of statistical (χ²) tests in Phase 2. A 2 × 2 contingency table was constructed for each question, comparing the observed number of Yes and No responses to the expected values of 56 Yeses and 56 Nos.

Should Visual Observers:	Yes	No	χ2 Value	p Value
(1) Receive formal classroom/online training?	58	54	0.07	0.71
(2) Receive formal practical (hands-on) training?	55	57	0.16	0.57
(3) Take a formal classroom/online exam?	61	51	0.64	0.26
(4) Take a formal practical (hands-on) exam?	43	69	5.79	0.00006

, participants were approximately evenly split when answering the questions of whether visual observers involved in operations of UAS 25 kg and heavier should receive formal classroom/online and hands-on training (survey data available in the Supplementary Materials). While participants favored having to pass a formal classroom/online exam, the difference in Yes/No response counts was not significant. Furthermore, participants were generally against a formal practical exam; this was the only significant difference identified by χ² (chi-squared) tests.

Multinomial logistic regression and maximum likelihood ratio tests were then conducted to determine whether participants’ roles in aviation and age predicted their responses to the questions in Table 1. The models could not significantly predict Yes/No responses to the first three questions. However, participants’ roles did significantly predict responses to the final question of whether visual observers should take a formal practical exam (χ²(12) = 25.4, p = 0.013) and accounted for approximately 28% of the variance (Nagelkerke pseudo R² = 0.28). More specifically, commercial UAS pilots (χ²(1) = 3.6, p = 0.058) were marginally more likely to respond ‘No,’ whereas air traffic controllers (χ²(1) = 3.2, p = 0.073) and aircraft technicians (χ²(1) = 8.1, p = 0.004) were significantly more likely to say ‘Yes.’ Furthermore, older participants were marginally more likely to respond ‘Yes’ (χ²(1) = 3.55, p = 0.06).

Fewer than half of the participants provided reasons for their Yes/No answers. When examining the reasons for why participants felt that training and examination were necessary, participants noted the usefulness of classroom training in understanding NAS regulations and manned/unmanned aircraft operations, as well as increased risk due to platform size. For example, one participant stated, “Visual observers must be familiar with aeronautical information and performance standards in order to judge performance and assess hazards properly.” Another stated, “Knowledge of airspace, meteorology and human factors and crew resource management will help make for a more competent and safe crew.” Those participants which felt that practical training was needed stated that hands-on knowledge is irreplaceable. For example, one participant noted that, “(Visual observers need to) practice acquiring and re-acquiring UAS at different distances and altitude, (and to) understand how to determine the attitude and distance from the (ground control station) of the UAS,” and another said, “(Hands-on training is needed) because practical lessons provide a more realistic perspective to real world situations.”

When recommending training regimens, participants mainly suggested online/print materials, such as the sUAS airmen certification standards manual [10], the ALC-451: small Unmanned Aircraft Systems (sUAS) online course [11], the Aeronautical Information Manual [12] and commercially-available visual observer self-study guides. One participant that thought practical training was necessary mentioned simulator training as an option. Furthermore, another participant stated, “(Hand-on testing should be a) continuation of the formal practical training, another day would be the test day, where a spotter would be paired up with an operator who will then ensure the spotter goes through the list of operations correctly and may even test them by trying something not allowed.”

4. Discussion

To ensure safe UAS operations, visual observer duties primarily require them to assist UAS pilots in effective see-and-avoid. To date, visual observer training regimens have yet to be standardized and feature both practical and classroom/computer-based components. As Williams and Gildea [7] note, there are no adequate human perceptual models of aircraft detection, primarily due to the large number of variables involved. Thus, given some acceptable level of visual competency (such as medical clearance), it is likely that a visual observer training regimen should focus on non-perceptual aspects of operations, such as rules, regulations, procedures and related matters. Although 14 CFR § 107 [8], the Small UAS Rule, did not specify a requirement for visual observer certification, visual observer certification may be appropriate for personnel engaged in operations involving larger UAS, which carry added risk.

While aviation stakeholders acknowledge the additional risk of operating a UAS larger than 25 kg, there is currently a lack of consensus regarding whether training and/or certification needs to be mandated. Although participants in Phase 2 somewhat favored classroom/online certification, the difference in Yes/No counts was not statistically significant. However, this trend is indicative of numerous factors that make operations of such aircraft more dangerous than their lighter counterparts. Notably, significantly more participants thought that practical (hands-on) testing was not necessary than those that did. Moreover, while UAS and manned aircraft pilots were well represented in the sample, other NAS stakeholders were not. This limitation underlines the need for additional research in this domain that carefully considers the needs of all those that occupy the sky.

Furthermore, due to the human factors of UAS operations [5,7,13,14], one training and certification solution may not be optimal. Licensed pilots have already mastered many, if not all, of the essential skills needed to carry out visual observer duties. In addition, licensed pilots have already been trained and certified in their knowledge of rules and regulations pertaining to operations in the NAS. Likewise, manned aircraft pilots have a multitude of hours performing seek-and-avoid duties, are well-versed in aviation verbiage and are trained in maintaining proper cockpit discipline. Thus, it appears that pilots and non-pilots should require different training regimens and certification criteria, akin to the distinction between licensed pilots and non-licensed individuals made by the FAA when persons apply for sUAS certification.