Human Factors as Predictor of Fatalities in Aviation Accidents: A Neural Network Analysis

Abstract

In the area of aviation safety, the importance of human factors is indisputable. This research endeavors to assess the importance of human factors in predicting fatalities during aviation mishaps. Utilizing reports from the Aviation Safety Network Database, encompassing 1105 accidents and incidents spanning from 2007 to 2016, neural networks were trained to forecast the probability of fatalities. Our findings underscore that the human factors involved, by themselves, can yield strong predictions. As a term of comparison, other variables (type of occurrence, flight phase, and aircraft fate) were used as predictors, with poorer results; by combining these variables with human factors, the prediction is only marginally better, if at all, than that based on human factors alone. So, although these supplementary variables can marginally benefit the predictive results derived from human factors, their contribution remains minimal. Consequently, this study illuminates the paramount importance of human factors in influencing aviation fatalities, guiding stakeholders on the immediate interventions and investments which are most warranted to prevent them.

1. Introduction

Air transportation is one of the most reliable means of transportation, contributing significantly to social and economic development on a global scale. All the stakeholders in this highly regulated market have a strong interest in preventing operational hazards, thereby increasing the safety of persons and goods. The number of accidents and incidents has significantly decreased over the years [1], both because of technological improvements, and due to the introduction of policies, processes, and regulations to deal with human factors, organizational factors, and state safety programs. The evolution of the relative importance of technical and human causes—these encompassing human factors, organizational factors, and state safety programs—is depicted in Figure 1. Of course, accidents seldom have only one cause, being more often than not the result of a concurrence of causes. Nevertheless, for a long time, human-related errors have been the main cause of accidents in the air transport industry [2,3]. Consequently, avoiding accidents means addressing human errors and the factors that may cause them [4]. This is why the study of human factors is important to decrease accidents and incidents in this industry [3,5].

In this paper, data taken from reports of aviation accidents and incidents over the 2007–2016 are used to predict fatalities. This is carried out using different artificial intelligence techniques, in particular, neural networks and random forests. The importance of human factors in the prediction of fatalities is highlighted. While human factors have been shown in the literature to allow for the prediction of fatalities [7], our contribution shows that they are the best available predictor, and that neural networks are the best performing tool.

The remainder of the paper is organized as follows: Section 2 presents the state of the art of the relation between human factors and aviation safety; Section 3 describes the database of accident and incident reports that was used and the extraction of information from these reports; Section 4 describes the neural networks with which fatalities were predicted for each case; and the results are given in Section 5. Finally, Section 6 concludes the paper with a discussion of the results and of what is work still to be carried out in this area.

2. State of the Art

The study of human factors in aviation safety has evolved significantly over the years. Early research was primarily focused on individual errors, often attributing accidents to pilot mistakes. However, a paradigm shift occurred with seminal works such as Reason’s study on human errors, which emphasized a systems approach to understanding the root causes of accidents [8]. One of the early methodologies that gained prominence is Crew Resource Management (CRM). CRM aims to improve communication, leadership, and decision making among flight crews. Studies have shown that CRM training is effective in reducing errors attributable to human factors [9]. Recent advancements in technology have also contributed to the depth of this field. For instance, eye-tracking technology is increasingly being used to study pilot attention and situational awareness. Research in this area has shown that technology such as eye-tracking can provide valuable insights into cognitive processes, thereby aiding in the design of more ergonomic cockpits and effective training programs [10]. The International Civil Aviation Organization (ICAO) has been instrumental in shaping the regulatory landscape. ICAO’s Safety Management System (SMS) [11] has set the global standard for proactive hazard identification and risk management. The ICAO’s Document 9859 [6] provides comprehensive guidelines that have been adopted globally, emphasizing the importance of human factors in safety management systems. Different organizations and regulatory bodies have adopted various approaches to tackle human factors in aviation safety. While the ICAO focuses on a broad systems approach, entities like Boeing have specialized initiatives targeting specific aspects such as cockpit design and human–machine interfaces. A comparative analysis of these approaches reveals that no single methodology is universally effective; rather, a multi-faceted approach is often the most effective strategy for mitigating risks associated with human factors [12].

According to the International Civil Aviation Organization (ICAO), aviation safety is “The state in which risks associated with aviation activities, related to, or in direct support of the operation of aircraft, are reduced and controlled to an acceptable level” [6]. Since it is not possible to eliminate risk, the ICAO promotes a process of incorporating defenses, preventive controls, or recovery measures to lower the severity and/or likelihood of a hazard’s projected consequence such as an accident or incident. This process is widely adopted by the aviation ecosystem and encompasses the relation of the Risk Probability of occurrence vs. the Risk Severity of the event. This methodology can be illustrated in a matrix [6,13], like the one in

Table 1. Risk matrix showing different levels of risk acceptability in different colors (adapted from [11]).

Risk Probability	Risk Severity
	Catastrophic A	Danger B	Major C	Minor D	Insignificant E
Frequent—5	5A	5B	5C	5D	5E
Occasional—4	4A	4B	4C	4D	4E
Remote—3	3A	3B	3C	3D	3E
Improbable—2	2A	2B	2C	2D	2E
Extremely improbable—1	1A	1B	1C	1D	1E

.

Based on Table 1, special programs can be created to deal with errors and hazards behaviors. One of the mainstream programs is the already mentioned ICAO Safety Management System, commonly referred to as Annex 19 [11]. The objective of Safety Management Programs is to analyze the environment and build defenses against errors and hazards. These defenses are then responsible for creating specific mitigating actions, preventive controls, or recovery measures, putting them in place to prevent a hazard or its escalation into an undesirable consequence.

Human errors have been implicated in most aviation accidents [3] and are defined by the ICAO as follows: “Human Factors is concerned to optimize the relationship between people and their activities, by the systematic application of human sciences, integrated within the framework of systems engineering” [14]. Thus, it can be said that human factors in aviation represent an interdisciplinary sphere that aims to elucidate the intricate interactions between aviation personnel, the environments they operate in, and the equipment they utilize, with the ultimate objective of improving safety, performance, and well-being within the aviation sector. The root causes of all factors can be reduced to twelve. These twelve root causes are very well known as the “Dirty Dozen” and are an aviation standard and field of study by themselves. These Dirty Dozen causes delineate twelve prevalent factors that have been empirically shown to contribute to human error in the context of aviation. They are as follows: lack of communication, complacency, lack of knowledge, distraction, lack of teamwork, fatigue, lack of resources, pressure, lack of assertiveness, norms, lack of awareness, and stress [1,11,15,16,17]. By methodically identifying and addressing the elements within the Dirty Dozen, professionals in the aviation industry are better positioned to foster a culture oriented towards safety, thereby reducing the propensity for errors, and substantially enhancing operational safety metrics. This framework not only serves as an instrumental tool for training initiatives but also facilitates self-assessment procedures, laying the groundwork for the formulation of risk mitigation strategies pertinent to human factors in aviation. Through ongoing scrutiny and comprehension of these twelve critical elements, the aviation sector is poised for progression toward a more secure and efficient operational paradigm.

While every human, even with the best training, is of course always liable to make mistakes [18], many players have suggested different ways of reducing the impact of human factors. The ICAO has provided efforts to improve organizational structures and human management [11]; Boeing addressed visual perception, ergonomics, and the human–computer interface [14,19]; organizational safety culture was addressed in [20]; a constant monitoring of routine operations is envisioned in [8]; an Aviation Maintenance Monitoring Process (AMMP) is suggested in [21]; and a bibliometric analysis can be found in [22]. In [23], a probabilistic and statistical study of aeronautical accidents is carried out, in which the authors, using autoregressive integrated moving average (ARIMA) models, sought to assess safety levels in the aviation sector, analyzing available data and accident reports and the different variables involved, which, through time series analysis, present forecasts of values and respective trends for the future. In [24], a review of problems linked to the mental health of pilots and aviation professionals is presented. The study sought to find answers from the players about the awareness or lack of mental health in the sector, and its criticality and importance for aviation safety, as well as the identification of the factors (fatigue, stress, anxiety, depression, and exhaustion) that most contribute to the deterioration of mental health, with a view to finding results to optimize aviation safety. The topic surrounding mental health has become more relevant since the accidents that occurred with a LAM flight in 2013 and a Germanwings flight in 2015, caused by pilots with mental health problems, and has been attracting the attention of researchers in a constant search for mitigating and preventive measures for aeronautical safety.

There are several factors that contribute to human errors continuing to be classified as the main cause of aviation accidents in recent years. The study in [25] statistically analyzed the trend in the evolution of air accidents, and the authors state that the lack of personnel is also one of the factors that compromises the smooth functioning of operations in the sector, as it overloads the professionals present, increasing the probability of errors due to the stress faced by other team members. And although human errors have been identified as the most significant cause of accidents in several studies over the years, the importance of technical and environmental factors in this issue is recognized as being responsible for a smaller proportion of serious accidents. Airlines have included in their reports occurrences of unsafe events with high frequencies but potentially considered low-severity incidents. The article in [26] presents a study that proposes a technical scheme for automatically classifying and identifying risk factors from Chinese civil aviation incident reports in an intelligent way in order to prevent the recurrence of such events. The authors state that unsafe events during a flight are always a major threat and concern for aircraft safety and can cause damage to the aircraft or even injury to personnel involved or on board. Flight safety and the sustainable growth of aviation also depend on the good performance of air traffic control (ATC). In [27], an analysis of human factors in air traffic safety is carried out, based on the human factors analysis and classification system—Bayesian network (HFACS-BN) model, in which some machine learning language techniques were used, to evaluate ATC performance. In [28], a statistical method is applied to detect anomalies in flight data using a hybrid machine learning language.

A useful image is that of the Swiss cheese model of accident causation developed by Reason in 1990 [8]. The Swiss Cheese Model is widely used in aviation safety management for its simplicity and ease of understanding. It provides a visual representation that helps in identifying potential weaknesses in a system and offers a basis for implementing corrective actions. The model has been instrumental in the development of various safety management systems, including the ICAO’s SMS [11]. This human factor model serves as a conceptual framework for understanding how errors occur in complex systems like aviation. This model likens organizational defenses against accidents to layers of Swiss cheese, where each layer represents a barrier or a safeguard. While individual layers may have holes or weaknesses, only the alignment of these holes across multiple layers leads to the occurrence of an accident. This means that in an organization, there are safeguards against accidents at several levels (organizational, supervision, etc.), and, hence, even if there is a failure at one of these levels, a safeguard at another level should prevent an accident. But if all safeguards fail at all levels, it is as if there were a hole at each layer, and the holes are aligned so that an accident can pass through the efforts to prevent it. Human factors can affect each one of these layers. Despite its widespread use, the Swiss Cheese Model has been subject to criticism. One of the main criticisms is that the model is overly simplistic and does not account for the dynamic interactions between different layers of defense. It also tends to focus on individual errors and does not adequately address systemic issues or organizational culture [29].

In this work, human factors are classified according to a general, comprehensive framework developed in [3], called Human Factors Analysis and Classification System (HFACS), shown in Figure 2, for the four different layers of the Swiss cheese model [3].

HFACS was developed for the field of aviation (both civil and military) and has been used to this effect [30], but has since been used in other industries as well, for instance, in the naval industry (in which it was used to show that most accidents are due to decision errors, the planning of inappropriate operations, and supervisory violations) [31].

3. Database

The Aviation Safety Network (ASN) database is a worldwide repository of information on airliner accidents and safety issues kept by a non-profit organization [32]. The data used in this paper correspond to all the 1105 accidents and incidents reported during a period of ten years, from 2007 to 2016. This period was chosen since it is the same as that used in [2,7]. For each accident or incident, available reports were used to find out which human factors were involved. The list of human factors employed and its numeration, adapted from [3] and its extension in [33] for maintenance activities, is also taken from [2,7] and is given in

Table 2. Human factors, labeled from the contributory causes in the database, with their respective frequency in the database.

#	Factor According to HFACS-ME	Number of Cases
1	Adverse mental state	73
2	Adverse physiological state	19
3	Group resource management	11
4	Dated/Uncertified Equipment	67
5	Decision Error	62
6	Exceptional Violation	20
7	Failed to Fix Known Issue	12
8	Inaccessible	2
9	Inadequate Design	42
10	Inadequate Documentation	36
11	Inadequate supervision	63
12	Inappropriate Operations	76
13	Infraction	1
14	Lighting	1
15	Operational process	12
16	Misperception	131
17	Personal Readiness	134
18	Physical environment	216
19	Physical/mental limitations	18
20	Plan for improper operation	1
21	Resource management	9
22	Routine	114
23	Routine Violation	39
24	Rule	72
25	Skill	29
26	Skill-Based Error	104
27	Supervisory violation	15
28	Technological Environment	19
29	Training	4
30	Problem not fixed	8

[2,7].

In addition to the human factors involved in each occurrence, three variables in the metadata of the reports were also obtained:

• Type of Occurrence—This variable captures the nature of what happened. Its possible values are accident, criminal event, incident, and other occurrence.
• Flight Phase—Different phases of flight have different risk profiles. For example, take-off and landing are often considered the most critical phases of flight. By including this variable, the study can identify which phases are most susceptible to human factors-related accidents, thereby allowing for targeted interventions. The nine phases considered are, in alphabetical order, approach, en route, initial climb, landing, maneuvering, pushback, standing, take-off, taxi, and unknown.
• Aircraft Fate—The aircraft may be repaired or be in such a condition that it is scrapped. This is consequently a binary variable.

4. Data Processing

4.1. Machine Learning

Among the available machine learning algorithms, there are several that can be used to predict fatalities. Neural networks are an obvious choice given that their performance in this type of non-linear modeling tasks is regularly satisfactory [34], that an active learning system (which can be used with a small initial set of labeled data and obtain further data as it runs [35]) is not needed since the data are labeled, and that data are sufficient to train a neural network. Concerning this last point, it is true that it would be desirable to have more than the 1105 incidents and accidents above mentioned; data augmentation techniques, routinely applied in several areas, are not suitable for this problem, and neural networks with many hidden layers (such as used in deep learning) are completely out of consideration. Furthermore, for databases of this size, tree-based models may present comparable results [36]. For this reason, each model was developed both with a neural network and with a random forest to verify which of these tools is more suitable.

4.2. Models Developed

In this paper, different models to predict fatalities were developed, using different inputs, aiming to answer the following question: given some variables about an incident or accident, and the human factors involved, is it possible to predict if there were fatalities? These models will be referred to as experiments and are illustrated in Figure 3. Their rationale is the following:

Experiment 1. 

uses the three variables available in the metadata of the reports, and listed above in Section 3, as predictors of fatalities. This experiment is used as a baseline for the comparison of the utility of employing human factors as predictors of fatalities.

Experiment 2. 

uses the human factors identified in the reports as predictors of fatalities. Its purpose is to verify whether or not they suffice as predictors.

Experiment 3. 

uses one of the variables (type of occurrence) of the neural network in Experiment 1 and the human factors as predictors. The reason why the type of occurrence was chosen, rather than the other two, is that in an incident, by definition, there are no fatalities, and thus this variable can be expected to significantly improve results.

Experiment 4. 

uses the three variables of the neural network in Experiment 1 and the human factors as predictors. These last two experiments serve to verify what is most important in predicting fatalities: human factors, or the other variables.

For each of these experiments, a Multiplayer Perceptron (MLP) neural network is trained using the data described in Section 3 (i.e., all the 1105 accidents and incidents), combining the corresponding predictors. The architecture of the neural networks was chosen through trial and error as a compromise between performance and overfitting; the corresponding pseudo-code is given below in Algorithm 1. Notice that there are two hidden layers with the relu (rectified linear unit) activation function

$$\varphi \left(v\right)=\max \left\{0,v\right\}$$

after the input layer, and that the output layer uses the sigmoid activation function.

$$\varphi \left(v\right)=\frac{1}{1+e^{-v}}$$

This can be represented by the sequence {13 13 13 1} (i.e., an input layer consisting of 13 neurons, two hidden layers also with 13 neurons, and an output layer consisting of only 1 neuron) [37].

As explained above, a random forest was also used for each experiment. The data from the 1105 accidents of incidents were again used for this. The pseudo-code is similar to that in Algorithm 1 for neural networks; default parameters were used.

Algorithm 1. Pseudocode of the Python program used to train the neural networks

Load database information Select desired data (human factors, type of occurrence, flight phase, aircraft fate) Split data randomly: 20% test, 80% training Use Python library Keras to build a Neural Network (NN) with:     two 13-neuron hidden layers, using the rectified linear unit (ReLU) activation function    a 1-neuron output layer, using the sigmoid activation function Train the NN with the Adaptive Moment Estimation (AdaM) optimiser, learning rate 0.001, decay rates ${\beta }_1=0.9$$\mathrm{and} {\beta }_2=0.999$$, \epsilon ={10}^{-7}$ Test the model and plot the results

4.3. Performance Indexes

The confusion matrix is defined in

Table 3. Confusion matrix for binary classification: 0, no fatality; 1, fatality or fatalities.

Predicted Value	True Value
	0	1
0	True Negative	False Negative
1	False Positive	True Positive

. The following performance indexes, usually employed in similar problems, were found to quantify the results.

$$\begin{gathered} Precision=\frac{True Positive \left(TP\right)}{True Positive \left(TP\right)+False Positive \left(FP\right)} \\ Recall=\frac{True Positive \left(TP\right)}{True Positive \left(TP\right)+False Negative \left(FN\right)} \\ F{1}_{score}=2\times \frac{Precision\times Recall}{Precision+Recall} \\ Accuracy=\frac{True Positive \left(TP\right)+True Negative \left(TN\right)}{True Positive \left(TP\right)+False Positive \left(FP\right)+True Negative \left(TN\right)+False Negative \left(FN\right)} \end{gathered}$$

These performance indexes can be found for multiclass analysis as well. In that case, the confusion matrix will have as many lines and columns as the classes. It should be noticed that some performance indexes can be deceitful if there are over-represented classes. Precision-recall plots and receiver operating characteristic (ROC) curves (i.e., a plot of TP as a function of FP) are better suited for such cases.

5. Results

Training algorithms can be run for a fixed number of iterations (also known as epochs), until no more data are available, or until no further performance improvement is possible [34]. Our training ran in mini-batches of size 16, and terminated after, at most, 100 epochs.

Another important decision is how to split the dataset into a part used for training and another for validation. We settled for an 80–20% split, which is a usual choice in the literature as a heuristically sufficient relationship to solve linear problems. To confirm the suitability for the problem under study, different splits were used with the neural network models; the resulting final accuracies for the four experiments are shown in Figure 4. The 80–20% split led to the best accuracy in two experiments, and the second best in the others; no other split achieves this. For this reason, the results will be shown only for this case; the same split was also used with random forests, for obvious reasons of fairness in the comparison.

The evolution with the epochs of cross entropy for neural network models, found from predicted and true outcomes, is shown in Figure 5 for each of the four experiments. There seems to be no overfitting since after the initial exponential decay, the values for the training set keep decreasing, while those for validation do not. The evolution of accuracy, shown in Figure 6, follows a roughly constant pattern, and verifies that the neural network architecture is suitable.

Concerning the performance of neural network models after training is carried out, Figure 7 shows the confusion matrixes for the four experiments; all performance indexes are tabulated in

Table 4. Performance indexes of each experiment processed via neural network.

Type of Learning	Performance	Class		Averages
		No Fatality	Fatality	Macro	Weighted
Multiplayer Perceptron Experiment 1	Precision	0.91	0.76	0.83	0.87
	Recall	0.94	0.69	0.81	0.88
	F1-Score	0.92	0.72	0.82	0.88
	Accuracy	0.88
Multiplayer Perceptron Experiment 2	Precision	0.92	0.79	0.86	0.89
	Recall	0.94	0.75	0.84	0.89
	F1-Score	0.93	0.77	0.85	0.89
	Accuracy	0.89
Multiplayer Perceptron Experiment 3	Precision	0.93	0.80	0.86	0.90
	Recall	0.94	0.76	0.85	0.90
	F1-Score	0.94	0.78	0.86	0.90
	Accuracy	0.90
Multiplayer Perceptron Experiment 4	Precision	0.91	0.88	0.89	0.90
	Recall	0.97	0.69	0.83	0.90
	F1-Score	0.94	0.77	0.85	0.90
	Accuracy	0.90

. The results for random forest models are presented in a similar way: confusion matrixes are shown in Figure 8, and performance indexes are tabulated in

Table 5. Performance indexes of each experiment processed via random forest.

Type of Learning	Performance	Class		Averages
		No Fatality	Fatality	Macro	Weighted
Random Forest Experiment 1	Precision	0.91	0.76	0.83	0.87
	Recall	0.94	0.69	0.81	0.88
	F1-Score	0.92	0.72	0.82	0.88
	Accuracy	0.88
Random Forest Experiment 2	Precision	0.92	0.79	0.86	0.89
	Recall	0.94	0.75	0.84	0.90
	F1-Score	0.93	0.77	0.85	0.89
	Accuracy	0.90
Random Forest Experiment 3	Precision	0.92	0.79	0.86	0.89
	Recall	0.94	0.75	0.84	0.90
	F1-Score	0.93	0.77	0.85	0.89
	Accuracy	0.90
Random Forest Experiment 4	Precision	0.90	0.87	0.89	0.89
	Recall	0.97	0.65	0.81	0.90
	F1-Score	0.93	0.74	0.84	0.89
	Accuracy	0.90

. It can be seen that their performance is practically the same; numerical values are often equal, and, when not, these differ only in the second decimal case. It is reasonable to say that neural networks and random forests are equally good at modeling the data; since neural networks require fewer parameters, we conclude that they should be the preferred method to carry out the prediction. Furthermore, the results reported in [38,39] are outperformed.

The results show that Experiment 2, using human factors as inputs, achieves better results than Experiment 1, the inputs of which are the three variables from the reports’ metadata. It is true that Experiments 3 and 4, combining human factors with one of the three variables or with all of them, present slight improvements when compared to Experiment 2. Such improvements, however, are marginal and not consistent since for some indicators, Experiment 2 outperforms all others.

These results are not surprising since human factors are present is a vast majority of database reports. That is why they can play a role in fatality prediction.

The statements above are confirmed by the results obtained with random forests given in

Table 6. Performance indexes of the models predicted in [7].

Type of Learning	Performance	Class		Averages
		No Fatality	Fatality	Macro	Weighted
Multiplayer Perceptron	Precision	0.89	0.75	0.80	0.83
	Recall	0.92	0.59	0.76	0.83
	F1-Score	0.90	0.66	0.77	0.83
	Accuracy	0.83
Random Forest	Precision	0.92	0.84	0.88	0.89
	Recall	0.94	0.77	0.86	0.90
	F1-Score	0.93	0.80	0.87	0.89
	Accuracy	0.90

. Consequently, it is possible to conclude that human factors are the most important predictor of fatalities.

Finally, these results ought to be compared with those from [7], shown in Table 6. The similarity of the values confirms that good results are not outliers. From [7], it could be argued that human factors were useful in predicting fatalities. This paper’s novelty is the confirmation that they are the most important variable, and can in fact be used as the only one for this purpose.

To conclude, neural networks are shown to dissect and interpret complex aviation data with the objective of predicting the likelihood of fatalities in real aviation accidents. The inherent ability of neural networks to learn and adapt from the data they process significantly bolsters their predictive accuracy. This attribute is of paramount importance in the context of aviation safety as it enables the identification of risk patterns and contributing factors in accidents. This methodology extends beyond basic analytical capabilities; it offers a nuanced comprehension of the factors that culminate in fatalities. Such an understanding is instrumental in formulating targeted safety measures and policies. By leveraging the computational power of neural networks, our ultimate goal is to enhance the industry’s capacity to preempt and mitigate the risks associated with various flight phases and accident types, thereby fostering a safer aviation environment.

6. Conclusions

This paper shows the importance of human factors in predicting whether or not there will be fatalities in aviation accidents. These results allow the industry to adopt measures that will address the relevant factors, and thus mitigate the probability of deaths.

The novelty of these results, when compared for instance with those in [1], is that we show how, using human factors alone, without further inputs, good predictions can be obtained about whether or not there will be fatalities.

Future work includes using a larger period corresponding to more recent reports. Increasing the size of the database is expected to allow not only for more accurate results, but also for the identification of the factors that are more relevant to prevent fatalities. Finally, it is important to address a more advanced question: Given the human factors involved in an accident in which there were fatalities, is it possible to predict how many fatalities there were?