3.7.1. Challenges with Taxonomies

A particularly insightful description of the challenges faced when using taxonomies to code aircraft maintenance-related events was provided by this SME:

*"All taxonomies su*ff*er the problem that categorisation can condition results* ... *[Classifying] events after the event often requires a lot of imagination. There's an inverse relationship. Rare fatal accidents provide much detail whereas frequent occurrences can be one line in a log book."*

The challenge of attaining detailed information from low level occurrences was certainly faced in the initial data collection process for this study. It was behind the reasoning to scope the research to focus on the lower quantity, but higher quality, serious incidents and accidents as opposed to high volume, minimal detail, low-level occurrences. That does not mean to say that these low-level occurrences should be ignored, to the contrary, but rather that further action is required to ensure the associated reporting processes capture adequate, actionable information.

Regarding the statement on categorisation, it can be argued that using peer review to determine inter-rater concordance could certainly aid in reducing the subjectivity of taxonomies. It was also highlighted by another SME that this use of peer review to assess the inter-rater concordance of the taxonomy categorisation, as was undertaken for the purpose of this study, matched the methodology of the UK CAA [3] which had a peer review to try and validate the initial categorisation."

One SME argued that the importance in learning from occurrences does not lie in coding the existing data for interpretation, but rather in comparing theory with reality: *"Taxonomy is unimportant. What is important is comparing practice with prediction."*

These SME opinions sugges<sup>t</sup> that the utility of pre-existing aircraft maintenance taxonomies may perhaps be a source of some contention.

### 3.7.2. Feedback on the Study's Methodology

Feedback on the study's methodological framework was largely positive, as this extract demonstrates: "The study has done a grea<sup>t</sup> job in collecting the data and classifying it into useful information."

Another SMEs detected the traits of Bowtie within the taxonomy, referencing the "threats" and "escalating factors" as: "causal factors (remove them and the accident is avoided) and circumstantial factors (increased the probability of the event)."

Whilst not directly referring to bowtie, this statement does reflect the thinking that there are di fferent types of factors which can contribute to accidents, serious incidents and occurrences. The decision to name MxFACS Level 3 "maintenance factors" was done so for precisely this reason, so it is therefore encouraging to hear a SME mirror this sentiment.

3.7.3. Assigning Risk and Identifying High Risk Areas from Coded Event Data

One SME proposed an assessment of the e ffectiveness of remaining safety barriers for non-accident level occurrences as a means of assigning risk, acknowledging existing EASA methodology:

*"The risk should be based on the e*ff*ectiveness of the remaining barriers left before it ended up as a credible accident. See ARMS methodology." [28]*

This statement brings about the consideration of the use of MxFACS output in conjunction with the ARMS Event Risk Classification (ERC). The UK CAA [2] highlight that Bowtie is often used within the ARMS methodology regarding the ERC barrier e ffectiveness assessment. This shows a possible pathway for the integration of MxFACS with the ARMS ERC methodology as both have strong applicability to bowtie.

Another SME proposed the use of an expert panel to e ffectively assess and allocate risk:

*"Use a team of experts. [It is] hard work to get consensus but expert challenge is a good way of getting a realistic classification. Use a problem statement to get everyone on the same page."*

The use of MxFACS output, alongside the associated Bowties, could aid this approach by providing the experts with an outline of the risks and barriers involved in the events to be analysed.

3.7.4. Using the Findings of the Study

It was suggested by one SME that the key to better targeted action may be to address near misses rather than accidents: *"Focus on the near miss events rather than the accidents to try and determine how close to an accident we are."*

In contrast, another SME, who highlighted a regulatory resource shortage as being challenging to acting on occurrences, suggested a di fferent approach:

*"[Regulators] all struggle with limited resources-being driven by events provides only the basics. Continuous improvement means being proactive. Uncover the trends, pick o*ff *5 'low hanging' fruit and work them through. A Pareto analysis was used by US CAST to good e*ff*ect* ... *Target priorities on the higher risk items that are easiest fixed first."*

This complements the thinking that prevalence of high frequency occurrences does not necessarily indicate a propensity for accident propagation from their associated risks, particularly if su fficient barriers are in place. By instead focusing on identifying high risk areas within maintenance, regardless of the frequency of actual catastrophic outcome, it could be argued that the resultant targeted prevention strategies may be more e ffective.

### *3.8. Identifying High Risk Areas*

Maurino [29] proposes that the findings of investigations should encourage error tolerance and error recovery, as opposed to error suppression. By identifying the high-risk areas in aircraft maintenance, it is possible to understand what factors shaped particular human errors.

Upon reviewing the fatalities and damage count for the 112 analysed events, it was found that 16 flights had a fatal outcome and 77 lead to aircraft damage. EASA [4] identify damage to be of a medium level risk within their key risk areas. As 69% of the events identified within the dataset resulted in some level of damage, it can be said that the likelihood of event for this KRA was substantial over the past 15 years.

In order to better relate the frequency of events data from Figures 5–7 to risk, the events within the dataset were evaluated at each level of the taxonomy for number of fatal accidents and instances of aircraft damage. The fatal accident figures were then plotted alongside number of events and number of fatal events (represented as the size of the bubble), to replicate the same risk visualisation approach used by EASA [4] in Figure 9.

**Figure 9.** Key risk areas for CAT aeroplanes by fatalities 2013–2017, adapted from EASA (2018).

Figure 10 shows this chart for the Level 1 results, with more detail about the relationship between fatalities, the number of fatal events (represented by the size of the bubble), and the total number of Level 1 outcomes.

Three of the four event outcomes identified as having fatal outcomes are congruen<sup>t</sup> with three of the maintenance KRAs listed in EASA ASR [4]. One particular point of significance is the positioning of collision: this area has a large proportion of fatalities, damage and frequency; it could be a key area of focus for further risk analysis processes. The coded MxFACS data may be used in conjunction with analysis methodologies such as bowtie to examine the particular barrier failings which lead to these kinds of accidents. Further information about aircraft damage can be found in Table 7.


**Table 7.** Level 1 fatal accident and aircraft damage relationships.

**Figure 10.** Level 1 fatal accident relationship.

The ranked orders for greatest number of fatal accidents and aircraft damage for Level 2 of the dataset are given in Table 8.

Figure 11 shows the relationship between fatalities, number of fatal events (represented by the size of the bubble) and the total number of Level 2 events.

Engine-related events can be seen to have the largest propensity for both fatalities and aircraft damage. As shown in Figure 11, these events were also the most frequently occurring across the dataset. This would sugges<sup>t</sup> that maintenance related to aircraft powerplants should be placed high on the agenda for regulators when proposing better-targeted action and oversight, as well as being a key focus for maintenance organisations. Similar comment can be made in relation to landing gear and flight controls, which also rank highly across all three areas.


**Table 8.** Level 2 fatal accident and aircraft damage relationships.

**Figure 11.** Level 2 fatal accident relationship.

It is more difficult to directly compare the Level 3 events with the fatality and damage figures as many of the events have multiple maintenance factor categorisations assigned to them. Therefore, the events at this level were analysed as a percentage of the total number of instances where a maintenance factor was attributed to a fatal accident or aircraft damage. The maintenance factors

related to fatal accidents and aircraft damage are shown in Figure 12, with a full breakdown given in Table 9.

**Figure 12.** Level 3 fatal accident relationship.


**Table 9.** Breakdown of maintenance factors related to fatalities and aircraft damage.

Inadequate maintenance procedures and operator oversight can be seen as the two predominating areas within the top nine maintenance factors for fatalities. This may sugges<sup>t</sup> that organisational safety managemen<sup>t</sup> requires particular attention and would perhaps warrant further risk analysis to identify the interdependencies which interact with these maintenance factors.
