Level 1—Event outcome;

Level 2—System/componen<sup>t</sup> failure causing the accident/serious incident; and

Level 3—The maintenance contributing factor(s) which led to the system/componen<sup>t</sup> failure and the ultimate accident/serious incident

The selection of this hierarchical structure was developed from the Bowtie Model (illustrated in Figure 3) to complement risk analysis and assessment processes. While using this model, inevitably there is considerable level of subjectivity and interpretation involved. For example, Level 1 Event Outcomes can be considered as 'Top Events' or alternatively if Level 2 'System/Component Failures' are considered as top events, then the Level 1 'Event Outcomes' will be considered at 'Consequences' on the right-hand side of the bowtie. Subsequently, Level 3 factors can be considered as weaknesses in the barriers or escalation factors. In this paper, it is not our aim to define all of the specific components of bowtie model with a rigid approach but ultimately for each event that a bowtie analysis can be conducted, the 'hazard' would be the work(s) undertaken by the maintenance personnel which resulted in the accident or serious incident. We aim to make the strong link between the maintenance factors (high risk areas within maintenance environment) and the key risk areas identified in 'Safety Risk Portfolios' published by EASA. Without presenting this information to industry representatives contributing to the CAT-CAG, it is extremely challenging to influence the decision making for taking risk mitigation actions and include them in the European Plan for Aviation Safety.

Once distinguished into three levels, each event was then re-evaluated and coded in accordance with the template. This process allowed for evolution of the taxonomy as more appropriate themes became apparent throughout the coding. Once the MxFACS taxonomy had been fully established, each event was assessed and classified in accordance with MxFACS in order to allow for further analysis of the output, and associated high risk areas, to be undertaken.

Once the taxonomy and resultant output had been finalised, it was then possible to utilise this data to produce a sample Bowtie model. This was achieved using BowTieXP software and allowed for a demonstration of the applicability of MxFACS to existing risk assessment methodologies.

Following the collation of SME responses, their data was entered into NVivo. This allowed for the identification of the themes within each question answer in order to reflect upon the study's methodological framework and to provide guidance on future utilisation of the coded data.

Whilst inferential statistical analysis of other variables were considered (for example: country or operator type), it was concluded that the combination of the number of events within the dataset over an extended period of time would not allow for statistically significant results to subsequent from such analyses.

**Figure 3.** Bowtie risk assessment model [14].

### *2.4. Assessing the Rigour of the Research Process*

Whilst a study's validity and reliability are key concerns for any piece of research, Liamputtong and Ezzy [15] argue that such terms are problematic in their application to qualitative research, suggesting that they are more suited to quantitative methods. The term 'rigour' is therefore used as a more appropriate, conceptualised measure of the underlying themes addressed by 'validity' and 'reliability' [15].

Inter-rater reliability, or inter-rater concordance, is a tool used within qualitative analysis to assess the level of agreemen<sup>t</sup> amongs<sup>t</sup> two or more 'raters' [16]. However, as highlighted by Liamputtong and Ezzy [15], it does not guarantee the reliability or validity of interpretations but is a useful tool in assessing the rigour of qualitative research.

Cohen's Kappa [17] is a popular statistic for inter-rater concordance; it shows the proportion of agreement, corrected for chance. Equation (1) demonstrates how Cohen's Kappa is derived, while Equations (2) and (3) detail how the components of the formula are determined.

$$\kappa = \frac{P\_{\mathcal{O}} - P\_{\mathcal{C}}}{1 - P\_{\mathcal{C}}} \tag{1}$$

where κ = Cohen's Kappa; *Po* = joint probability of agreement; and *Pe* = chance agreement.

$$P\_0 = \frac{\sum\_{i=1}^{n} R}{n} \tag{2}$$

where *Po* = joint probability of agreement; *R* = rater agreements; and *n* = total number of ratings.

$$P\_c = \frac{\sum\_{i=1}^{n} \left(\frac{c\_i \times r\_i}{n}\right)}{n} \tag{3}$$

where *Pe* = chance agreement; *c* = column marginal; *r* = row marginal; and *n* = total number of ratings.

In order to assess the rigour of this study, a SME coded a sample of 10 events using the MxFACS taxonomy. The SME's responses were then compared with the researcher's so that Cohen's Kappa could ultimately be determined. IBM SPSS statistics software was used to aid in the determination of a Cohen's Kappa value for the taxonomy as a whole.
