Method Construction and Evaluation: A More Comprehensive and Reliable Classification of Coal Mine Gas Explosion Causes

Abstract

Coal is an important fossil fuel energy that occupies a high position in the energy use of China and even the world. However, gas explosions are still the deadliest coal mine accident in China, which has long plagued the safety of energy mining. Only through accident cause analysis can we know the exact cause of the accident so as to make targeted policies, safety trainings, etc. However, the lack of detailed accident cause classification in current coal mine gas explosion accidents affects the comprehensiveness and accuracy of energy mining safety strategies. Therefore, in this study, a classification method for coal mine gas explosion accident causes based on the sixth edition 24Model and the three-element classification of gas explosions was proposed. Then, the consistency and validity of the newly established classification system were evaluated based on the three indicators, i.e., observer consistency, content validity, and criterion validity, and the performance of the classification system was verified. The results showed that the classification method exhibits good consistency and validity, and, compared with other classification methods, it can significantly improve the comprehensiveness of accident cause analysis results so as to obtain a more scientific energy mining safety strategy. In addition, the process used in this study to construct the classification and evaluate the performance of the classification is transferable, and it can provide a reference for the construction and evaluation of accident cause classifications in other fields.

1. Introduction

Coal is an important fossil fuel energy that occupies a high position in the energy use of China and even the world. Because of the objective reason of coal occurrence, underground coal mining is always one of the most dangerous forms of work, and gas explosion accidents are the most serious issue for the safety of China’s coal mine production [1,2]. Although the safety of coal mine production has improved gradually in recent years, the situation remains complex and grim [3]. Statistics show that from 2013 to 2020, a total of 280 coal mine accidents in large coal mines occurred in China, resulting in 1943 deaths, including 73 gas explosion accidents (including gas and coal dust explosion accidents, gas outbursts, and explosion accidents). This accounts for 26.07% of all the accidents in this period, resulting in 673 deaths, which accounts for 34.64% of all accident deaths in the current period, as shown in Figure 1 [4]. Thus, gas explosions are a major kind of coal mine accident, and, in recent years, the death toll from gas explosions has been the highest, highlighting the need to focus on the prevention and control of such accidents to ensure the safe exploitation of energy.

Accident analysis is the basic means of accident prevention. By taking accidents as a guide and drawing lessons from them, we can formulate regulations, risk management strategies, impart knowledge and training, and perform other accident prevention-related activities [5,6,7]. Therefore, to realize the long-term prevention mechanism of gas explosion accidents, it is necessary to continue the relevant accident cause analysis.

At present, a large number of scholars have conducted accident statistical analyses and research on coal mine gas explosion accidents, which have made great contributions to the prevention of gas explosion accidents. Chen, Qi, and, Tan [2] analyzed and studied 410 major gas explosion accidents in China and concluded that human factors were the main cause of these accidents. Yin [8] analyzed 201 heavy and extremely large gas explosion accidents in China from 2000 to 2009 and put forward relevant prevention suggestions for unsafe high-frequency actions. Zhang et al. [9] analyzed the causes of 126 particularly major gas explosions in China from 1950 to 2015 and found that the chaotic management of ventilation equipment was the most common cause of gas accumulation, and illegal blasting was the main cause of ignition. Liu et al. [10] analyzed the causes of gas explosions in China from 2000 to 2020, identifying the highest frequency locations and causes of underground gas explosions and exploring the trend and correlation between the causes of accidents. Tong et al. [11] analyzed the causes of 200 coal mine gas explosion accidents in China and constructed a probabilistic risk assessment model by classifying unsafe behaviors according to the type of work, which provided a basis for risk management and control. Because of the complexity of the accident, the causes of the accident had many commonalities and some main characteristics. In order to strengthen this logic and facilitate its application, the above research often classifies the causes of accidents. Classification is not only an effective tool to deepen the statistical analysis results of accidents and explore the correlation between their causes but is also a necessary prerequisite for risk evaluation index quantification and risk assessment.

In the era of big data, it is an indisputable objective fact that computationalism (using the perspective of computing to examine, analyze, and solve security problems) and dadaism (using data to speak) is widely applied within various fields and different disciplines [12]. The complexity of security and system problems promotes the transformation of related research from low dimensional to high dimensional, from linear to nonlinear, from small to large, and from scalar to vector. Artificial intelligence, data mining, and machine learning are frequently used [13]. Previous models for coal mine gas explosion accident statistics do not meet the demand of the current analysis, such as the correlation between the cause of the accident and the exploration and classification of hazard control. Further, an accurate prediction of the risk value of the accident prevention work is required to support the cause of the accident categories, along with a more detailed classification of the coal mine gas explosion accident, which has become an inevitable requirement. However, there is a lack of a set of detailed accident cause classifications in the field of coal mine gas explosion accidents.

Therefore, in this study, we developed a coal mine gas explosion accident cause classification tool to improve the comprehensiveness of the current accident analysis, providing a basis for formulating a more scientific energy mining safety strategy. In addition, we prepared a set of methodologies to evaluate classification performance through the construction of the accident cause classification method, providing a reference for the construction of accident cause classifications in other fields.

The sections of this paper are arranged as follows. First, the basic structure of the proposed classification is clarified and explained in Section 2. Then, based on the theoretical and methodological basis established in Section 2, the classification system is constructed. In Section 3, three experiments are presented to test the performance of the classification method and verify whether the classification method can achieve the proposed goals. Finally, Section 4 summarises the results obtained in this study.

2. Theoretical Basis and Concept Definition

2.1. Classification Basic Structure

In this paper, the basic structure of the classification method is mainly referred to as the 24Model and the three-element classification of gas explosion. The former provides the framework for setting the classification categories, whereas the latter provides the basis for setting the classification subclasses.

2.1.1. 24Model

The 24Model was first proposed in 2005 and is mainly based on Heinrich domino theory [14], Bird’s linear causative model [15], Stewart’s MMOS [16], Reason’s Swiss cheese model [17], and partially nonlinear models [18]. Unlike the well-known systemic accident causation models, the 24Model has undergone an evolutionary process from a linear model to a system theory model; the early 24Model belongs to the epidemiological model (first edition, second edition, and third edition), and its systematism began to emerge after the fourth edition [19], and it matured in the latest sixth edition of the 24Model [20,21] (see Figure 2).

Organization and accident are the most basic concepts in the 24Model; “organization” is short for social organization, which has its own management functions and administrative structure and is a part of the administrative structure. Combined with the emphasis of the model on the concept of ‘organization’, the 24Model indicates that any accident occurs in at least one social organization; thus, any accident is an organizational accident, and the organization is the subject of accident prevention [22]. On this basis, the 24Model defines an accident as “a series of negative effect events prescribed by an organization and not expected by people that cause damage to life, health, property, or the environment”. Thus, the 24Model divides the causes of accidents into two behavioral levels: organization and individual. The causes of accidents at the organizational level are divided into two stages: culture and system, while at the individual level, they are divided into two stages: ability and individual action. Overall, it divides the causes of accidents into two levels and four categories, which is why it is called the 24Model.

The 24Model is a systematic accident cause model which can provide a more comprehensive and systematic perspective for accident prevention [23]. As it has experienced an evolutionary process from being a linear model to a systematic model, when compared with the most widely used classical systematic models such as STAMP [23], AcciMap [24], FRAM [25], it also has some unique advantages:

(1)

The 24Model is suitable for the analysis of large samples and big data. Owing to the limited number of accident causes in an accident case, only a large number of accidents can be analyzed to develop comprehensive and systematic accident prevention measures [26]. Therefore, the cause model of systemic accidents should be able to adapt to the statistics of large samples and big data. The logic of the 24Model is simple, and each module has a clear definition and boundary, so it is more suitable for accident statistics analysis with large samples and big data. In addition, the 24Model has been widely applied in related fields [27,28,29,30,31], and its performance in accident cause analysis based on a large number of accident cases has been verified;

(2)

The 24Model has a flexible demand for the abundance of accident case information. The comprehensive presentation of the accident causes and logic between the causes is the advantage of the systematic accident causation model, but the above advantages are based on sufficient information on accident cases. According to a relevant study [32], the current accident investigation report of coal mine gas explosion accidents has different degrees of detail descriptions of accident causes. The demand of the systematic accident cause model for the abundance of accident case information should be flexible to avoid a situation in which the accident cause model cannot be completed because the accident investigation report is too brief. The 24Model has low requirements on accident cases, and there is no need to build a system control chart similar to STAMP or a functional network diagram of FRAM in advance. Analysts collect information related to accidents (as far as is possible), and then analyze them individually according to the cause module of the model to complete the accident analysis. Even if the accident case information is not sufficiently comprehensive, the accident can also be analyzed according to the existing information, and the cause of the accident can be classified. Thus, an insufficiently comprehensive accident case does not make the accident analysis work difficult [33,34];

(3)

Each module of the 24Model exhibits strong scalability. As mentioned in the introduction section, to improve the consistency of the accident analysis results, it is necessary to perform index refinement based on the accident causation model. Therefore, the accident cause model must have a certain extensibility. The definition of each module leading the system block of the 24Model is derived from long-term practical experience or authoritative standards and has high adaptability with other theories, models, and methods. At the same time, each module, as a separate concept, has a very rich connotation in the field of safety science. At present, relevant studies have been extended based on the 24Model. For example, Miao [35] used the classification of unsafe acts in HFACS as the subindex of the unsafe action module in the 24Model, which not only enriches the index division of the 24Model but also retains the significant advantages of the 24Model for clearly defining various modules. Suo [36] built the 24Model-MGE based on the 24Model. The unsafe condition was divided into gas accumulation, ignition source, technical facilities, and environmental conditions; the unsafe acts and personal abilities were classified according to the coal mine functional departments, and the safety management system causes were subdivided by reference to OHSAS 18,001. The above application studies have proved that each module of the 24Model has strong scalability;

(4)

The output of the 24Model is relatively concise, which is beneficial to the popularization of scientific research achievements and the application of field personnel. Due to the characteristics of the early linear accident cause model, the 24Model can not only complete the complex system modelling but can also show the accident analysis results in a concise diagram [37].

Based on the above analysis, the systematic accident cause model, i.e., the sixth edition 24Model, was selected as the main framework for the coal mine gas explosion accident cause classification proposed in this paper.

2.1.2. Three-Element Classification of Gas Explosion

The three elements of gas explosion refer to the three necessary conditions for gas explosion: (1) a cumulative gas concentration of 5–16%. (2) An ignition energy greater than 0.28 mJ, a temperature higher than 595 °C, and a fire source that is maintained throughout the explosion. (3) An oxygen concentration greater than 12% with CO₂ as an inert gas or greater than 9% with N₂ as an inert gas [38]. In the workplace, the oxygen concentration always exceeds 12%, so the main consideration for gas explosion accidents is gas concentration accumulation and the ignition source [39].

The three-element classification of gas explosion is highly consistent with the aim of the present study because of the following three points:

(1)

The use of consequences as the basis for classification. The establishment of accident cause classification based on consequences can explain the logical relationship between the accident causes through the category level and can then improve the consistency of the accident analysis results and the readability of the accident cause classification;

(2)

High recognition in the industry. The industry has basically reached a consensus on the physical conditions of a gas explosion and the division of the ignition sources, which is highly accepted and is simple and easy to use by onsite personnel;

(3)

Adding details to the 24Model. The classification of the three-element classification of gas explosions is exclusive to the field of gas prevention and control. The subindex division based on this method can improve the practicability of the 24Model in the analysis of coal mine gas explosion accidents.

Based on the above analysis, the three-element classification of gas explosions was selected as the main basis for establishing the classification method for the coal mine gas explosion accident causes presented in this paper.

2.1.3. Fusion Interface between Classifications

When considering the communication between classes as the basis of the fusion of different classifications, although there is obvious complementary relationship between the 24Model and the three-element classification of gas explosion, they do not have a condition of category compatibility from top to bottom. In order to integrate the two, it is necessary to clarify the interface where the two integrate.

The three elements of gas explosion usually take the event of ‘gas explosion’ as the starting point of accident analysis and accident cause classification, but this approach misses a series of causes and events that contribute to further losses, including unreasonable labor organization, improper emergency rescue, and secondary accidents. The investigation report of coal mine gas explosion accidents considered the data source and the 24Model as the basis for the classification method, with both taking the final loss event as the starting point for accident analysis, while expanding the top event of the three elements of a gas explosion to improve the scope of locating the accident cause and make it fit the classification system of the 24Model.

From the causes obtained in the process of accident analysis, the corresponding parts of the three elements of a gas explosion and the 24Model are mainly focused on individual acts (acts and conditions). The organizational factors (management factors) are difficult to reflect in the three-element classification of gas explosions due to the characteristics of the complex social and technical configurations of coal mine systems and the limitations of the classification method itself, which, alone, cannot correspond to the 24Model. Therefore, after extending the analysis starting point of the three elements to the loss events, the main interface for the fusion of the two classifications is the individual acts, as shown in Figure 3a.

Notably, the occurrence of coal mine accidents involves many organizations, mainly including the accident of the coal mine, the contract unit, the superior unit, and the regulatory department. Considering the information of accident causes shown in accident case reports and the main responsibility of accident prevention, in this paper, coal mining enterprises are considered to be the main organization for the accident analysis. In turn, the causes of the accidents in other organizations become external to the organization. According to the 24Model, after determining the main organization of accident analysis, factors outside the organization can be understood as the deficiency of internal response measures, which can then be included in the consideration of internal causes, and the deficiency of response measures can be divided into the four types accident cause. Therefore, the causes of the accidents, except for the coal mines, are uniformly classified as factors outside the organization, and no category is set.

The established basic structure of the classification is shown in Figure 3b. Afterward, based on the classification framework (subcategory construction principles and basic concepts), the classification of Coal Mine Gas Explosion Causes was constructed through accident analysis. The classification includes three categories and 138 subcategories in total, and the upper-level index structure is shown in Figure 4. The construction process and the description of the classification subclasses are detailed in the Supplementary Material.

3. Classification Performance Evaluation

Guiding accident cause analysis and accident cause classification and improving the comprehensiveness and accuracy of accident analysis results are the expected functions of the classification constructed in this study. The application value of the classification system was determined by its performance when it reaches a certain level. Therefore, the performance (above) must be fully evaluated and verified before the classification can be used.

3.1. Evaluation Metrics

On the one hand, the usefulness of the method depends on its ability to provide reproducible results, and on the other hand, on its ‘plug and play’ performance, which can provide results comparable to experts even if the user is not yet experienced [40]. The above two aspects can be summarized considering two indicators: consistency and validity, both of which are concepts in statistics and have been applied by many researchers to test and validate accident analysis methods [41,42,43,44].

3.1.1. Consistency

Consistency refers to the degree of similarity between the results obtained by the same method under similar conditions over a given period of time. According to previous studies, interobserver reliability and test–retest reliability are used as a representation of the consistency of accident analysis results (that is, the degree to which similar results are produced by different analysts analyzing the same case or the same analyst analyzing the same case at different times) [45,46,47]. Considering the controllability of the variables, interobserver reliability was chosen as the index to evaluate the consistency of the accident analysis results [48].

3.1.2. Validity

Validity refers to how accurate a measurement is, i.e., how close it is to what it is trying to measure. Regarding the validity of accident analysis methods, Igene and Johnson [41] believe that the validity of accident analysis methods refers to the degree of difference between the output results of the analysts and the experienced experts. According to Underwood, Waterson, and Braithwaite [44], the validity of an accident analysis method refers to the ability of the analyst to correctly identify the accident causes when applying the method. Jacinto [42] believes that a valid accident analysis method should promote (as much as possible) the consistency between the analysis results and reality. Katsakiori, Sakellaropoulos, and Manatakis [43] believe that the validity and consistency of accident analysis methods are closely related. If different investigators can reach the same conclusion through accident analysis methods, and the method proves that the accident analysis results will appear in future accident scenarios, the validity of the method can be proved. To sum up, the validity of an accident analysis method mainly refers to the consistency between the accident analysis results obtained by applying the method and the reality or expert results.

There are three main evaluation indicators of validity: construct validity, content validity, and criterion validity, which are aimed at different inspection objectives:

(1)

Construct validity refers to the degree to which the method can actually achieve the theoretical structure and characteristics of the target [49], corresponding to the classification established in this paper, i.e., assessing whether the structure or theoretical basis of the method is reasonable;

(2)

Content validity, which refers to whether the measurement content is suitable for the measurement target or whether it can reflect the characteristics or attitudes of the measurement target [50], corresponding to the classification established in this paper, that is, whether the categories in the classification can achieve comprehensive coverage of the coal mine gas explosion accident causes;

(3)

Criteria validity refers to the degree of correlation between the test results and the validity criteria. Validity criteria are a reference standard for measuring the validity of the test. It is a standard that is independent of the test method and can reflect the purpose of the test. When corresponding to the classification established in this paper, the validity criteria is the correct result of the classification, and the criterion validity can be regarded as the deviation between the user’s classification result and the correct result.

Since the classification proposed in this paper is established based on accident analysis, all subcategories can find their corresponding reasons in the accident investigation report, and both the 24Model and the three-element classification of gas explosions, as the theoretical basis, have been verified and applied for a long time. Therefore, construct validity is not within the scope of this chapter. Combining the definitions of content validity and criterion validity, it can be seen that both are important evaluation indicators to measure the performance of classification.

To summarize, this chapter uses interobserver reliability, content validity, and criterion validity as evaluation indicators for the consistency and validity of the classification results.

3.2. Evaluation Process

To evaluate the performance of the classification, three groups of experiments were designed using the classification to evaluate the consistency and validity of the classification under different usages. The specific design contents are as follows:

(1)

Experiment 1: two groups of analysts, A and B, were recruited to analyze the same batch of accident cases. Among them, the analysts of Group A used the sixth version of the 24Model to conduct accident analysis (as the control group); the analysts of Group B used the classification to conduct accident analysis (the experimental group). Observer consistency was calculated based on the accident analysis results obtained by the two groups of analysts to assess the consistency of the classification as an accident analysis tool;

(2)

Experiment 2: We organized professionals to analyze a batch of accident cases, discuss and reach a consensus on the accident analysis results, and use them as ‘validity criteria’. Afterward, the classification information of the accident analysis results was withheld, and a group of analysts was recruited to apply the classification to classify them. The validity criterion is verified based on the accident classification results obtained by the analyst to assess the validity of the classification as a guiding tool for accident causes classification;

(3)

Experiment 3: Professionals organized and applied the classification system to classify the accident analysis results of Group A in experiment 1, testing whether they achieved a high degree of coverage of the accident causes for the coal mine gas explosion accidents. This experiment verifies the content validity to evaluate the validity of the classification as an accident cause analysis tool.

3.3. Participants

To save labor costs, the participants in the three groups of experiments were reused, and the participants in each group of experiments were arranged as follows:

(1)

The analysts participating in experiment 1 were divided into two groups, A and B. Each group was composed of six graduate students majoring in safety science and engineering. The analysts received courses related to the 24Model;

(2)

The analysts participating in experiment 2 were divided into the expert group and the analyst group. Among them, the expert group was composed of the author and two experts in the field of coal mine accident analysis, who participated in the construction of the classification; the analyst group consisted of six analysts, all of whom were members of Group B of experiment 1;

(3)

The analysts who participated in experiment 3 were the three members of the expert group who also participated in experiment 2.

In summary, a total of 12 analysts and three experts participated in the three experiments to assess the performance of the classification system; the specific research background and demographic variables are shown in

Table 1. Analysis of specific information.

Group	Analyst ID	Gender	Age	Education	Proficiency of 24Model	Research Background
Group A	Analyst A1	Male	24	Doctor	Proficient	Accident analysis
	Analyst B1	Female	25	Master	General	Coal spontaneous combustion
	Analyst C1	Male	26	Doctor	Proficient	Gas explosion Accident analysis
	Analyst D1	Female	27	Doctor	Proficient	Process chemical accident Accident analysis
	Analyst E1	Female	25	Master	Proficient	Accident analysis
	Analyst F1	Female	24	Master	General	Safety disciplines system
Group B	Analyst A2	Female	25	Master	General	Safety culture
	Analyst B2	Female	25	Master	General	Safety culture
	Analyst C2	Male	30	Doctor	Proficient	Coal mine accident Accident analysis
	Analyst D2	Male	27	Doctor	General	Coal spontaneous combustion
	Analyst E2	Female	25	Master	Proficient	Accident analysis Energy Model
	Analyst F2	Male	32	Doctor	General	Coal spontaneous combustion
Expert Group	Expert A	Male	33	Doctor	Proficient	Coal mine accident Accident analysis Accident causation theory
	Expert B	Female	33	Doctor	Proficient	Coal mine accident Accident analysis Accident causation theory

The proficiency of the 24Model is divided into skilled and general. ‘Proficient’ means that the number of accidents analyzed by the 24Model exceeds 20; ‘General’ means that the number of accidents analyzed by the 24Model is greater than 10 but less than or equal to 20.

.

3.4. Accident Cases

A total of two groups of accident cases were used as the samples in the three groups of experiments, denoted as a and b. The accident cases were randomly selected from major coal mine gas explosion accidents in China from 2000 to 2016, and the source of the accident information was the official accident investigation report. Among them, the first and third experiments used the accident cases of Group I, and the number of accident cases was 14; experiment 2 used the accident cases in Group II, and the number of accident cases was six. Details of the 20 accident cases are shown in

Table 2. Accident case information.

Number	Name of Accident Case	Accident Time
1	Hunan Province “3.29” Gas Explosion Accident	2001
2	Inner Mongolia “5.8” Gas Explosion Accident	2001
3	Heilongjiang Province “1.20” Gas Explosion Accident	2003
4	Shanxi Province “4.30” Gas Explosion Accident	2004
5	Liaoning Province “2.14” Gas Explosion Accident	2005
6	Xinjiang “10.4” Gas Explosion Accident	2005
7	Shanxi Province “2.1” Gas Explosion Accident	2006
8	Gansu Province “10.31” Gas Explosion Accident	2006
9	Liaoning Province “8.18” Gas Explosion Accident	2008
10	Heilongjiang Province “11.21” Gas Explosion Accident	2009
11	Jilin Province “3.29” Gas Explosion Accident	2013
12	Xinjiang “7.5” Gas Explosion Accident	2014
13	Chongqing “10.31” Gas Explosion Accident	2016
14	Inner Mongolia “12.3” Gas Explosion Accident	2016
15	Chongqing “6.26” Gas Explosion Accident	2000
16	Henan Province “11.19” Gas Explosion Accident	2000
17	Yunnan Province “5.14” Gas Explosion Accident	2001
18	Heilongjiang Province “2.5” Gas Explosion Accident	2001
19	Guizhou Province “2.24” Gas Explosion Accident	2003
20	Shaanxi Province “4.29” Gas Explosion Accident	2006

.

In addition, an instruction manual of the 24Model [18,19], details of the subcategory descriptions of the classification system (Section 3 of this paper), and the Coal Mine Safety Regulations (2018) were provided to the analysts as reference materials for accident analysis.

3.5. Indicator Calculation Method

3.5.1. Interobserver Reliability Calculation Method

Intraclass correlation coefficient (ICC), Kappa coefficient, and percentage concordance are commonly used statistical variables to evaluate interobserver reliability [51]. In the field of accident analysis, the number of accident analysis results obtained by different analysts may be inconsistent, and the intragroup correlation coefficient and Kappa consistency coefficient both regard the observation results of the observers as quantitative and disordered categorical variables, so they cannot be used. Therefore, percentage concordance was chosen as the statistical variable for evaluating the interobserver reliability of the accident analysis results, and the concordance index was used as the method for calculating interobserver reliability.

C-index was proposed by Harrell Jr. et al. [52], and it is mainly used to calculate the degree of discrimination between the predicted value and the true value of the COX model in survival analysis; corresponding to this article, the formula is stated as follows:

$$\mathrm{C-index}=\frac{\mathrm{C}}{\mathrm{C}+\mathrm{D}},$$

(1)

where C is the number of reasons for consensus among the analysts, and D is the number of reasons for disagreement among the analysts. It can be seen from the formula that the value of C-index is [0,1]. According to a previously reported study [53], 0.7 is the smallest reasonable value for the consistency criterion.

3.5.2. Content Validity Calculation Method

As mentioned above, content validity is a measure of the coverage of the coal mine gas explosion accident causes. A classification with good content validity should strive to avoid overlapping and missing categories. Therefore, in an ideal outcome, the causes from the accident analysis should all be successfully classified using the classification system. This paper uses the ratio of the number of causes for successful categorization to the number of causes obtained by analysis as an indicator to characterize content validity, and calls it “causes coverage”, which is defined as follows:

$$\mathrm{p}=\frac{\mathrm{m}}{\mathrm{n}},$$

(2)

where:

• n is the number of all accident causes obtained through an analysis;
• m is the number of accident causes that can be classified into the classification;
• p is located in the interval [0,1] and the closer it is to 1, the higher the causes coverage rate, indicating that the classification can cover all accident causes, and its content validity is higher.

3.5.3. Criterion Validity Calculation Method

The calculation method for criterion validity refers to a recent study. Hulme et al. [54] used a confusion matrix as an evaluation method to test the validity of the accident causation model. Our study was inspired by that of Hulme et al. and is based on the confusion matrix, using the indicators in machine learning to evaluate the performance of the classification and utilizing the accident analysis results obtained by expert analysis as the validity criterion to evaluate the validity of the classification system established previously. The details are presented in this section.

The confusion matrix is a standard format for representing the prediction results of a classifier, and it is represented in the form of a matrix with n rows and n columns (n represents the number of categories). The confusion matrix puts the predicted results of all categories and the real results into the same table by category. The table indicates the number of correct identifications and the number of misidentifications for each category.

Table 3. Example of a dichotomous confusion matrix.

Confusion Matrix	Predicted as Positive	Predicted as Negative
The category is positive	True Positive, TP	False Negative, FN
The category is negative	False Positive, FP	True Negative, TN

is a confusion matrix for binary classifications, where n = 2; that is, there are only two categories: positive and negative. According to the combination of real results and model prediction results, it can be divided into True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). The meanings of the four combinations are:

TP: the result classified into the positive class is the same as the real result, and the classification is correct; TN: the result classified into the negative class is the same as the real result, and the classification is correct; FP: the result classified into the positive class is different from the real result, and the classification is wrong; FP: the result classified into the negative class is different from the real result, and the classification is wrong.

The indicators for evaluating the classification performance based on the confusion matrix are accuracy, precision, recall, and the F1-Score.

(1)

Accuracy refers to the ratio of correctly classified samples to the total number of samples. Accuracy is a statistic for all samples [55]. It is defined as:

$$\mathrm{Accuracy}=\frac{\left(\mathrm{TP}+\mathrm{TN}\right)}{\left(\mathrm{TP}+\mathrm{FP}+\mathrm{TN}+\mathrm{FN}\right)}=\frac{\mathrm{Number} \mathrm{of} \mathrm{correctly} \mathrm{classified}}{\mathrm{Total} \mathrm{number} \mathrm{of} \mathrm{items} \mathrm{classified}}.$$

(3)

(2)

Precision indicates the ratio of the number of samples classified as positive by the user and correctly classified to the number of samples classified as positive by the user [55]. It is defined as:

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\left(\mathrm{TP}+\mathrm{FP}\right)}=\frac{\mathrm{Number} \mathrm{of} \mathrm{classified} \mathrm{as} \mathrm{positive} \mathrm{and} \mathrm{correctly} \mathrm{classified}}{\mathrm{Number} \mathrm{of} \mathrm{classified} \mathrm{as} \mathrm{positive}}.$$

(4)

(3)

Recall refers to the proportion ratio of the number of correctly classified positive samples to the number of positive samples. It can be expressed as:

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\left(\mathrm{TP}+\mathrm{FN}\right)}=\frac{\mathrm{The} \mathrm{number} \mathrm{of} \mathrm{correctly} \mathrm{classified} \mathrm{positive} \mathrm{samples}}{\mathrm{The} \mathrm{number} \mathrm{of} \mathrm{positive} \mathrm{samples}}.$$

(5)

(4)

F1-Score is the harmonic mean of precision and recall, taking into account the precision and recall of the classification [55], and is defined as:

$$\mathrm{F}1=\frac{2 \times \mathrm{Precision} \times \mathrm{Recall}}{\left(\mathrm{Precision}+\mathrm{Recall}\right)}.$$

(6)

The maximum and minimum values of the F1-Score are 1 and 0, respectively. The larger the value, the better the model’s performance. Numerous variant formulas of the F1-Score have been subsequently derived in order to deal with different task types, such as Macroaverage F1-score, Microaverage F1-score, weighted F1-score, and Matthews correlation coefficient (MCC). [56]

Because experiment 2 involves many categories and belongs to a multiclassification task, the accuracy and Macroaverage F1-score are selected as indicators to assess the validity. The calculation idea of the Macroaverage F1-score is to split the N classification problem into N binary classification problems and calculate the precision and recall of each category separately. After that, it is added and averaged to get Macroaverage Precision and Macroaverage Recall; then, in order to obtain the Macroaverage F1-score, the following formula is used [57]:

$$\mathrm{Macro} \mathrm{avg} \mathrm{F}1-\mathrm{Score}=\frac{2 \times \mathrm{Macro} \mathrm{avg} \mathrm{Precision} \times \mathrm{Macro} \mathrm{avg} \mathrm{Recall}}{\left(\mathrm{Macro} \mathrm{avg} \mathrm{Precision}+\mathrm{Macro} \mathrm{avg} \mathrm{Recall}\right)},$$

(7)

where:

$$\mathrm{Macro} \mathrm{avg} \mathrm{Precision}=\sum \frac{{\mathrm{Precision}}_{\mathrm{i}}}{\left|\mathrm{L}\right|},$$

(8)

$${\mathrm{Precision}}_{\mathrm{i}}=\frac{{\mathrm{TP}}_{\mathrm{i}}}{\left({\mathrm{TP}}_{\mathrm{i}}{+\mathrm{FP}}_{\mathrm{i}}\right)},$$

(9)

$$\mathrm{Macro} \mathrm{avg} \mathrm{Recall}=\sum \frac{{\mathrm{Recall}}_{\mathrm{i}}}{\left|\mathrm{L}\right|},$$

(10)

$${\mathrm{Recall}}_{\mathrm{i}}=\frac{{\mathrm{TP}}_{\mathrm{i}}}{\left({\mathrm{TP}}_{\mathrm{i}}{+\mathrm{FN}}_{\mathrm{i}}\right)},$$

(11)

where L is the total number of labels (categories), and i is the ith label.

When calculating the index, the validity criterion is used as the standard classification result, and the classification results provided by other analysts are used as the classifier prediction result.

In summary, to test the consistency and validity of the classification, interobserver consistency, content validity, and criterion validity were used as evaluation indicators to evaluate the classification system. Among them, the C-index is used to calculate interobserver consistency, the causes coverage is used to test the content validity, and the Accuracy, Macroaverage Precision, Macroaverage Recall, and the Macroaverage F1-score are used to test the criterion validity.

3.6. Specific Process

3.6.1. Preanalytical Training

Two groups of analysts were first trained before the accident analysis. Different groups of analysts received different training contents. The Group A analysts mainly explained the format of the accident analysis results, the limitations in the accident analysis process, answering any questions; the Group B analysts primarily explained the definitions of the concepts of the classification system, the basis for dividing it into subcategories, examples of accident analysis, the format of the accident analysis results, and the limitations of the accident analysis process, while also answering any questions.

3.6.2. Accident Analysis

Subsequently, the three groups of personnel analyzed the cause of the accidents:

(1)

Expert group. The expert group only analyzed the six accident cases of sample B, and the accident analysis time was limited to 4 days, during which the accident analysis could be communicated at any time to correct the accident analysis results. Finally, a consensus was reached on each accident cause, which was used as the validity criterion to test the criterion validity of the classification in experiment 3;

(2)

Group A analysts. The Group A analysts only analyzed 14 accident cases in sample A, and the accident analysis time was limited to 10 days. During this period, the analysts could not communicate with each other about the accident analysis and could only use accident investigation reports as information sources. After the accident analysis was completed, the accident analysis results of each analyst were collected, and all participants were called through to a seminar to judge whether the accident analysis results obtained by the different analysts reached a consensus through discussion;

(3)

Group B analysts. The Group B analysts participated in experiment 1 and experiment 2 successively. The accident analysis time of experiment 1 was limited to 10 days, and that of experiment 2 was limited to 4 days. The two studies continued uninterrupted, but the material and incident analysis results were recovered and distributed into stages:

(1)

The relevant materials of experiment 1 were distributed at the beginning of the study;

(2)

On the 10th day, the accident analysis results of experiment 1 were recovered, and the relevant materials from experiment 2 were distributed;

(3)

On the 14th day, the accident classification results of experiment 2 were recovered.

During this period, the analysts could not communicate with each other on the accident analysis and could only use the accident investigation reports as a source of information. After the accident analysis was completed, the accident analysis results from each analyst were collected, all the analysts convened in the form of a seminar, and the accident analysis results obtained by the different analysts in Group B were evaluated to judge whether a consensus could be reached through discussions.

3.6.3. Summary and Calculation of Analysis Results

A total of two sets of data were obtained from experiment 1, which were the analysis results of the Group I accident cases by the two groups of analysts (A and B). in order to facilitate the calculation of the C-index, the accident causes of the two sets of data were deduplicated and coded respectively, and the accident analysis results were marked with the coding. Afterward, the interanalyst C-index was calculated pairwise for each accident. There were six analysts in each group, so 15 C-indexes were available for each accident in each group.

A total of two sets of data were obtained from experiment 2. The first set of data was the result of the expert group applying the classification system to analyze the accident causes from Group II of the accident cases. The data format was a two-tuple set consisting of accident investigation report paragraphs and accident cause categories; the second set of data was a two-tuple set consisting of accident investigation report paragraphs and accident cause categories (after the first group of data was classified by the first group of data using the classification).

In order to facilitate the calculation of the accuracy and Macroaverage F1-Score, the original paragraphs of the accident investigation report and the accident causes were coded in order to label the two sets of data. After that, the first set of data was used as the criterion, and the second set of data, along with the established criterion, were used to construct a confusion matrix in the unit for the analyst. There were six analysts in total, so a total of six confusion matrices were obtained. Criterion validity was tested using accuracy, Macroaverage Precision, Macroaverage Recall, and Macroaverage F1-Score.

Experiment 3 involved two sets of data. The first set of data contained the results of the analysts from Group A in experiment 1 after deduplication. The second set of data was the result of the secondary classification of the first set of data by the expert group using the classification system. During this period, the reasons and quantities that could not be classified (marked as ‘NONE’) were recorded and checked by two professionals. Causes coverage was calculated using the formula contained in Section 3.5.2.

In all the three experiments, Python was used as a tool for the indicator calculations.

3.7. Evaluation Result

3.7.1. Results of Experiment 1

The accident analysis results obtained by the two groups of analysts for experiment 1 were uploaded to the Mendeley Data database [58]. The calculation results for the C-index are shown in Figure 5 and Figure 6.

In the Group A samples, the average value of the C-index calculation results was 0.34, the highest value was 0.83, and the lowest value was as low as 0.00, which is far below the threshold of 0.7. It was found that the consistency of the accident analysis results from the Group A samples was at a low level, which further verifies the necessity of this study; in the Group B samples, except for accident case 3 (0.63) and accident case 8 (0.67), for which the average C-index scores were lower than the threshold of 0.7, the average C-index scores of the other accident cases were all higher than the threshold. The average C-index score was 0.78. Among them, the average C-index score of a single accident was 0.86 (the highest) and 0.63 (the lowest), and the C-index score of some analysts achieved excellent results, as high as 0.97. Comparing the C-index calculation results of the two groups, it can be seen that the interobserver reliability of the classification is high, which can significantly improve the consistency of the accident analysis results.

3.7.2. Results of Experiment 2

The coding table from the accident investigation report, the coding table from the accident causes, and the accident causes classification results have all been uploaded to the Mendeley database [58]. The Accuracy, Macro-average Precision, Macroaverage Recall, and Macroaverage F1-score of each confusion matrix in experiment 2 are shown in Figure 7. Due to the large size of the constructed confusion matrix (a total of 61 accident cause categories appears in the summary results, so each matrix size is 61 × 61), the total number is six, which is no longer listed to save the paper layout.

It can be seen from Figure 7 that the classification results of the six analysts achieved high scores for the four indicators. Among them, the accuracy rate scores of the six analysts are all above 0.9, and the average score is 0.94, indicating that the correct classification of all accident causes accounts for more than 90% of the total; the minimum value of the Macroaverage Precision rate is 0.85, the maximum value is 0.91, and the average value is 0.88, indicating that, for a specific reason, the probability of using the classification to classify a case correctly is 88% on average. The minimum value of the Macroaverage Recall rate is 0.86, the maximum value is 0.91, and the average value is 0.89, indicating that, for a specific cause, the probability of successfully finding the correct accident cause is 89% on average. Finally, the Macroaverage F1-score is the combined performance of the first two indicators, with a minimum value of 0.85, a maximum value of 0.90, and an average value of 0.88. The classification method proposed in this paper has as many as 140 categories, but from the perspective of the four indicators for evaluating classification accuracy, the settings between the categories can ensure that users maintain a low rate of missed judgments and misjudgements and shows good criterion validity.

3.7.3. Results of Experiment 3

A total of 1033 items were coded into the results of experiment 1. After the classification was applied, a total of 139 accident cause chains were obtained. The original accident analysis results and the secondary classification results using the classification system have been uploaded to the Mendeley Data database [58]. After the deduplication of the data obtained in experiment 1, the resulting total number of original accident causes is shown in

Table 4. Accident cause quantity statistics and causes coverage calculation results (based on accident cases).

Accident Number	Individual Act (De-Duplication)	Management System	Total Number of Accident Causes	Number of Causes Uncategorized	Causes Coverage
Case 1	20	33	53	3	94.34%
Case 2	20	37	57	6	89.47%
Case 3	13	33	46	5	89.13%
Case 4	32	48	80	3	96.25%
Case 5	26	49	75	2	97.33%
Case 6	31	55	86	3	96.51%
Case 7	12	35	47	5	89.36%
Case 8	17	33	50	1	98.00%
Case 9	27	58	85	5	94.12%
Case 10	16	33	49	3	93.88%
Case 11	22	40	62	3	95.16%
Case 12	30	65	95	10	89.47%
Case 13	33	53	86	9	89.53%
Case 14	24	25	49	5	89.80%
Total	323	597	920	63	93.15%

. Taking the accident case as the unit, the lowest causes coverage rate is 89.36%, the highest is 98.00%, and the average causes coverage rate is 93.15%; that is, 93.15% of the accident causes found by the analyst in the accident investigation report can be successfully classified into the classification system. Currently, there is no criterion for evaluating the causes coverage of the classification. However, considering that there are nearly 920 accident causes included in experiment 3, while ensuring certain classification details, it is believed that the current causes coverage rate of the classification reached a relatively satisfactory level and has good content validity.

4. Conclusions

At present, the cause analysis of coal mine gas explosion accidents lacks detailed accident cause classification, which affects the comprehensiveness and accuracy of gas explosion accident prevention. In order to solve the above problems and ensure the safe exploitation of energy, we first analyzed the characteristics of the three-element classification of gas explosions and the 24Model, and clarified the process and basic structure of the fusion of the two classifications. Next, classification for the causes of the coal mine gas explosion accidents was constructed through accident analysis and cause generalization.

In order to evaluate the performance of the newly established classification and establish its consistency and validity, interobserver reliability, content validity, and criterion validity were used as the evaluation indicators. By organizing three experiments, the above indicators were calculated using the C-index, causation coverage and accuracy, Macroaverage Precision, Macroaverage Recall, and Macroaverage F1-score.

These three experimental results showed that the classification system exhibited good consistency and validity regarding the results. Some analysts without a coal mining background even obtained results that were comparable to those obtained by the professionals in terms of consistency and validity. In addition, the classification of the coal mine gas explosion accident causes could be displayed using the accident cause chain. The simplicity of the accident analysis results was maintained, whereas the readability was drastically improved. In the case of skilled classification, the approximate information of an accident can be inferred from the chain of the accident causes alone. The main findings of this study are as follows:

(1)

Based on the 24Model and the three-element classification for gas explosions, a classification system for coal mine gas explosion accident causes, including three categories and 138 subcategories, was constructed. This constructed classification system can effectively improve the consistency and comprehensiveness of the analysis results and can then obtain a more scientific accident prevention strategy;

(2)

The general construction process of accident cause classification was presented, including the determination standard of the basic structure of the classification system, the construction principles of the classification subclasses, and the construction method of the classification subclasses. Many of the above basic concepts can be transferred to other fields as well;

(3)

A general process for evaluating the performance of the classification was constructed using the two indicators: consistency and effectiveness, including the criteria for determining the evaluation indicators, the design of the evaluation process, the calculation method for the evaluation indicators, and the specific implementation process, which can provide a reference for evaluating other accident analysis methods.