(1) Collection of experts' comments

Several experts were invited to complete the questionnaire form (Table 1) in accordance with the procedure and requirements of the Delphi method [42]. The experts ranked the importance of each index independently according to their own knowledge and experience. The indexes were ranked from high to low according to their importance; for example, mark "1" represented "most important", mark "2" represented "more important", mark "3" represented "important", and so on. Some indexes could be recognized as equally important, and the final rank of the indexes could be discussed by the experts.


**Table 1.** Collection of experts' comments.

#### (2) Blind degree (uncertainty) analysis

The potential deviation and uncertainty of the experts' comments on the index ranks might arise, due to noisy data. To eliminate noisy data and reduce uncertainty, the qualitative judgement conclusion from the experts should be statistically analyzed and addressed. To reduce the uncertainty of the experts' ranking, the entropy value was calculated by the entropy theory. The execution steps are shown below [21,43].

Supposing that *k* experts were invited to take the questionnaire survey, then *k* questionnaire forms would be returned, and every form would be recognized as an index set and marked as *R*={*r*1, *r*2, ... *rk*}; where *ri* refers to the expert ranking array denoted by {*ai*1, *ai*2, ... *ai*n}(*i*=1, 2, ... *k*) and *ai*1, *ai*2, ... *ai*<sup>n</sup> can be any natural number from {1, 2, ... *n*}. As previously mentioned, "1" represents the highest level of importance. The index sort matrix obtained from the k table is shown as matrix *A*.

$$A = \begin{bmatrix} a\_{11} & a\_{12} & a\_{13} & \cdots & a\_{1n} \\ a\_{21} & a\_{22} & a\_{23} & \cdots & a\_{2n} \\ \dots & \dots & \dots & \dots & \dots & \dots \\ a\_{k1} & a\_{k2} & a\_{k3} & \cdots & a\_{kn} \end{bmatrix} \tag{1}$$

where *aij* represents the *i*th expert's evaluation of the *j*th index.

The qualitative ranking result could be transformed into quantitative results by a membership function, which can be defined as follows:

$$
\chi(I) = -\lambda p\_n(I) \text{lnp}\_n(I), \tag{2}
$$

where *pn*(*I*) <sup>=</sup> *<sup>m</sup>*−*<sup>I</sup> <sup>m</sup>*−<sup>1</sup> , <sup>λ</sup> <sup>=</sup> <sup>1</sup> ln(*m*−1), which can be input into Equation (2):

$$\chi(I) = -\frac{1}{\ln(m-1)}(\frac{m-I}{m-1})\ln(\frac{m-I}{m-1}).\tag{3}$$

Dividing both sides by (*m-I*)/(*m*-1), assume that 1-χ(*I*)/(*m*-*I*)/(*m*-1)=μ(*I*). Then,

$$\mu(I) = \frac{\ln(m - I)}{\ln(m - 1)},\tag{4}$$

where *I* is defined as the qualitative ranking number of a certain index evaluated by an expert. For example, a set of qualitative ranking numbers 5, 2, 3, 4, 1 for the five indexes *r*1, *r*2, *r*3, *r*4, *r*<sup>5</sup> was evaluated by one expert. Thus, index *r*<sup>5</sup> was the most important, because *I* = 1. *M* is the transformation parameter and defined as *m* = *j* + 2, and *j* is the number of indexes.

The qualitative ranking number*I* is input into Equation (4) to obtain the quantitative transformation value of *bij*. *Bij* = μ(*aij*) is the membership degree of the qualitative ranking number *I*, and the matrix *B* = (*bij*)*k\*n* is defined as the membership degree matrix. A new parameter, average understanding degree *bj,* was introduced to present the consistency degree of the evaluation of index *rj* by *k* experts; its calculation is as follows:

$$b\_{j} = \frac{b\_{1j} + b\_{2j} + \dots + b\_{kj}}{k}.\tag{5}$$

Blind understanding degree σ*<sup>j</sup>* is defined as the uncertainty of the evaluation of index *rj* by *k* experts,

$$
\sigma\_{\dot{j}} = \left| \left\{ \left[ \max\{b\_{1\dot{j}}, b\_{2\dot{j}}, \dots, b\_{k\dot{j}} \} - b\_{\dot{j}} \right] + \left[ b\_{\dot{j}} - \min\{b\_{1\dot{j}}, b\_{2\dot{j}}, \dots, b\_{k\dot{j}} \} \right] \right\} / 2 \right|. \tag{6}
$$

The global understanding degree *Xj* is defined as the degree of the evaluation of every index *rj* by all *k* experts invited,

$$X\_j = b\_j \mathbf{(}1 - \sigma\_j \mathbf{)}.\tag{7}$$

#### (3) Normalized treatment

To obtain the weight of index *rj*, Equation (7) needs further normalized treatment,

$$
\omega\_{\bar{j}} = \chi\_{\bar{j}} / \sum\_{j=1}^{k} \chi\_{\bar{j}}.\tag{8}
$$

Obviously, ω*<sup>j</sup>* > 0 and *k j*=1 ω*j*=1. The ω=(ω*1,* ω*2,* ... ω*j*) was expressed as the weight vector of the index set *R*=(*r*1, *r*2, ... *rj*).

#### 2.2.2. Calculation of the Index Weight

In this study, a total of 12 experts who have worked in the CBM development industry for a long time, including three experts in CBM resource exploration, three experts in CBM mining technology, two experts in coal economy, two experts in energy policy, and two managers of CBM development enterprises. To more comprehensively formulate risk assessment criteria from multiple perspectives, the experts were randomly divided into four groups, each with three persons. In this way, the diverse understandings of various research fields could be fully explored, and different knowledge and experience could be used to perform a qualitative evaluation of each assessment index.

The weight determination of the indexes in the first grade was taken as an example:



(3) The calculated membership matrix *B* was based on Equation (4) and rank matrix *A*, and *m* was set as 8.


(4) The average degree of understanding of a particular dimension from all experts:

$$b\_j = \frac{b\_{1j} + b\_{2j} + b\_{3j} + b\_{4j}}{4} = (0.770, 0.980, 0.960, 0.851, 0.460, 0.638).$$

(5) Based on the previous results and Equations (5) and (6), the blind understanding degree σ*<sup>j</sup>* for the indexes from all experts could be obtained. Then, the evaluation vector *X* could be calculated according to the blind understanding degree σ*<sup>j</sup>* and Equation (7). Finally, the weight of each index could be achieved by the normalized treatment method. The calculated result of each parameter is shown in Table 3. Similarly, the weight distribution of the second grade indexes can be obtained by refining. The index distribution is shown in Table 4. The detailed calculation processes are shown in Appendix B (Tables A2–A7.).


**Table 2.** Expert ranking results.


**Table 3.** Weight distribution of the first grade index.

**Table 4.** Index weights and actual scores.



#### **Table 4.** *Cont.*
