*3.4. System Evaluation Methodology*

To measure the performance of the FD models, the most commonly used evaluation metrics are the false alarm rate (FAR) and fault detection rate (FDR). FAR uses the probability to quantify the occurrence of alarm when there is no fault. FDR uses the probability to quantify the occurrence of the alarm method in the case of actual failure.

According to the threshold calculated above, FARs and FDRs can be expressed as follows

$$FAR = \frac{M\_{\bar{j}}}{M\_{th}} \times 100\% \tag{13}$$

where *Mj* is the number of test statistics higher than the threshold in fault-free conditions, *Mth* is the total number of test statistics.

$$FDR = \frac{B\_j}{B\_{t\text{lt}}} \times 100\% \tag{14}$$

where *Bj* is the number of test statistics higher than the threshold after injection of fault, and *Bth* is the total number of test statistics after fault injection.

Receiver Operating Characteristic (ROC) curves represent the performance of the model at different thresholds. The *X* axis of the curve is the false positive rate, and the *Y* axis is the true positive rate. The ideal is an inverted L-shaped curve [37]. The calculation formulas of the true positive rate (TPR) and false positive rate (FPR) are [38]

$$\begin{aligned} TPR &= \frac{TP}{TP + FN} \\ FPR &= \frac{FP}{TN + FP} \end{aligned} \tag{15}$$

where *TN* is actually the number of samples classified into negative samples, *FP* is actually the number of samples classified into positive samples, *FN* is actually the number of samples classified into negative samples, *TP* is actually the number of samples classified into positive samples.

The Area Under a ROC Curve (AUC) is a comprehensive measure of sensitivity and specificity across all possible threshold ranges. It represents the probability that a classifier will rank randomly selected positive instances higher than randomly selected negative instances. The AUC ranges from 0 to 1. The closer AUC is to 1, the better FD performance [38]. The calculation formula is

$$AUC = \sum\_{i=1}^{N} \frac{(TPR(i) + TPR(i+1))(FPR(i+1) - FPR(i))}{2} \tag{16}$$

### **4. Experimental Results and Discussion**

High-speed train running gears systems are considered to verify the reliability of the proposed algorithms. When the data of the running gears is chosen, and in order to guarantee the consistency of the experiment input, signal data of the running parts was adopted from the same train and the same carriage. In order to guarantee the validity of the data, the monitoring data at the speed of 1000 r/min or above were utilized in the model. The paper uses real data of a running gears system with fault signals to simulate the settings of the experiments very close to the real situations.
