*3.2. Classification of States of Operation*

The prediction accuracy of the trained RF model was determined to be 0.991, corresponding to an OOB score of 0.996. Five-fold cross-validation yielded a mean prediction accuracy of the model of 0.993 ± 0.001. These values indicate a classification error rate between 0.5% and 1%.

Figure 7 shows the locations of the 20 most important features used for splitting nodes in the RF model. The *x*-axis label 'Relative position' refers to a relative position in time during the duration of one stroke rather than an actual physical position. One can see clearly that the most important areas are located around the two turning points of the stroke direction of the steady-state cycles, i.e., around 60 for the change between positive and negative stroke and around 100 or 0 for the change from negative to positive. Another region, where important features are located, can be found around 80, corresponding to the location of turning points of the critical cycles in the positive stroke direction. The feature around 50 may be associated with critical cycles in the negative stroke direction.

**Figure 7.** The 20 features with the highest importance are marked as red crosses.

In order to assess the prediction quality of the RF algorithm on other datasets, the dataset of experiment 8, which was not used for training the RF algorithm, was labelled according to the procedure described above, and classification metrics were calculated. A comparison between the labels assigned to each cycle and the labels predicted by the RF algorithm is shown in Figure 8.

The overall classification accuracy of experiment 8 was 0.939. Table 5 shows the precision and recall values for the four classes. Both steady states as well as the precritical state were recognised with high precision and recall. Of the cycles classified by the algorithm as 'Critical', only 78% were actually labelled as 'Critical'. The remaining 12%, or 188 cycles, had the true label 'Pre-critical'. However, 88% of the actual states labelled as 'Critical' were identified correctly. The corresponding absolute values are shown in the confusion matrix in Figure 9. The colour scale indicates the fraction between predicted labels and the total number of true labels assigned to the respective class, summing up to 1 for each row. For the diagonal elements, this corresponds to the recall. The last row and column, labelled as 'None', indicates cycles, for which the algorithm was not able to issue a prediction. This was predominantly the case for the two steady states, with about 4.5% of the cycles labelled 'Steady2' not classified. This is also the reason for the relatively low recall of 0.9 for this class.

**Figure 8.** (**a**) Manually labelled dataset from Experiment 8, (**b**) the classification made by the trained RF algorithm.



Based on the results of the RF classification, Table 6 shows a summary of the lengths of the pre-critical phases preceding the end of the respective experiment. As already mentioned, the experiments could be divided in two distinctly different groups according to their behaviour towards the end of the experiment. In the first group, an extended pre-critical phase was observed before the termination of the experiment. This pre-critical phase was found to last between 60 and 211 min or between 7.4 and 19.2% of the total running time. Before reaching the stop criterion, individual critical cycles were observed during the pre-critical phase, with an increasing abundance of critical cycles towards the end, as shown in Figure 10.


**Table 6.** Start of pre-critical phase for each experiment.

<sup>1</sup> Experiment 3 was stopped manually, before the stop criterion was reached.

**Figure 10.** Change between pre-critical and critical operation at the end of experiment 1. Critical operation started with a pronounced increase in the friction force at one turning point.

The second group shows a pre-critical and critical operation rather suddenly. The stop criterion was exceeded within less than 10 min. Experiment 6 reached pre-critical operation as few as 1.5 min before termination of the experiment. This sudden critical behaviour may be due to a sudden loss of the lubricant supply, resulting in a pronounced increase in the lateral force, whereas in the first group, lubricant supply was sufficient to keep the system in an operable state over a longer period.
