Research on Hard Rock Pillar Stability Prediction Based on SABO-LSSVM Model

Abstract

The increase in mining depth necessitates higher strength requirements for hard rock pillars, making mine pillar stability analysis crucial for pillar design and underground safety operations. To enhance the accuracy of predicting the stability state of mine pillars, a prediction model based on the subtraction-average-based optimizer (SABO) for hyperparameter optimization of the least-squares support vector machine (LSSVM) is proposed. First, by analyzing the redundancy of features in the mine pillar dataset and conducting feature selection, five parameter combinations were constructed to examine their effects on the performance of different models. Second, the SABO-LSSVM prediction model was compared vertically with classic models and horizontally with other optimized models to ensure comprehensive and objective evaluation. Finally, two data sampling methods and a combined sampling method were used to correct the bias of the optimized model for different categories of mine pillars. The results demonstrated that the SABO-LSSVM model exhibited good accuracy and comprehensive performance, thereby providing valuable insights for mine pillar stability prediction.

1. Introduction

In conventional open stope mining methods, certain ore bodies are left as supporting pillars to maintain the stability of underground mining workings without the presence of additional support or auxiliary structures [1,2]. The strength of the mine pillars, hanging wall, and footwall are critical factors that determine the ability of the void area to withstand disturbances and prevent collapse. Unstable mine pillars can lead to mining safety accidents, such as collapse of the void area and interlocking failure of multiple pillars [3]. For instance, the large-scale instability event of the Lewiston–Stockton coal seam (1986) resulted in the collapse of the void area, spreading to a distance of 100 m within 8 h after the accident. This led to more than 0.5 m of subsidence in the unstable region and triggered numerous instances of tilting and instability of mine pillars [4]. As mining operations continue to extend deeper underground, in situ stress increases rapidly with depth, posing significant challenges and higher strength requirements for supporting mine pillars [5,6]. Consequently, analysis of mine pillar stability has become an inevitable research problem in underground mining.

Early research on mine pillar stability primarily focused on analyzing it from the perspective of rock mechanics. Researchers have continuously developed and optimized empirical formulas for estimating pillar strength to explore the key indicators affecting pillar stability and their functional relationships [7,8]. An earlier estimation formula for pillar strength was developed by Salamon and Munro. They constructed a power function formula based on the pillar width, height, and unit volume strength of the pillars using a collection of coal mine pillar cases, and used it to estimate pillar strength [9]. For a long time afterward, numerous scholars built upon their research and developed various empirical formulas, adhering to the viewpoint that pillar strength is linearly related to material strength while considering negligible effects of size when the volume of the rock mass increases [10,11]. Currently, the widely accepted and applied pillar strength formula is the Lunder-Pakalnis pillar strength formula [12]. This formula, which is based on a large amount of field data, comprehensively considers various factors that influence pillar stability. It is widely applicable to different types of rocks and mining conditions and provides valuable references for pillar design.

With the advancement of research and interdisciplinary development, an increasing number of new methods are being applied to analyze mine pillar stability. Currently, research methods for mine pillar stability analysis mainly include the safety factor method [13], reliability analysis method, statistical learning theory, and numerical simulation methods [2,14].

The safety factor method evaluates the stability of mine pillars by calculating the ratio between the theoretical strength of the pillars and actual working loads. Esterhuizen et al. [15] developed a method for estimating pillar strength and selecting design safety factors based on observations of stable and failed pillars supplemented with numerical models. Zheng et al. [16] proposed a new calculation method, called the point safety factor calculation method, which considers the multidirectional stress state and ultimate strength values at each point, allowing for an even distribution of the point safety factor within the pillar. Thus, the average overall point safety factor can be used to evaluate the overall safety factor of the pillar. The safety factor method is simple and easy to use; however, it does not consider the characteristics of rock mass engineering randomness and heterogeneity, resulting in issues of insufficient accuracy and subjectivity.

The reliability analysis method differs from the safety factor method in that it establishes a stochastic model for reliability assessment, treats the input variables of the pillar as random variables, and describes them using distribution functions. Liu and Zhai [17] proposed a reliability design method to overcome the shortcomings of the safety factor method, which neglects the influence of the inherent joint and fracture characteristics of the rock mass in pillar design. The reliability design method fully considers the non-uniformity and discontinuity of the rock mass and has a greater practical value.

Statistical learning theory involves analyzing the mine pillar stability through reverse analysis based on a large number of actual cases. Yu and Zhao et al. [14] proposed the application of multivariate mathematical statistical methods from nonlinear science in the analysis of mine pillar stability using quantization theory II (QTII) for pillar state discrimination models. Ghasemi et al. [18] applied fuzzy logic to predict the safe sizes of mine pillars based on a dataset of 399 coal mine pillar samples. Although statistical learning theory can provide guidance for engineering practice, it also has limitations in terms of interpretability and the inability to explain the essence of the problem [2].

The numerical simulation method involved establishing a pillar model using computer simulation software and conducting simulation calculations based on reasonable mechanical principles. Yao et al. [19] conducted instability mechanism analyses for different types of underground mine pillar failures, performed three-dimensional numerical simulations using the Drucker–Prager yield criterion for various mining scenarios, and evaluated the sensitivity of the factors affecting pillar stability using orthogonal range analysis.

The stability analysis of mine pillars is a complex nonlinear problem that involves indicators, such as pillar size, rock mass strength, and their interrelationships [14]. Despite attempts by many scholars to use classical methods, such as the safety factor approach and reliability analysis, to comprehensively evaluate pillar stability, there have always been limitations in terms of accuracy and applicability. In recent years, with the advancement of computer technology, neural networks and intelligent algorithms have been widely applied in various fields. Machine learning statistical analysis methods based on monitoring information have become a research hotspot owing to their high accuracy and efficiency [20]. These methods have been applied in studies related to rock-burst prediction, blast casting effect prediction, mine pillar stability prediction, and other mining safety issues. Researchers such as Zhou and Jiang et al. [21,22] have successfully applied machine learning models, such as support vector machines (SVM) and GA-XGB for rock-burst grade prediction, achieving good prediction accuracy. Han et al. [23] combined the Weibull model with a BP neural network for blast pile shape prediction. Sun [24] compared the prediction performance of BP, SVM, and radial basis function (RBF) models in predicting the blasting effect in open-pit mines. In addition to the aforementioned areas, scholars have continuously explored the application of machine learning for predicting the stability grade of mine pillars. Zhou et al. [25] evaluated the performance of SVM and Fisher’s discriminant analysis (FDA) in mine pillar stability prediction, and the results showed that the SVM model performed better. Zhao and Liu et al. [2] used a large number of measured pillar samples to compare ten prediction models, including distance discriminant analysis (DDA), Bayesian discriminant analysis (BDA), extreme learning machine (ELM), and least-squares support vector machine (LSSVM), for mine pillar state recognition from various perspectives.

In summary, the application of machine learning methods to the prediction of mine pillar stability has become a research hotspot. However, there are currently few application cases that optimize classical models using intelligent algorithms for mine pillar stability prediction. In this study, based on Lunder’s mine pillar dataset, the SABO intelligent optimization algorithm is used for parameter optimization of the LSSVM model, and the optimized model is applied to predict the stability grade of mine pillars. First, by analyzing the redundancy of features in the mine pillar dataset and conducting feature selection, five parameter combinations were constructed to examine their effects on the performance of different models. Second, the performance of the SABO-LSSVM model was compared with that of other prediction models. Finally, various data sampling methods were used to correct the bias of the optimized model for different categories of mine pillars.

2. Materials and Methods

2.1. Pillar Dataset

The mine pillar sample dataset used in this study was sourced from Lunder’s database [26]. The dataset consists of 162 complete mine pillar samples, including 58 “stable” samples, 37 “unstable” samples, and 67 “failed” samples. The mine pillar samples in this database primarily consist of large sulfide pillars with a rock mass rating (RMR) ranging from 60% to 85%. The structural characteristics of the pillars were not considered to be stability-influencing factors. The data sources for the dataset included Westmin Resources Ltd., Hudyma (1988), Von Kimmelman et al. (1984), Hedley and Grant (1972), and Sjoberg (1992) [26].

The original dataset features include the width (W) and height (H) of the mine pillar, aspect ratio (K) of the pillar, average pillar constraint (C_pav), uniaxial compressive strength of the rock (UCS), and pillar stress (σ). The pillar stress was calculated using sublevel-area theory or numerical simulation methods. Zhao et al. [2] added pillar strength (σ_p) as a feature in their mine pillar stability prediction. They used Lunder and Pakalnis’ empirical formula to calculate pillar strength, which is shown in Equation (1):

$${\displaystyle \frac{{\mathsf{\sigma }}_{\mathrm{p}}}{0.44\mathrm{U}\mathrm{C}\mathrm{S}}}=0.68+0.52\tan \left({\cos }^{-1}\left({\displaystyle \frac{1-0.46{\left({\log }_{10}\left(K+0.75\right)\right)}^{\frac{1.4}{K}}}{1+0.46{\left({\log }_{10}\left(K+0.75\right)\right)}^{\frac{1.4}{K}}}}\right)\right)$$

(1)

Walton and Sinha et al. [27] conducted an in-depth study on the mechanics of hard rock pillars and re-evaluated the Lunder pillar database. They found that the aspect ratio of pillars varied with changes in brittleness when the strength trend underwent a transition, which can be approximated by the uniaxial compressive strength (UCS). Based on this observation, they developed a new empirical formula for pillar strength and achieved better pillar classification results. In their formula, for pillar samples with 50 MPa < UCS < 350 MPa and 0 < W/H < 3, the recommended formula for estimating the pillar strength is as follows:

$${\displaystyle \frac{{\sigma }_p}{UCS}}=0.405+\left({\displaystyle \frac{0.2408-0.405}{1+e^{(\frac{W}{H}-\frac{UCS}{105}-0.1)/0.3}}}\right)+\left(0.096+\left({\displaystyle \frac{0.063-0.096}{1+e^{(\frac{W}{H}-\frac{UCS}{105}-0.1)/0.3}}}\right)\right){\displaystyle \frac{W}{H}}$$

(2)

For pillar samples with UCS > 300 MPa and W/H > 2, it is recommended to use the following conservative estimation formula to calculate the pillar strength:

$${\displaystyle \frac{{\sigma }_p}{UCS}}=0.462+\left({\displaystyle \frac{0.227-0.462}{1+e^{(\frac{W}{H}-\frac{UCS}{195}-1.42)/0.71}}}\right)+\left(0.0871+\left({\displaystyle \frac{0.0245-0.0871}{1+e^{(\frac{W}{H}-\frac{UCS}{195}-1.42)/0.71}}}\right)\right){\displaystyle \frac{W}{H}}$$

(3)

In the mine pillar dataset used in this study, all the samples satisfied the conditions for using the first formula. Therefore, when the pillar strength feature (σ_p) was added, the calculation was performed using this formula.

The boxplots of the complete dataset are shown in Figure 1. In the boxplots, the upper and lower edges of the box represent the upper and lower quartiles of the data group, the horizontal line inside the box represents the median, the small square represents the mean, the lines extending from the box represent the upper and lower limits of the data, and the black circles represent the outliers.

The graph shows that the distribution of most feature data across different categories was relatively balanced. However, for the three sets of features, pillar width (W), pillar height (H), and uniaxial compressive strength (UCS), the medians of each category’s data showed noticeable deviations. This indicates that the data distribution exhibited skewness, which was mainly due to the significant differences in the sizes of the different mine pillar samples. However, structural characteristics were not considered as stability-influencing factors [26]. Apart from the size parameters, the distributions of the other mine pillar features were relatively balanced, and the number of outliers in each group was within a reasonable range.

2.2. Feature Correlation Analysis

In machine learning classification tasks, conducting feature correlation analysis is crucial, as it helps identify and eliminate redundant or unnecessary features, reduces data dimensionality, simplifies the model, and reduces computational resource consumption [28,29]. In research using machine learning to predict mine pillar stability, several scholars have tested different combinations of features in their datasets. Zhou et al. [25] selected six features: pillar width (W), pillar height (H), aspect ratio of the pillar (K), uniaxial compressive strength of the rock (UCS), pillar stress (σ), and pillar strength (σ_p). Zhao and Liu et al. [2] chose seven features, including pillar width (W), pillar height (H), aspect ratio of the pillar (K), pillar constraint (C_pav), uniaxial compressive strength of the rock (UCS), pillar stress (σ), and pillar strength (σ_p), for mine pillar state recognition. They compared the effects of different feature combinations on various discriminant criteria using feature elimination. Yu et al. [14] selected five features: pillar width (W), pillar height (H), pillar aspect ratio (K), uniaxial compressive strength of the rock (UCS), and pillar stress (σ). These features were used as evaluation indicators in the quantitative theory II discriminant model, and the results showed that this combination performed well in practical applications and had certain engineering significance.

The present study investigated all seven mine pillar indicators and utilized correlation analysis to identify redundant features. To assess the correlation between different sets of features, the Pearson correlation analysis method [30,31] was employed to calculate the correlation coefficients between each pair of features. Outliers in the data can significantly affect the correlation analysis. Before conducting correlation analysis, it is advisable to replace outliers with the mean value within each group. To check for the presence of outliers, boxplots were generated using SPSS software (v.26).

The intergroup correlation coefficients and scatter plots are shown in the figure. From the Figure 2, it can be observed that the groups with higher correlation coefficients included “UCS-σ_p”, “K-C_pav”, and “W-H”. The absolute values of the correlation coefficients were 0.88, 0.81, and 0.59, respectively. The first two groups exhibited a strong correlation relationship, while the “W-H” group showed a moderately strong correlation relationship.

Based on correlation analysis, we identified features that exhibited strong correlations. These highly correlated features provided redundant information as inputs for the predictive model. To simplify the model, it was necessary to eliminate redundant features. The selection process involved considering the correlation coefficients between each set of features and class labels and removing the one with weak correlation. Through the selection of redundant features, this study attempted to use five parameter combinations, namely, M-1, M-2, M-3, M-4, and M-5, as the dataset for the predictive model and explore their impact on predictive performance. The specific parameters of these combinations are listed in

Table 1. Feature combinations for the mine pillar dataset.

	W	H	K	Cpav	UCS	σ	σp
M-1	√	√	√	√	√	√	√
M-2	√	√	√		√	√	√
M-3	√	√	√		√	√
M-4	√		√		√	√
M-5			√		√	√

.

2.3. Introduction of the LSSVM Model

The least-squares support vector machine (LSSVM) is a variant of the support vector machine (SVM) that aims to simplify the optimization process of the standard SVM [32]. This model reduces computational complexity by transforming the original quadratic programming problem into a system of linear equations. Unlike traditional SVM, LSSVM utilizes a least-squares linear system to solve the optimization objective function, which includes a squared error term. This approach performs well in handling small-scale datasets and offers a higher computational efficiency.

In the principle of the LSSVM model, we first need to provide a training set {(x_i, y_i), I = 1, 2, …, n}, where x_i is a d-dimensional vector with dimensions consistent with the input features of the dataset used, such as in the M-1 model mentioned in this study, where d = 7. y_i represents the predicted value of the model, which can be a class label or a regression prediction. The mapping relationship between x_i and y_i is expressed by Equation (4):

$$y_i=\mathit{\omega }\phi \left(x_i\right)+b$$

(4)

where ω is the weight vector, b is the bias value, and φ(x_i) is the mapping function.

The LSSVM model transforms an original quadratic programming problem into a linear equation system. It selects the squared error term to represent empirical risk and formulates the evaluation problem as an optimization problem based on the principle of structural risk minimization:

$$\left\{\begin{array}{l}{min}_{\mathit{\omega },b,e}J\left(\mathit{\omega },\mathit{e}\right)={\displaystyle \frac{1}{2}}{‖\mathit{\omega }‖}^2+c{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^n}{\mathit{e}}_i^2\\ \omega \phi \left(x_i\right)+b+{\mathit{e}}_i=y\end{array}\right.$$

(5)

where c is the penalty parameter (c > 0), e is the error vector, and J is the loss function.

To solve the aforementioned optimization problem, it is necessary to construct the corresponding Lagrangian function and determine the minimum value of the optimization function, as shown below:

$$L\left(\mathit{\omega },b,\mathit{e},\mathit{\xi }\right)=J\left(\mathit{\omega },\mathit{e}\right)-{\displaystyle \sum _{i=1}^n}{\mathit{\xi }}_i\left[\mathit{\omega }\phi \left(x_i\right)+b+{\mathit{e}}_i-y_i\right]$$

(6)

where ξ represents the Lagrange multiplier, ξ = [ξ₁, ξ₂, …, ξ_n]^T.

By taking the partial derivatives of L(ω, b, e, ξ) with respect to ω, b, e, and ξ, and setting the derivatives of L to zero, we can obtain the conditions for the optimal solution of the problem:

$$\left\{\begin{array}{l}\omega ={\displaystyle {\displaystyle \sum _{i=1}^n}}{\mathit{\xi }}_i\phi \left(x_i\right)\\ {\displaystyle {\displaystyle \sum _{i=1}^n}}{\mathit{\xi }}_i=0\\ {\mathit{\xi }}_i=c{\mathit{e}}_i\\ \omega \phi \left(x_i\right)+b+{\mathit{e}}_i-y_i=0\end{array}\right.$$

(7)

By eliminating ω and e from the equation, we can obtain:

$$\left[\begin{array}{c}0\\ \mathit{Y}\end{array}\right]=\left[\begin{array}{cc}0& {\mathit{Z}}^T\\ \mathit{Z}& \mathit{K}+c^{-1}\mathit{E}\end{array}\right]\left[\begin{array}{c}b\\ \mathit{\xi }\end{array}\right]$$

(8)

where Z = [1, 1, …, 1]^T, Y = (y₁, y₂, …, y_n)^T, E is an n-dimensional identity matrix, and K is the kernel function matrix that satisfies Mercer’s condition, K = K(x_i, x_j) = φ(x_i)φ(x_j). The kernel function has the ability to reduce the computational complexity in high-dimensional space and plays a crucial role in constructing high-performance least-squares support vector machines.

Therefore, the final optimization function of LSSVM is:

$$y\left(x\right)={\displaystyle \sum _{i=1}^n}{\xi }_{i}K\left(x,x_i\right)+b$$

(9)

And the kernel function adopts the radial basis function (RBF):

$$\mathit{K}\left(x_i,x_j\right)=\mathit{exp}\left(-{\displaystyle \frac{{‖x_i-x_j‖}^2}{2{\sigma }^2}}\right)$$

(10)

where σ is the bandwidth parameter of the kernel function, σ > 0.

In the application of the LSSVM model, the selection of the hyperparameters c and σ plays a crucial role in the overall performance of the model. Therefore, the subsequent discussion addressed the process of determining these parameters.

2.4. Introduction of the Subtraction Average Optimization Algorithm

The subtraction average-based optimizer (SABO) is an intelligent optimization algorithm based on mathematical behavior, proposed by Trajkovski and Dehghani [33] in 2023. It updates the positions of individuals in the search space using the subtraction average of individuals, exhibiting strong optimization capabilities and fast convergence speed. Currently, there are a limited number of cases in which this algorithm has been effectively developed and applied. Hong et al. [34] addressed the issues of insufficient harmonic data support and inadequate harmonic monitoring capabilities in power grid harmonic pollution. They proposed an improved SABO-BP algorithm for harmonic prediction, which achieved good results and had engineering significance.

The SABO algorithm refers to the solution space in different optimization problems as the search space, which is a subset of the dimensional space with dimensions consistent with the number of features in the given training set. The individuals in this algorithm are called search agents, contain information about the decision variables, and are mathematically modeled using vectors. Their collection forms the overall algorithm. Similar to other optimization algorithms, the SABO algorithm requires a random initialization of the population in the optimization space:

$$x_{i,j}={lb}_j+r\left({ub}_j-{lb}_j\right)$$

(11)

where x_i,j represents an individual in the population, lb_j represents the lower bound of the optimization interval, ub_j represents the upper bound of the optimization interval, and r ∈ [0, 1].

Trajkovski and Dehghani incorporated mathematical concepts, such as the average value, differences in search agent positions, and differences in the signs of the objective function values, when designing the SABO algorithm. In traditional approaches, search agents are updated by calculating the arithmetic mean positions of all the individuals. However, the SABO algorithm introduces a special operation called “−v” or “v-subtraction” from search agent B to search agent A. The formula used is as follows:

$$\mathit{A}-v\mathit{B}=sigum\left(F\left(\mathit{A}\right)-F\left(\mathit{B}\right)\right)\left(\mathit{A}-\overrightarrow{v}\ast \mathit{B}\right)$$

(12)

where $\overrightarrow{\mathit{v}}$ is a vector of randomly generated data from the set {1, 2}, F(A) and F(B) are the fitness values of individuals A and B, the sign function represents the positive or negative sign of F(A) − F(B), and “*” denotes the Hadamard product (i.e., multiplication of the corresponding components of two vectors). The flowchart of SABO is shown in Figure 3.

It can be observed from the diagram that when F(A) − F(B) > 0, A − vB =A − $\overrightarrow{\mathit{v}}$ ∗ B, and the resulting operations are distributed around the red line region. Conversely, when F(A) − F(B) < 0, A − vB = $\overrightarrow{\mathit{v}}$ ∗ B − A, and the resulting operations are distributed around the blue region. Principle of the operation process $\overrightarrow{\mathit{v}}$ is shown in Figure 4.

This unique “−v” operation is applied throughout the process of updating the search agents in the algorithm. In the search space, the displacement of each search agent, X_i, is calculated using the arithmetic mean of “v-subtraction” from search agent X_j. The formula is as follows:

$$X_i^{new}=X_i+{\overrightarrow{r}}_i{\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=1}^N}(X_i-v{X}_j),i=1,2,\dots ,N$$

(13)

$$X_i=\left\{\begin{array}{l}{X}_i^{new},F_i^{new}<F_i\\ X_i, else\end{array}\right.$$

(14)

where N represents the population size, r ∈ [0, 1], and X_new represents the updated position of the search agent. The update condition is shown in Equation (14), where the new position is only replaced with the update value if it is superior (i.e., has a lower fitness value). Otherwise, the position remains unchanged.

It is worth mentioning that the SABO optimization algorithm requires fewer control parameters than other algorithms. It only requires setting the population size N and maximum number of iterations, making it simple and efficient. In performance tests on unimodal, high-dimensional multimodal, fixed-dimension multimodal types, and the CEC 2017 test suite, the SABO optimization algorithm showed good solving effectiveness for most benchmark functions.

2.5. Oversampling Methods

Imbalanced data are often encountered in the practical applications of machine learning, where the accuracy of the minority class, which has a smaller proportion, is often not ideal. Therefore, oversampling algorithms were used. The role of oversampling methods in addressing imbalanced data problems is to increase the number of samples in the minority class, balance the sample distribution across different classes, and improve the performance of the classifiers [35,36]. They can enhance the classification accuracy of the minority class, prevent information loss, and strengthen the robustness and generalization ability of the model. However, oversampling methods must be carefully selected and combined with other processing techniques to avoid issues, such as overfitting and increased sensitivity to noise.

Common oversampling algorithms include synthetic minority oversampling (Smote), adaptive synthetic sampling (Adasyn), and random oversampling (Ros). Smote increases the number of samples in the minority class by synthesizing new samples in the feature space rather than simply duplicating existing samples, which differentiates it from Ros’s random generation of samples, and the principle of smote is shown in Figure 5. The specific method involves randomly selecting minority class samples and generating new sample points between their nearest neighbors, thereby expanding the distribution of minority class samples and improving the classifier’s ability to recognize the minority class.

The Adasyn algorithm is an improvement of the Smote algorithm. Unlike Smote, Adasyn adaptively generates new samples by focusing on minority class samples that are difficult to classify. Specifically, Adasyn determines the number of new samples to be generated based on the density of the minority class samples, generating more samples in regions that are challenging to classify.

In methods for balancing datasets, in addition to oversampling the minority class, under-sampling can be applied to boundary and noisy data. Tomek–Smote sampling is an approach that combines these two concepts. It was originally proposed by Batista et al. in 2003, and often achieves better model accuracy than under-sampling alone. The key concept of this sampling method is Tomek Links [36], an under-sampling technique developed by Ivan Tomek in 1976. It is inspired by the condensed nearest neighbors (CNN) and primarily balances the dataset by removing sample pairs (also known as Tomek Links pairs) with the smallest Euclidean distance between different classes. Principle of the Tomek Links is shown in Figure 6.

As shown in the illustration, the data group within the blue circles represents the Tomek Links pairs. During the under-sampling process, the samples removed from Class 2 included both the newly generated samples from Smote and the original samples located at the decision boundary. This approach helps reduce the noise introduced by oversampling algorithms and cleans up confused samples between classes, making it more beneficial for the model’s predictions.

3. Model Construction and Optimal Selection of Dataset Balancing Methods

When LSSVM is applied to practical prediction cases, it is often combined with intelligent optimization algorithms to optimize the hyperparameters of the model itself in order to achieve better accuracy. In this study, we attempted to use the novel SABO algorithm for model optimization and applied it to the identification of stability states in mine pillars. Furthermore, to evaluate the optimization effects objectively and comprehensively, it is necessary to perform both horizontal and vertical model comparisons. In the horizontal comparison, the prediction results of the SABO-LSSVM model need to be compared with those of the initial models, such as LSSVM, SVM, and ELM, to assess whether the optimization algorithm effectively improves the model or has a negative impact. In addition, there are various choices for optimization algorithms, and the effectiveness of optimizing the initial model is a criterion for selecting them. Therefore, in this study, in addition to applying SABO for optimization, we also selected classical and recent optimization methods as references. The classical methods included the genetic algorithm (GA) and particle swarm optimization (PSO), while the recent methods include grey wolf optimization (GWO), the whale optimization algorithm (WOA), and the Kepler optimization algorithm (KOA).

3.1. Construction of Optimized Model

During the training process of the predictive model, it was not trained on raw data. To eliminate the differences between different feature scales and prevent features with larger value ranges from having a disproportionate impact on model training, it is necessary to normalize the initial dataset [37,38]. Normalization methods include min–max scaling and standardization, with this study primarily selecting the former. Min–max scaling achieves normalization by linearly transforming each feature value according to the following formula:

$$x\mathrm{\prime }={\displaystyle \frac{x-\mathit{min}\left(x\right)}{\mathit{max}\left(x\right)-\mathit{min}\left(x\right)}}$$

(15)

where x represents the original feature value and min(x) and max(x) are the minimum and maximum values of that feature, respectively. Using this method, each feature value was scaled to a range of 0 to 1. This approach not only balances the impact of different features on the model, but also speeds up the convergence of gradient descent during the training process, improving the overall performance and stability of the model. Additionally, normalized data are better suited to the assumptions of various machine learning algorithms, ensuring that the model performs better during training and prediction.

K-fold cross-validation is widely used in the field of machine learning algorithms [39,40] and can be divided into model selection and model evaluation based on its purpose. When using the LSSVM, SVM, and ELM models, this study employed ten-fold cross-validation to select the best hyperparameter combination for predicting the stability of mine pillars. For the LSSVM and SVM models, the RBF function was used as the kernel function, so the hyperparameters to be determined were γ and σ, whereas the ELM model’s hyperparameter was the number of hidden layer units, n. This study used a grid search to systematically evaluate the performance of different hyperparameter combinations, and the parameter values are listed in

Table 2. Range of values for model hyperparameter grid search.

Model	Hyperparameter	Range of Values
LSSVM SVM	γ	{2−8, 2−6, 2−4, 2−2, 20, 22, 24, 26, 28}
	σ
ELM	n	{10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000}

.

When using datasets with five different feature combinations, the optimal combinations for the two hyperparameters γ and σ of the LSSVM model were concentrated in the range of [2⁻⁴, 2⁴]. For the SVM model, the optimal position for the penalty parameter (also known as the regularization parameter), γ, is [2³, 2⁵], and for σ is [2⁻⁶, 2⁻⁸]. The ELM model achieved the best prediction performance with a hidden layer neuron count between 20 and 60.

The basic principle of the LSSVM is described in the Materials and Methods Section, and it is evident that the selection of hyperparameters significantly affected the performance of the model. The default LSSVM model generally employs a simple one-by-one search method for hyperparameter selection, which is ineffective in terms of the search range and precision. Therefore, the emergence of intelligent algorithms has effectively addressed this problem [41,42]. Algorithms such as the GA and PSO simulate evolutionary or physical processes in nature, enabling efficient search and optimization in a larger parameter space. These algorithms utilize population search and random selection mechanisms, allowing the exploration of a wider range of parameter combinations and finding global optimal solutions to avoid local optima. The flowchart of the SABO-LSSVM prediction model is shown in Figure 7.

The specific process of mine pillar stability state recognition based on the SABO-LSSVM model is as follows:

Step 1. Prepare the dataset for model training and perform pre-processing. The mine pillar sample dataset was randomly divided into a 70% training set and a 30% testing set, and the data were normalized for model training.

Step 2. Build the initial LSSVM model. First, we determined the type of kernel function to be used in the LSSVM model. Common options include linear kernel, polynomial kernel, and radial basis function (RBF). In this study, a commonly used RBF function was selected. Second, we defined the range of model parameters that needed to be optimized, namely, the regularization parameter, γ, and kernel function parameter, σ. Based on the experience of the grid search optimization mentioned earlier, the optimal parameter combinations were concentrated in the range of [2⁻⁴, 2⁴]. Therefore, the lower and upper bounds for the search of parameters γ and σ were set to 2⁻⁴ and 2⁴.

Step 3. Initialize the SABO optimization algorithm. Before applying this algorithm, we input the optimization problem and relevant information, and determined the population size and number of iterations for SABO. It is worth mentioning that compared to other optimization algorithms, SABO does not require complex control parameter settings. It only requires the determination of the number of population members and the maximum number of iterations to initialize the optimization algorithm. This is an advantage of the SABO algorithm.

Step 4. The search agent positions are updated. After completing the initialization settings, the search agents can be optimized using the “−v” operator in Formula (13). After each position update, the search agent must obtain the corresponding fitness function value. Only when this value meets the screening condition in Equation (14) will it be saved and used in the next iteration.

Step 5. Iterative loop. If the fitness values of the search agents and number of iterations during the optimization process do not meet the requirements, return to Step 4 and repeat the process until the termination condition is satisfied. After the optimization was completed, the optimized hyperparameter values were input into the LSSVM model, and the entire process of optimizing the model for mine pillar state prediction was finalized.

Step 6. Multiple runs and average results. The results of the optimized model may fluctuate within a certain range for each run. Therefore, to comprehensively evaluate the accuracy of the model predictions, all steps need to be repeated, and the results of each prediction should be recorded. In addition to the SABO optimization model, the five comparative optimization models used in this study also needed to complete the prediction under the same conditions. Considering that the WOA-LSSVM model requires 90 s for a single prediction, completing the entire prediction process for the five parameter combinations, M-1, M-2, M-3, M-4, and M-5, would take a total of 25 h. Increasing the number of repetitions would incur significant time costs, which is not in line with the efficiency requirements. Therefore, this study attempted to set the number of repetitions to 200.

3.2. Workflow for Balancing Dataset

In the process of comparing the predictive performance of different optimization models, the dataset was balanced using the Smote oversampling algorithm to effectively improve the overall prediction efficiency. Using only one sampling method inevitably leads to suboptimal or even negative effects on optimization. Therefore, after completing a comprehensive comparison of various models, to further improve the prediction accuracy of the best model, this study attempted to use multiple sampling methods and select the optimal one. As described in the Materials and Methods Section, the main focus of this research was on the Smote, Adasyn, and Tomek–Smote sampling methods.

Both Smote and Adasyn are oversampling interpolation algorithms that share the same basic principles. The main tool used in these two methods was MATLAB R2021b. The original dataset was imported into the main file, and a balanced dataset was obtained after code execution. During the interpolation process, Smote uniformly generates new data for all minority class samples, whereas Adasyn adaptively generates new samples by focusing on minority class samples that are difficult to classify correctly. It automatically adjusts the number of newly generated samples to make the model pay more attention to samples in the boundary region.

Tomek–Smote is a combination of oversampling and under-sampling methods. Implementing this sampling method required the use of Python tools (v.3.11), including sampling functions in the imblearn and sklearn libraries. Unlike the individual sampling methods, the Tomek–Smote sampling method consists of two steps.

(1)

Preliminary oversampling using Smote. Similar to the individual Smote oversampling method, new sample points were constructed through linear interpolation between minority class samples until the imbalance between different classes was resolved.

(2)

Cleaning noise and boundary data using Tomek Links. Tomek Links pairs on class boundaries were identified and removed based on the minimum geometric distance principle to achieve a new balance.

The dataset used in this study included five parameter combinations: M-1, M-2, M-3, M-4, and M-5. After completing the model optimization step, the best-performing combination among these five was also required for further research on sampling methods.

Taking the M-3 combination as an example, this dataset contained five pillar features: pillar width (W), pillar height (H), pillar width-to-height ratio (K), uniaxial compressive strength of the rock (UCS), and pillar stress (σ). It included 162 sample cases with proportions of 35.80%, 22.84%, and 41.36% for “stable”, “unstable”, and “failure” samples, respectively. It is evident that the “unstable” pillar samples were significantly underrepresented among these three classes, indicating the need to balance the dataset.

It can be observed from the pie chart in Figure 8 that after balancing the default dataset, the pillar samples in the three categories appeared to be evenly distributed. Among the Smote group, the sample distribution was the most balanced, while the Adasyn and Tomek–Smote groups, although not entirely equal, also exhibited a relatively balanced distribution at a 1:1:1 ratio.

4. Results and Discussion

The performance of the SABO-LSSVM model in predicting pillar stability needs to be evaluated from both vertical and horizontal perspectives. First, the impact of the SABO optimization process on the original model must be analyzed by comparing the average accuracy of the model before and after optimization to determine whether it had a positive or negative effect. Second, the LSSVM can be combined with various optimization algorithms. To assess the optimization ability of SABO compared with other algorithms, a comprehensive comparison of the performance of SABO-LSSVM with common optimization models is necessary.

4.1. Models Performance Comparison

To validate whether the improved LSSVM model performed better, it is necessary to compare the prediction accuracy of the model before and after improvement. To examine the effectiveness of traditional pillar classification methods, safety factor analysis was also employed. The pillar safety factor evaluation involves assessing the stability level of an object by calculating the ratio of the pillar’s own strength “σ_p” to the actual load “σ” it bears, denoted as K = σ_p/σ, where K represents the safety factor index. According to commonly used stability rating criteria, the pillar cases can be classified into three states: K_z ≥ 2 indicates a stable state, 1.2 ≤ K_z < 2 signifies an unstable state, and K_z < 1.2 corresponds to a failure state.

From the statistical

Table 3. Vertical comparison of the prediction results between the SABO-LSSVM model and the basic models (%).

	SABO-LSSVM	LSSVM	SVM	ELM	SF	Average
M-1	84.74	71.43	47.62	70.95	49.38	64.82
M-2	85.33	61.90	47.62	71.94	49.38	63.23
M-3	85.58	71.43	52.38	68.10	/	69.37
M-4	83.04	61.90	52.38	60.00	/	64.33
M-5	80.04	57.14	57.14	63.81	/	64.53
Average	83.75	64.76	51.43	66.96	49.38

, it can be observed that the average accuracy of SABO-LSSVM in the five datasets improved by 18.99% compared to that of LSSVM. It also showed improvements of 30.32% and 16.78% compared with the SVM and ELM models, respectively. Therefore, the original model, after parameter optimization with the SABO algorithm, can effectively enhance the accuracy of pillar state prediction. The classification results of the pillar cases using the safety factor (SF) method showed significantly lower accuracy compared to other machine learning methods. Specifically, with parameter combinations M-1 and M-2, the SABO-LSSVM model achieved accuracy rates 35.36% and 35.95% higher, respectively, than the safety factor method.

Additionally, to compare the effectiveness of SABO with that of other optimization algorithms, a horizontal comparison of the optimization algorithms is required. From the results in the statistical

Table 4. Horizontal comparison of the prediction results between the SABO-LSSVM model and other optimized models (%).

	SABO	PSO	GWO	GA	KOA	WOA	Average
M-1	84.74	82.93	81.93	73.28	78.92	79.30	80.18
M-2	85.33	83.46	83.53	73.82	79.29	79.52	80.82
M-3	85.58	84.28	83.52	74.73	79.72	79.57	81.23
M-4	83.04	83.18	83.23	73.66	76.62	76.09	79.30
M-5	80.04	80.87	80.53	73.83	75.93	73.48	77.45
Average	83.75	82.94	82.55	73.86	78.10	77.59

, it can be observed that for datasets composed of different indicator combinations, the SABO-LSSVM model had the highest average prediction accuracy of 83.75%. The PSO-LSSVM and GWO-LSSVM models followed, with average accuracies of 82.94% and 82.54%, respectively. The models optimized using GA, WOA, and KOA showed slight improvements over the initial model, but their average accuracies were all below 80%, which is significantly lower than that of the top three models.

In order to further evaluate the prediction results exhibited by the LSSVM models optimized by different algorithms, this study presents the overall distribution of the prediction results from six optimization models in the form of cumulative distribution curves, the results are shown in Figure 9. These curves provide a visual representation of the concentrated distribution areas of prediction accuracy on the test set for each model. The closer the curve is to the bottom-right corner, the more concentrated the results are in the high-accuracy range [43], indicating a more stable and accurate prediction capability of the model. Origin was used to plot the curves, and the Boltzmann nonlinear fitting function was employed to fit the cumulative distribution curves. Orthogonal distance regression was used as the iteration algorithm, resulting in a good fit of the processed curves.

From the impact of different indicator combinations on various models, the M-3 combination, after excluding C_pav and σ_p, resulted in an overall average accuracy of 81.23% (as shown in the boxplot), which was slightly higher than that of the M-1 and M-2 combinations, which had average accuracies of 80.18% and 80.82%, respectively. Thus, the accuracy of the model improved slightly. However, when the number of features was further reduced, the average accuracy of the models began to decrease significantly. From the statistical table, it can be observed that the M-4 combination, which removed pillar height, H, from the M-3 combination, experienced a decrease in average prediction accuracy by 1.93%. When pillar width, W, was also removed, the average accuracy decreased by 3.79% compared to that of M-3.

Looking at the response of each model, both the LSSVM and the six optimized LSSVM models exhibited similar changes in their results during the feature selection process, following the overall trend of the average values. The ELM model showed relatively low sensitivity to these manipulations, with fluctuating overall trends, and parameter combination M-2 achieved the highest accuracy. However, the SVM model demonstrated a gradually increasing trend in prediction accuracy when redundant features were removed, contrary to the trends observed in the other models.

This indicates a strong correlation between the aspect ratio, K, and C_pav, as well as between the uniaxial compressive strength (UCS) and σ_p, when used as input for the models. The removal of redundant features can effectively improve the predictive ability of some models. Although pillar height and pillar width exhibited only slight correlations during the correlation analysis, removing either of them during the actual model prediction weakened the model’s performance to varying degrees. Therefore, it was necessary to consider the inclusion of both W and H as features when constructing the dataset. There is a relationship between the aspect ratio, K, W, and H, but W and H represent the dimensional characteristics of the pillar and K represents the shape characteristic of the pillar, and they together constitute the pillar dataset as distinct fundamental features. Therefore, when eliminating redundant features, the aspect ratio, K, cannot be disregarded.

From the graph, it can be observed that for the M-1, M-2, and M-3 combinations, the SABO-LSSVM model performed the best, followed closely by the PSO-LSSVM and GWO-LSSVM models. The GA-LSSVM, KOA-LSSVM, and WOA-LSSVM models had lower prediction accuracies, indicating that the SABO-optimized model outperformed the baseline model and other optimization models in the first three parameter combinations. The overall performance of the models in the M-4 and M-5 combinations declined compared with the first three combinations, with the most significant decline observed in the M-5 combination. In these two parameter combinations, the performance of the SABO optimization was comparable to that of PSO and GWO, but in M-5, the latter had a slight advantage, indicating that the two feature removals diminished the predictive performance of the optimized models, with the SABO-LSSVM being the most affected.

The cumulative distribution graph provides an overall view of the distribution of prediction results for each model. To further compare quantitatively, line graphs of A10, A50, and A90 were plotted to assess the accuracy performance of the prediction results in different ranges, the results are shown in Figure 10. Taking A10 as an example, the y-axis represents the upper limit of accuracy when the cumulative distribution value of the prediction results of the model reached 10%, indicating that 10% of the prediction results in the sample were lower than or equal to this value.

The A10 comparison graph reflects the upper limit of accuracy for each model when they performed poorly. A lower value indicates that the model was more likely to incorrectly predict pillar samples in the test set. From the graph, it can be seen that the SABO-LSSVM model had the best A10 value in the M-1 and M-2 combinations and ranked high in the other three combinations. The prediction results of the PSO, GWO, and WOA optimization models fluctuated between 70% and 78% when performing poorly, whereas the KOA and GA optimization models performed poorly, mostly below 70%.

The A50 graph shows that the SABO-LSSVM model achieved the highest value when using the M-3 combination dataset, indicating that at least half of the prediction results were below 84.85%. The best A50 values for the PSO-LSSVM and GWO-LSSVM models were 83.45% and 82.75%, respectively, indicating that, under the same number of runs, the SABO-optimized LSSVM model slightly outperformed the latter and other optimization models in terms of half of the predicted samples.

The A90 graph reflects the ability of different models to achieve good prediction results. From the graph, it can be seen that the SABO-LSSVM model had an A90 value below 92% for the top 90% of the sample prediction accuracy, while the A90 values for the other models were below 90%, indicating a higher likelihood of higher accuracy for the SABO-optimized model compared to the other models. The WOA and GA optimization models performed poorly in terms of A90, with a maximum value of only 82%, indicating that these two optimization models had poor effects on improving the original model.

In summary, the SABO-LSSVM model had the best accuracy in identifying pillar states, followed by the PSO-LSSVM and GWO-LSSVM models. The KOA-, WOA-, and GA-optimized models performed poorly. Among the five parameter combinations used in the SABO-LSSVM model, the best was the M-3 combination, which consisted of features W, H, K, UCS, and σ.

4.2. Performance of Different Balancing Methods

The oversampling methods used in this study were the Smote and Adasyn algorithms. Additionally, a balancing method that combines an under-sampling algorithm, called the Tomek–Smote sampling method, was employed. These methods were applied to balance the dataset of the SABO-LSSVM model under the M-3 parameter combination. The results of each method are shown in Figure 11.

It can be observed from Figure 11 that the SABO-LSSVM model performed poorly in predicting the second-class (unstable) samples when using the unbalanced (Default) dataset. Out of the 14 samples, eight were incorrectly predicted, resulting in an accuracy of only 42.86% for this class. This indicates that an imbalanced dataset can lead to inconsistent prediction results for different classes, as the samples from the minority class contribute less information during model training, resulting in significantly lower accuracy compared with other classes.

Therefore, various oversampling algorithms can be employed to supplement an imbalanced dataset with new samples, thereby achieving a balanced representation of each class. After applying the Smote and Adasyn algorithms for oversampling, there was a noticeable improvement in the ability of the model to identify unstable pillar samples. The number of incorrectly predicted samples from the second class was reduced from eight to two, and the accuracy increased to 85.71%. Thus, by supplementing samples from the minority class, the information provided to the model was enhanced, resulting in improved prediction accuracy. However, after oversampling, some of the originally correctly predicted samples from the other two classes were misclassified. For example, in the “stable” and “failed” pillar samples, one sample from each class was predicted as unstable after oversampling, whereas these samples were correctly predicted in the default dataset’s model predictions. This also highlights the negative impact of oversampling, as it introduces noise into the dataset.

To balance the sample representation while removing noise, this study explored the use of the Tomek-Links-improved Smote algorithm, the Tomek-Smote balancing method. This method removes abnormal samples caused by dataset recombination from a Smote-balanced dataset, thereby reducing the noise introduced during the balancing process. From the graph, it can be observed that the Tomek-Smote algorithm, compared to the Smote algorithm, corrected one misclassified sample in both the “stable” and “failed” pillar samples due to noise. This suggests that Tomek Link under-sampling, when applied, removed the noise samples introduced by Smote, effectively improving the prediction accuracy of these two classes.

In order to quantitatively analyze the performance of different balancing algorithms in the model, various evaluation metrics must be calculated, as shown in the table. The quantitative analysis of different models primarily included an overall and classification evaluation. The overall evaluation calculated the metrics for all samples, whereas the classification evaluation considered metrics for each specific pillar sample class. In the classification evaluation, precision and recall were calculated for each class to assess the model’s ability to distinguish between positive and negative samples in different classes. Precision and recall had mutually exclusive relationships. If only precision is considered to be increased by predicting samples with a high probability as positive, it may lead to missing positive samples with lower probabilities, resulting in a decrease in recall. To strike a balance between the two, the F1 score was calculated to provide a comprehensive assessment of the classification performance of the model [44]. For ease of understanding, we list the meaning of the relevant indicators and how to calculate.

Table 5. Confusion matrix indicator meaning.

		Predicted Value
		Positive	Negative
True value	Positive	TP	FN	TPR
	Negative	FP	TN	TNR
		PPV	FPR	ACC

shows meaning of confusion matrix, and

Table 6. Formulas and significance of model performance evaluation metrics.

Index	Formula	Significance
Accuracy (ACC)	$ACC={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$	Proportion of correctly predicted samples among all predictions
Precision (PPV)	$PPV={\displaystyle \frac{TP}{TP+FP}}$	Proportion of correctly predicted positive samples among positive predictions
False-positive rate (FPR)	$FRP={\displaystyle \frac{FP}{FP+TN}}$	Proportion of incorrectly predicted negative samples among negative predictions
Recall (TPR)	$TPR={\displaystyle \frac{TP}{TP+FN}}$	The ratio that is accurately predicted in an actual positive example
Specificity (TNR)	$TNR={\displaystyle \frac{TN}{TN+FP}}$	The ratio that is accurately predicted in an actual negative example
F1 score	$F1 score={\displaystyle \frac{2PPV\ast TPR}{PPV+TPR}}$	The harmonic mean of precision and recall
Micro F1	$Micro F1={\displaystyle \frac{2{PPV}_{micro}\ast {TPR}_{micro}}{{PPV}_{micro}+{TPR}_{micro}}}$	Calculate F1 score using precision and recall based on the overall sample set without distinguishing between classes
Macro F1	$Macro F1={\displaystyle \frac{2{PPV}_{macro}\ast {TPR}_{macro}}{{PPV}_{macro}+{TPR}_{macro}}}$	For class distinction, calculate average precision and recall for each class separately to compute F1 score

shows formulas and meanings of model performance metrics.

The F1 score is commonly used for binary classification problems. For the multi-class problem in this study, the concepts of micro-average (Micro F1) and macro-average (Macro F1) must be introduced [45]. When calculating the F1 score, a particular class was considered the positive class, while the other samples were treated as negative samples. When calculating Micro F1, all class samples were combined, and the formula is as follows (where N represents the total number of samples):

$${Precision}_{micro}={\displaystyle \frac{{\sum }_{i=1}^{N}TP}{{\sum }_{i=1}^{N}TP+{\sum }_{i=1}^{N}FP}}$$

(16)

$${Recall}_{micro}={\displaystyle \frac{{\sum }_{i=1}^{N}TP}{{\sum }_{i=1}^{N}TP+{\sum }_{i=1}^{N}FN}}$$

(17)

$$Micro F1={\displaystyle \frac{2{Precision}_{micro}\ast {Recall}_{micro}}{{Precision}_{micro}+{Recall}_{micro}}}$$

(18)

When calculating Macro F1, it is necessary to first calculate the precision and recall for each class and then take the arithmetic average.

The formula is as follows (where L represents the number of classes):

$${Precision}_{macro}={\displaystyle \frac{{\sum }_{i=1}^N{Precision}_i}{\left|L\right|}}$$

(19)

$${Recall}_{macro}={\displaystyle \frac{{\sum }_{i=1}^N{Recall}_i}{\left|L\right|}}$$

(20)

$$Macro F1={\displaystyle \frac{2{Precision}_{macro}\ast {Recall}_{macro}}{{Precision}_{macro}+{Recall}_{macro}}}$$

(21)

Micro F1 and Macro F1 have slightly different applications. Micro F1 tends to favor datasets with balanced distributions, whereas Macro F1 tends to emphasize small-sample classes. Generally, both evaluation metrics can be applied when the sample class distribution is relatively balanced. If the importance of small-sample events is higher in practical applications, Micro F1 can be considered. On the other hand, if the importance of large-sample events is higher, the Macro F1 metric should be considered.

From

Table 7. Calculation results of model performance evaluation metrics.

Model (Sampling Method)	Accuracy	Micro F1	Macro F1	Class	Precision	Recall	F1 Score
SABO-LSSVM(Tomek–Smote)	0.9286	0.9286	0.9284	1	0.9333	1.0000	0.9655
				2	1.0000	0.8571	0.9231
				3	0.8667	0.9286	0.8966
SABO-LSSVM(Smote)	0.8810	0.7857	0.7708	1	0.7368	1.0000	0.8485
				2	0.8750	0.5000	0.6364
				3	0.8000	0.8571	0.8276
SABO-LSSVM(Adasyn)	0.9048	0.8095	0.7939	1	0.7368	1.0000	0.8485
				2	1.0000	0.5000	0.6667
				3	0.8125	0.9286	0.8667
SABO-LSSVM(Default)	0.8095	0.8095	0.7939	1	0.7368	1.0000	0.8485
				2	1.0000	0.5000	0.6667
				3	0.8125	0.9286	0.8667
LSSVM(Tomek–Smote)	0.7857	0.8095	0.8040	1	0.9286	0.9286	0.9286
				2	0.8889	0.5714	0.6957
				3	0.6842	0.9286	0.7879
SVM(Tomek–Smote)	0.6190	0.6190	0.5583	1	1.0000	0.8571	0.9231
				2	0.3333	0.0714	0.1176
				3	0.4815	0.9286	0.6341

, it can be observed that the SABO-LSSVM model based on the M-3 parameter combination, when using the Tomek–Smote balancing algorithm, had a higher accuracy than the LSSVM and SVM models under the same conditions, with improvements of 14.29% and 30.96%, respectively. This indicates that even after removing some noise from the oversampled samples using the Tomek Links, the SABO-LSSVM model outperformed the latter two in terms of predictive performance. Additionally, after optimizing the LSSVM model with the Tomek algorithm, the gap in accuracy between the LSSVM and SABO-LSSVM models was slightly reduced. Furthermore, the SABO-LSSVM model had higher Micro F1 and Macro F1 values, compared to the LSSVM and SVM models, respectively. Among the three categories of pillar samples, the “unstable” category had a significantly better performance in terms of the F1 score when predicted using the SABO-LSSVM model, with improvements of 22.74% and 80.55% compared to the LSSVM and SVM models, respectively. Therefore, the overall performance of the SABO-optimized model was superior to that of the unoptimized models under the same conditions.

To compare the effects of different sampling methods on the prediction results of the model, a horizontal comparison between balancing algorithms is required. In terms of overall prediction accuracy, the SABO-LSSVM model without data balancing achieved an accuracy of 80.95%. After balancing the dataset using the Smote and Adasyn oversampling algorithms, the accuracy of the model was improved by 7.15% and 9.53%, respectively. The Tomek–Smote group achieved an overall accuracy of 92.86%, which was 11.91% higher than that of the Default group, showing the most significant improvement.

Similar to the occurrence of sample misclassifications in the previous prediction results, the F1 scores of the “stable” and “failed” pillar sample categories in the oversampling groups showed a decrease compared to the Default group. The Adasyn group showed no significant change, whereas the Smote group experienced a decrease of 3.03% and 4.21% in the F1 scores of the second and third categories, respectively. This suggests that the introduction of new samples through oversampling introduced noise that can affect the predictive performance of certain categories. The Tomek–Smote group showed improvements of 11.7%, 25.62%, and 2.99% in the F1 scores of the three categories, compared to the Default group. Therefore, when using the Tomek–Smote balancing algorithm, not only can the overall model prediction efficiency be effectively improved, but it can also mitigate the negative impact of noise information on certain category samples, comprehensively enhancing the predictive capability of the model.

5. Conclusions

After optimization using the SABO algorithm, the average prediction accuracy of the LSSVM model reached 83.75%. Compared to the conventional LSSVM, SVM, and ELM models under the same conditions, it improved by 18.99%, 30.32%, and 16.78%, respectively, demonstrating excellent performance in predicting the state of the pillar samples.

Compared with other optimization algorithms, the SABO algorithm showed better overall optimization performance for the LSSVM prediction model. When using different feature combinations, the SABO-LSSVM model generally outperformed the other algorithms in terms of prediction accuracy. From the cumulative distribution curves, it can be observed that the SABO-optimized model had a higher likelihood of achieving high accuracy (above 90%) and higher numerical values of prediction accuracy, indicating that the model had superior generalization and stability compared with other optimization models.

In this study, seven pillar feature indicators, including pillar width (W), pillar height (H), pillar aspect ratio (K), average pillar constraint (C_pav), uniaxial compressive strength of rock (UCS), pillar stress (σ), and pillar strength (σ_p), were selected as input values for training the model, and redundant feature analysis and selection were conducted. From the performance of different datasets in various models, the M-3 feature combination performed the best, which is consistent with the results cited in [2,14]. This indicates that it is necessary to consider the pillar width (W) and pillar height (H) during feature selection, and not treat them as redundant features to be removed. In conclusion, the optimized prediction model used in this study performed best when using the M-3 feature combination as the dataset, which included W, H, K, UCS, and σ.

To further investigate the impact of data balancing methods on the SABO-LSSVM model, this study employed two commonly used oversampling methods, Smote and Adasyn, and the Tomek–Smote method improved by the Tomek Links. The distribution plots of prediction results and evaluation metrics, such as F1 scores, showed that the Smote and Adasyn oversampling algorithms could improve the overall prediction accuracy of the model but introduced interpolated samples with noisy information, leading to misclassification of certain samples. The Tomek–Smote sampling method effectively reduced the probability of misclassification while improving the overall accuracy of the model.

In summary, the SABO-LSSVM model demonstrated the advantages of a simple initial setup and good predictive performance in this study, providing assistance in predicting the stability state of pillars. However, the model also has issues, such as fast convergence and susceptibility to local optima, which may require the further introduction of new methods for improvement. Machine learning methods provide a new perspective and tool for pillar stability analysis, but they still rely on the experience and effective information summarized from methods currently used in the actual mining industry. When introducing these emerging methods, it is crucial to objectively and reasonably select datasets that can reflect the essence of the problem. With the development of machine learning technology, high-performance, simple, and efficient prediction models and techniques will emerge in the future. Improvement strategies may include, but are not limited to, refining the evaluation indicator system for pillar stability, constructing effective pillar sample datasets, and developing new data mining and application technologies.