Study on the Optimization and Stability of Machine Learning Runoff Prediction Models in the Karst Area

Abstract

Runoff prediction plays an extremely important role in flood prevention, mitigation, and the efficient use of water resources. Machine learning runoff prediction models have become popular due to their high computational efficiency. To select a model with a better runoff simulation and to validate the stability of the model, the following studies were done. Firstly, the support vector machine Model (SVM), the Elman Neural Network Model (ENN), and the multi-model mean model (MMM) were used for the runoff prediction, with the monthly runoff data from 1963–2007 recorded by the Pingtang hydrological station in the Chengbi River Karst Basin, China. Secondly, the comprehensive rating index method was applied to select the best model. Thirdly, the indicators of the hydrologic alteration–range of variability approach (IHA-RVA) was introduced to measure the model stability with different data structure inputs. According to the comprehensive rating index method, the SVM model outperformed the other models and was the best runoff prediction model with a score of 0.53. The overall change of the optimal model was 10.52%, which was in high stability.

1. Introduction

Accurate runoff prediction plays a critical role in the efficient and optimal allocation of water resources. Effective runoff prediction can be used for flood control, drought resistance, power generation, and irrigation [1,2,3,4]. In recent years, runoff prediction has received much attention. These studies focus on the exploration and application of neural network models, and the development and performance improvement of physical models. Gauche et al. [5] proposed two multi-timescale Long Short-Term Memory architectures that jointly predict multiple timescales within one mode, which can process different input variables at different timescales. An Adaptive Neural Fuzzy Inference System (ANFIS) model was developed to determine the discharge coefficient of the labyrinth spillway and thus predict runoff [6]. A Bayesian geostatistical method was used to interpolate hydrological data with different spatial support, and the model was evaluated by predicting the annual runoff in the watershed around Vos, Norway [7]. It was found that combining the point and surface data produced a higher prediction accuracy compared to using only one data type. In addition, to describe the nonhomogeneous characteristics of runoff series, a novel prediction model was established based on a nonhomogeneous Markov chain (NHMC-RPM) [8]. Furthermore, Excel solver was a promising way to reduce the problems of the parameter estimation of the nonlinear Muskingum routing models [9]. A reverse flood routing approach was developed based on the Muskingum model, and results indicated that the proposed methodology could substantially (up to almost 82%) improve the comparison with the observed inflows [10]. It has been shown that the Bayesian network-cluster model provides a suitable approach for predicting pollutant transport in natural rivers [11].

With the continuous development of machine learning, machine learning methods have been widely used in runoff predictions because of their excellent performance and relatively high applicability. For instance, the Artificial Neural network model and Long Short-Term Memory network model were applied to simulate the rainfall-runoff process according to flood events from 1971 to 2013 in the Fen River basin, monitored by 14 rainfall stations and one hydrologic station [12]. The results show that the two networks are both suitable for rainfall-runoff models and are better than conceptual and physical-based models. Liang et al. [13] proposed a support vector regression hydrologic uncertainty processor (SVR-HUP) model to predict the monthly, flood season, and annual runoff volumes. It was demonstrated that an SVR-HUP can predict the long-term runoff accurately and quantify the prediction uncertainty reasonably. A study of the daily runoff simulation was carried out based on the SVM-Copula model [14], and the results showed that the model can maintain the statistical characteristics of the original daily runoff series well, which means it could be a new way of random simulation in the case of insufficient data. In 2020, a Support Vector Machine–Artificial Flora (SVM-AF) hybrid model was used for the prediction of the daily streamflow [15]. The results showed that the proposed hybrid SVM-AF model outperformed the wavelet support vector machine model and the Bayesian support vector machine model in effectively predicting the flow and daily discharge. In the arid watershed, Samantaray and Sahoo [16] applied the Back Propagation Neural Network (BPNN), the Feed Forward Back Propagation Neural Network (FFBPNN), and the Cascade Forward Back Propagation Neural Network CFBPNN) algorithms for the prediction of the runoff. The results showed that the BPNN model is more suitable for predicting the connection between rainfall and runoff.

As different evaluation criteria were chosen, it is evident that the best model identified by each criterion is different. Due to the model-dependence, a scientific method is a key issue in runoff prediction. The SWAT, SWAT-ANN, and SWAT-MLP were applied to the runoff prediction. Analysis in Lv et al. [17] showed that the Soil and Water Assessment Tool–Whale Optimization Algorithm model was the best, as it was evaluated with four indicators (R2, RMSE, NSE, RE). According to Sibtain et al. [18], five statistical indexes were employed, which contain RMSE, MAE, MAPE, and MSE and R2 to evaluate the performance of eight runoff prediction models, and the results showed that complete ensemble empirical decomposition with the additive noise-variation mode decomposition–support vector machine (CEEMDAN-VMD-SVM) was the best model. Additionally, multiple linear regression (MLR), the back propagation neural network (BPNN), the Elman neural network (ENN), the particle swarm optimization–support vector machine for regression (PSO-SVR), and a coupling model composed of BPNN, ENN, and PSO-SVR were constructed to predict the annual runoff. The coupling model was considered to have the best predictive performance after the evaluation of the four metrics [19].

Although the previous study on runoff prediction has achieved certain results, there are still several problems, as follows. At present, there is no unified standard and method for the prediction model selection, so a more scientific and reasonable evaluation method for model selection is very necessary. Moreover, the stability of the model prediction performance has not been considered in previous studies, and how to test the stability of the model is still a major challenge for current research. Generally, most of the previous studies focused on runoff in non-karst areas, but the prediction models based on artificial intelligence and whether they can be applicable for runoff prediction analysis in karst areas needs to be further explored. Therefore, the objective of this study is to construct a suitable monthly runoff prediction model, establish a scientific evaluation system for selecting the model, and test the stability of the optimal model. For these purposes, the following studies will be conducted: firstly, the support vector machine model (SVM), the Elman Neural Network Model (ENN), and multi-model mean model (MMM) will be developed for the runoff prediction. Then, a scoring evaluation system with five indicators will be established to select the best model. Finally, the stability of the models will be tested by the IHA-RVA method with different data structure inputs.

2. Data and Methods

In order to better reflect the research idea of this paper, the specific flow chart is as follows in Figure 1.

2.1. Study Area and Data

The Chengbi River Karst Basin, located in Baise City, northwest of Guangxi, which covers an area of 2087 km² and contains 1121 km² of Karst area [20], was selected for this study. The basin has a subtropical monsoon climate, with a mild climate and abundant precipitation, which accounts for about 87% of the total precipitation during the flood season. The upper part of the basin is a typical karst peak forest area, with many karst features such as volute rivers, water caves, and skylights; the lower part is hilly, with high forest cover and dense vegetation [21]. There is a Chengbi River Reservoir in the basin, which is located 7 km upstream from the outlet of the basin. As the water source of drinking water of Baise city, Chengbi River Reservoir also has many roles such as power generation, flood control, and irrigation of farmland. Therefore, developing a suitable runoff prediction model with high accuracy is essential and significant for water resource allocation and sustainable economic growth in the Chengbi River Karst Basin area. There are 12 telemetric rainfall stations, 2 hydrological stations, and 1 meteorological station in the Chengbi River Karst Basin. Due to historical reasons, only the Pingtang and Bashou stations have collected representative long-duration data. The Pingtang station was selected as a representative station, and the monthly runoff data of the station for a total of 55 years from 1963 to 2017 were used for the study and analysis, which was provided by the water resources management department. Moreover, the runoff depth was obtained by calculating the total runoff volume of the specified section and dividing it by the rainfall catchment area above the section. The approximate location of the Chengbi River Karst Basin is shown in Figure 2.

2.2. Methods

2.2.1. Model for Prediction

Support Vector Machine

Suppose the training set of the SVM is ${\left\{X_i,Y_i\right\}}_i^N$, where $X_i$ is the input variable, $Y_i$ is the expected value, and $N$ is the number of training sets. The SVM regression function is determined according to statistical learning theory [22] as follows.

$$f(x)=w_i{\varphi }_i(x)+b$$

(1)

where ${\varphi }_i$ is the nonlinear transfer function that enables the mapping projection of the input variables from the low-dimensional space to the high-dimensional feature space, $w_i$ is the representative weight vector, and $b$ is the bias.

$$r(C)=C\frac{1}{N}{\displaystyle \sum _{i=1}^N{L}_{\epsilon }}\left(d_i,y_i\right)+\frac{1}{2}\Vert w{\Vert }^2$$

(2)

$$L_{\epsilon }(d,y)=\{\begin{array}{r}\hfill |d-y|-\epsilon ,|d-y|\ge \epsilon \\ \hfill 0,|d-y|<\epsilon \end{array}$$

(3)

In Equation (2), the first part of the right-hand side of the equation is the empirical risk, which can be found by Equation (4) below. The loss value of the insensitive loss function is 0 when within the allowed range [23]. The regularization constant $C$ can determine the criticality of the empirical risk in the optimization of real-world problems. The larger the value of $C$, the higher the corresponding percentage of empirical risk. The error tolerance $\epsilon$ denotes the approximate accuracy during training, $\xi$ and ${\xi }^{*}$ are positive relaxation variables, thus Equation (4) can be transferred to a constrained form as follows.

$$Minimize:\frac{1}{2}\Vert w{\Vert }^2+C\left({\displaystyle \sum _i^N\left({\xi }_i+{\xi }_i^{*}\right)}\right)$$

(4)

$$Subjectto\{\begin{array}{l}{\displaystyle \sum _{i=1}^N\left({\alpha }_i-{\alpha }_i^{*}\right)}=0\hfill \\ {\alpha }_i,{\alpha }_i^{*}\in [0,C],i=1,2,3,\cdots ,N\hfill \end{array}$$

(5)

To solve the optimization problem of Equation (4), the Lagrangian function is introduced as:

$$\begin{array}{l}L=\frac{1}{2}\Vert w{\Vert }^2+C\left({\displaystyle \sum _{i=1}^N\left({\xi }_i+{\xi }_i^{*}\right)}\right)-{\displaystyle \sum _{i=1}^N{\alpha }_i}\left(w_i\varphi \left(x_i\right)+b-d_i+\epsilon +{\xi }_i\right)\\ -{\displaystyle \sum _{i=1}^N{\alpha }_i^{*}}\left(d_i+w_i\varphi (x)-b+{\epsilon }_i^{*}\right)-{\displaystyle \sum _{i=1}^N\left({\beta }_i{\xi }_i+{\beta }_i^{*}{\xi }_i^{*}\right)}\end{array}$$

(6)

In Equation (6), the minimum values of $\mathrm{a}$, $b$, and $c$ need to be calculated, while the maximum values of $d$ and $f$ need to be calculated. By adding the Karush–Kuhn–Tucker condition to the regression algorithm the equation is transformed into a double Lagrangian form,

$$\begin{array}{l}v\left({\alpha }_i,{\alpha }_i^{*}\right)={\displaystyle \sum _{i=1}^N{d}_i}\left({\alpha }_i-{\alpha }_i^{*}\right)-\epsilon {\displaystyle \sum _{i=1}^N\left({\alpha }_i+{\alpha }_i^{*}\right)}\\ -\frac{1}{2}{\displaystyle \sum _{i=1,j=1}^N\left({\alpha }_i-{\alpha }_i^{*}\right)}\left({\alpha }_j-{\alpha }_j^{*}\right)K\left(x_i,x_j\right)\end{array}$$

(7)

$$\mathit{Subject}\mathit{to}\{\begin{array}{l}{\displaystyle \sum _{i=1}^N\left({\alpha }_i-{\alpha }_i^{*}\right)}=0\hfill \\ {\alpha }_i,{\alpha }_i^{*}\in [0,C],i=1,2,3,\cdots ,N\hfill \end{array}$$

(8)

where: ${\alpha }_i$ and ${\alpha }_i^{*}$ satisfy the equation. The calculation method of the optimal expected weight vector of the regression hyperplane is:

$$w^{*}={\displaystyle \sum _{i=1}^N\left({\alpha }_i-{\alpha }_i^{*}\right)}K\left(x,x_i\right)$$

(9)

$$f\left(x,\alpha ,{\alpha }^{*}\right)={\displaystyle \sum _{i=1}^N\left({\alpha }_i-{\alpha }_i^{*}\right)}K\left(x,x_i\right)+b$$

(10)

The kernel function in the above equation is expressed as

$$K\left(x_i,x_j\right)=\phi \left(x_i\right)\ast \phi \left(x_j\right)$$

(11)

The kernel function is the inner product of the high-dimensional feature space, and the Mercer condition is considered when choosing the kernel function, i.e., choosing a semi-positive definite function. The kernel functions have more types, and the four main types of kernel functions commonly used in SVM are as follows [24].

(1)

Linear kernel function:

$$K\left(x_i,x_j\right)=x_i^T{x}_j$$

(12)

(2)

Sigmoid kernel function:

$$K\left(x_i,x_j\right)=\tanh \left(v{x}_i^T{x}_j+c\right)$$

(13)

(3)

Gaussian kernel function:

$$K\left(x_i,x_j\right)=\exp \left(-\frac{{\left|x_i-x_j\right|}^2}{2{\sigma }^2}\right)$$

(14)

(4)

Polynomial kernel functions:

$$K\left(x_i,x_j\right)={\left(x_i^T{x}_j+c\right)}^d$$

(15)

The Gaussian kernel function performs best when compared to other kernels in terms of prediction error rate and rejection rate (i.e., correct prediction rate). Therefore, we chose the Gaussian kernel function as the kernel function of the SVM model in this study. The structure of the SVM model is schematically shown in Figure 3.

According to Izmailov et al. [25], when Gaussian kernel function is chosen as the kernel function, the parameter $g=1/2{\sigma }^2$ that comes with the function has an impact on the speed of training and prediction. Specifically, the smaller the value of $g$, the more the number of support vectors. The franker the training period, the lower the training accuracy, and the worse the model prediction; the larger the value of $g$, the fewer the number of support vectors, the easier the model is to over-train. The larger the value of $C$, the lower the tolerance to error and the easier it is to overfit, while the smaller the value of $C$, the higher the tolerance to error and the easier it is to underfit. Therefore, it is important to determine the appropriate penalty parameter $C$ and kernel function $g$. In this paper, the penalty factor and kernel function parameters in the support vector machine are optimally selected using a genetic algorithm, due to the advantages of simplicity, access, global parallelism, and good resistance to interference.

Elman Neural Network

The Elman neural network was established by Jeffrey and has four layers of network structure, that is the input layer, hidden layer, context layer, and output layer [26]. The function of the input layer is for passing the signal, the output layer is used for linear weighting, and the hidden layer is for passing the signal. In addition, Elman neural networks have nodes connected to the hidden layer and the context layer (or association layer, context layer, state layer) that receive feedback signals and can be considered as a one-step delay operator with a short-term memory function [27]. This property makes it sensitive to historical states. The network’s own ability to handle dynamic information is improved to accomplish time-varying adaptation, enhance global stability, and enable dynamic modeling. The hidden layer transfer function generally uses the nonlinear function, Sigmoid function, while the output layer and the correlation layer are linear functions. Therefore, compared with the forward neural network, Elman neural network is more suitable for predicting time series values. The structure of the Elman neural network is shown in Figure 4.

The calculation process of the model can be expressed as follows:

$$y_q^{(t)}=f\left({\displaystyle \sum _{j=1}^L{x}_j^{(t)}}{\omega }_{q,j}\right)$$

(16)

$$x_j^{(t)}=f\left({\displaystyle \sum _{i=1}^L{u}_i^{(t)}}{w}_{j,i}+{\displaystyle \sum _{k=1}^L{c}_k^{(t)}}{w}_{j,k}\right)$$

(17)

$$c_k^{(t)}=x_j^{(t-1)}$$

(18)

where $y_q^{(t)}$ is the output at time $t$ of the output layer, $x_j^{(t)}$ is the output at time $t$ of the hidden layer, $c_k^{(t)}$ is the output at time t of the context layer, $w_{q,j}$ is the connection weight function of the output layer, $w_{j,i}$ is the connection weight function of the hidden layer, $w_{j,k}$ is the connection weight function of the context layer.

Multi-Model Mean Model

The principle of MMM is that the results are obtained by averaging the results of multiple model predictions. The MMM models are widely used because of the ease of operation and the ability to balance data differences among models. The MMM model is a simple way to reduce biases in individual model outputs [28]. MMM models have a wide range of applications in climate modeling. In Sandor‘s study, Nash-Sutcliffe efficiency coefficients indicated that MMM outperformed individual models in 92.3% of cases [29].

2.2.2. Model Selection Method

Firstly, five metrics, Nash efficiency coefficient (NSE), mean absolute percentage error (MAPE), root mean squared error (RMSE), mean absolute error (MAE), and qualification rate (QR), were chosen to compare the performance of the models. The focus of each indicator is different, so the first five indicators can be used for the initial screening model. The NSE is capable of evaluating the predictive power of a model concerning measuring the degree to which the simulated process value is close to the observed value. The closer the value is to 1, the better the predictive power of the model. MAPE can evaluate the analysis for the accuracy of predicting the smooth component in the runoff series, ideally the value of which is 1. RMSE is applied to measure the degree of variability of the data, and the smaller the value, the better the accuracy of the model [30]. The range of MAE is [0, +∞), and zero indicates that the predicted value exactly matches the true value. QR is the ratio of the number of qualified predicted flow values to the total number of flow values. The formulas for calculating the above-mentioned indexes are as follows.

$$NSE=1-\frac{{\displaystyle \sum _{i=1}^N{\left(y_i-y_i^{*}\right)}^2}}{{\displaystyle \sum _{i=1}^N{\left(y_i-\overline{y}\right)}^2}}$$

(19)

$$MAPE=\frac{1}{n}{\displaystyle \sum _{i=1}^N\left|\frac{y_i-y_i^{*}}{y_i}\right|}\times 100\%$$

(20)

$$RMSE=\sqrt{\frac{1}{n}{\displaystyle \sum _{i=1}^N{\left(y_i-y_i^{*}\right)}^2}}$$

(21)

$$MAE=\frac{1}{n}{\displaystyle \sum _{i=1}^N\left|y_i-y^{\ast }\right|}$$

(22)

$$QR=\frac{k}{l}\times 100\%$$

(23)

where $y_i$ is the measured value, $y_i{}^{\ast }$ is the simulated value, $\overline{y}$ is the average of the observed value, and $n$ is the total number of time series.

Considering that individual models may exhibit better performance in one aspect, but cannot win across the board, a more integrated approach is needed. The composite rating index method (M_R) is based on the ranking of each rating index, which is a method to determine their consistency and obtain a composite ranking [31].

$$M_R=1-\frac{1}{nm}{\displaystyle \sum _{i=1}^n{r}_i}$$

(24)

where $m$ is the number of models, $n$ is the number of indicators to be evaluated, and $r_i$ is the ranking of the simulation capability of each model in each assessment metric (i = 1, 2, 3). The value $r_i=1$ indicates a model with the strongest simulation capability. It can be seen that 0 ≤ M_R < 1. The value $M_R$ is closer to 1, indicating the stronger comprehensive simulation capability of the model.

2.2.3. Stability Test Method

Hydrological series can be divided into two types according to their organization: the sequential structure and the monthly structure. Many articles have investigated the prediction of time-series data with the traditional sequential series. The runoff series can be reconstructed as twelve series of the same month, from January to December, and this new data structure, called monthly series [32], is relatively rare in research. In this paper, the monthly series is utilized to test the stability of the model. The two data structure forms are shown in Figure 5.

Next, IHA-RVA was introduced to measure the stability of the model to analyze the influence of different data structures on the stability of the model. Since there are 33 indicators in the original formula and only five here, including NSE, MAPE, RMSE, MAE, and QR, a slight change was made to the formula in this study. The expression of IHA-RVA is given by the following equation [33].

$$D_i=\left|\frac{Nm-N_{\epsilon }}{N_{\epsilon }}\right|\times 100\%$$

(25)

$$D_0=(\frac{1}{\mathrm{n}}{{\displaystyle \sum _{i=1}^5{D}_i}}^2{)}^{0.5}$$

(26)

where $D_i$ denotes the degree of change of each indicator, $D_0$ denotes the overall degree of change, $N_m$ denotes the value before the change of its indicator, and $N_{\epsilon }$ denotes the value of its indicator after the change. When $D_i$ and $D_0$ are between about 0% and 33%, it is the low degree of change, which also means that the model is highly stable. When $D_i$ and $D_0$ are between 33% and 67%, it is a moderate change, which means that the model is moderately stable. When $D_i$ and $D_0$ are between 67% and 100%, it is a high change, which means that the model is stable at a low level.

3. Results

3.1. Prediction Results

The continuous monthly runoff data were input into the model to gain the runoff prediction results from different models, as shown in Figure 6. To better analyze the eigenvalues of each model, the relevant data were calculated and plotted in

Table 1. Monthly runoff characteristic values.

Model	Average Value (mm)	Minimum Value (mm)	Maximum Value (mm)
Observed model	67.25	1.51	440.57
SVM model	61.36	−4.18	253.28
ENN model	70.15	−12.69	264.18
MMM model	65.75	−2.23	234.41

.

For the average value, the observed monthly average runoff depth is 67.25 mm. The monthly average runoff depth simulated by the MMM model is 65.75 mm, which is the closest to the observed value, proving that the MMM model shows a promising prediction in this regard. The monthly average runoff depth predicted by the SVM model and ENN model is 61.36 mm and 70.15 mm, respectively. It can be seen that the predicted values of the SVM model and ENN model are different from the actual observed values, and the prediction results are not as outstanding as the MMM model.

The minimum value of the measured runoff was 1.51 mm, while the minimum values of the runoff predicted by the model are negative. In terms of minimum prediction, the SVM model, ENN model, and MMM model need to be improved to ensure that the data are all positive.

To further analyze the maximum monthly runoff, the runoff data graphs were extracted and the details are shown in Figure 6. For the maximum monthly runoff, the observed maximum monthly runoff is 440.57 mm. As shown in Figure 7, the maximum runoff predicted by the SVM, MMM, and ENN models were 253.28 mm, 264.18 mm, and 234.41 mm, respectively, with absolute errors of 187.29 mm, 176.39 mm, and 206.16 mm, obtained at sequence numbers of 128, 103, and 127. The predicted value of the SVM model is closest to the observed value, indicating a better prediction. In terms of time, the maximum monthly runoff predicted by the SVM model and the observed maximum monthly runoff is the same month, again indicating that the SVM model predicts better. The maximum monthly runoff predicted by the ENN model and the maximum observed monthly runoff occurred 25 months apart. In this respect, ENN had the worst prediction.

Meanwhile, the maximum monthly runoff predicted by the MMM model differs from the observed maximum monthly runoff by only one month on the time scale.

To further analyze the prediction performance, the absolute error (AE) was calculated, and the results are shown in Figure 8, Figure 9 and Figure 10. A positive AE value means that the predicted value is higher than the observed value, which means that the predicted runoff value is larger than the actual one. A negative AE value means that the predicted value is lower than the observed value, which means that the predicted amount of water is smaller than the actual amount. For the SVM model, 64 of the AE values were positive, accounting for 48.48% of the total data, while the ENN model and the MMM model had more than half of the positive AEs, reaching 59.85% and 56.06%, respectively. A positive AE value indicates that the predicted runoff depth is on the high side. The larger the predicted flow depth is, the safer it is for engineering designs (such as reservoir projects and bridge projects). From the above engineering safety point of view, the ENN model embodies a good performance.

A box plot of absolute errors was drawn in order to further reveal comparisons among the SVM model, ENN model, and MMM model in Figure 11. Thus, a positive absolute error denotes an overestimation of an observed value. Correspondingly, a negative absolute error indicates an overestimation of an observed value. A box plot is also used to show the dispersion of a set of data [34]. It is mainly used to reflect the characteristics of the original data distribution, but also to compare the characteristics of the distribution of multiple groups of data. In terms of the median, the SVM model has the best prediction with a median absolute error of 0 mm, the ENN model has a median absolute error of 5 mm, and the MMM model has a median absolute error of 4 mm. In terms of the fluctuation range, the normal range of absolute error for the SVM model is −72 mm to 65 mm, the normal range of absolute error for the ENN model is −72 mm to 104 mm, and the normal range of absolute error for the MMM model is −74 mm to 83 mm. The SVM model has the least fluctuation in absolute error.

The prediction accuracy of the SVM model, ENN model, and MMM model was further evaluated using the Taylor Diagram in Figure 12. The Taylor diagram was plotted based on the geometric relationship among the correlation coefficient (equal to the root of R²), standard deviation, and RMSE [34]. The Taylor diagram accurately and efficiently reflects the performance of the prediction model, with a closer proximity to observations indicating better predictions [35]. All three metrics indicate that the SVM model predictions were close to the observed values. Overall, the SVM model was closest to the observed values in comparison with the ENN model and MMM model.

Through the above analysis of the prediction performance of each model, it is concluded that each model has its own advantages and disadvantages, and it is impossible to directly determine which model is better, so further research is needed to make a better choice.

3.2. Model Selection Results

Firstly, the model evaluation is performed using different metrics, as shown in

Table 2. Evaluation of model performance results through five metrics.

Model	NSE (mm)	MAPE	RMSE	MAE (mm)	QR
SVM model	0.607	102.18%	52.89	34.31	0.54
Elman model	0.572	160.07%	55.21	38.54	0.42
MMM model	0.615	123.95%	52.34	35.34	0.48

. For NSE, the MMM model outperforms the other two models, with an extremely narrow advantage over the SVM, with a difference of only 0.01. For MAPE, the SVM model outperforms the other two models, with a MAPE of 102.18%, which is significantly better than those of the ENN model (160%) and the MMM model (120%). For RMSE, the MMM model has the best prediction effect, which is 2.87 mm lower than ENN’s RMSE and slightly superior to SVM’s RMSE by 0.55 mm. Regarding the MAE values, the SVM model has the best prediction ability, and the overall error for all data is significantly smaller than the prediction errors of ENN and MMM. In terms of QR, the predictive power of SVM is much higher than ENN and MMM, being 0.12 and 0.06 higher than both, respectively. In conclusion, the simulation effect of ENN is the worst by all aspects of evaluation, while further calculations and discussion on the simulation effect of SVM and MMM are needed to find out which is better.

Secondly, the comprehensive rating index method was applied to rank the evaluation indexes of different models, and the calculation results of $r_i$ are shown in

Table 3. The ranking of the three models under five indicators ($r_i$).

Model	NSE	MAPE	RMSE	MAE	QR
SVM model	2	1	2	1	1
Elman model	3	3	3	3	3
MMM model	1	2	1	2	2

. Based on the above scoring rules, the final scoring results of the three models were calculated and shown in

Table 4. Results of the combined assessment of the three models.

Method	SVM Model	ENN Model	MMM Model
MR	0.53	0.00	0.47

. The results show that the SVM model is the best runoff simulation model with a score of 0.53, the MMM model is the next best with a score of 0.47, and the ENN model has the worst simulation with a score of only 0. Therefore, combining these results, the SVM model has the highest evaluation score among the three models and is the model with the best simulation effect.

3.3. Optimal Model Stability

First, we changed the data structure to build a new model. The results of the model preference in the previous step show that SVM is the best model among the three runoff simulation models. By changing the data structure, the new SVM model was employed with twelve bunches of streamflow data after naming it SVM (M). The new model prediction results are shown in Figure 13.

The stability of the model was measured by measuring the degree of variation in five indicators. The degrees of model variability derived by the IHA-RVA method are shown in

Table 5. Model stability analysis results.

Indicator	Nm	Nε	Di	D0
NSE	0.61	0.53	15.11%	10.52%
MAPE	102.18%	90.66%	12.71%
RMSE	52.89	58.00	8.81%
MAE	34.31	37.44	8.36%
QR	0.54	0.56	4.05%

. From the table, we can find that five indicators have changed simultaneously. Among them, two indicators show that the model constructed with the new data structure has a better prediction, namely MAPE and QR, where MAPE is reduced by 11.52% from the original 102.18% and QR is changed from 0.54 to 0.56. The other three indicators show that the original data structure can perform a better runoff prediction, and the initial SVM model has a better NSE value than the new one. The NSE value of the initial SVM model is 0.08 higher than that of the new data, while both RMSE and MAE are lower than that of the original model. Overall, the original model structure facilitates a better runoff prediction. In addition, the variability D_i of all five indicators is less than 33%, and the overall variability D₀ of the model is only 10.52%, indicating that the model is of low variability (a highly stable state).

4. Discussion

4.1. Similarities and Differences

The SVM model has a better performance than the ENN model, which is consistent with the results of the previous study. According to the model evaluation results, the performance of the SVM simulation is higher than that of Elman. In a study for the time series prediction, it appeared that prediction results were equally well for the SVM model, whereas the Elman neural network could not be used to predict these series as well [35]. In a study in other areas, such as cyber security posture, the SVM model also showed more advanced predictions than the ENN model. In terms of error, the RMSE and MAE values of SVM were lower than ENN [36]. In these studies mentioned above, SVM showed better predictions than ENN, whether it was a time series prediction or a network safety period prediction.

The runoff results predicted by the MMM model did not show an optimal prediction, which is inconsistent with the findings of most scholars, who generally agree that the MMM model has a significant advantage in balancing the internal variability of the data. As stated by Mishra et al. [37], MMM is a useful way of extracting first-order climate change projection information from the CMIP5 model. However, according to the scoring results of this study, the MMM model outperforms the ENN model in all metrics, while the overall scoring results are inferior to the SVM model, indicating that the prediction results of the MMM are not always better than those of a single model. Analysis of the results shows that the data of the MMM model are derived from both the SVM model and the MMM model, and although the MMM model can balance the data differences between the models, it does not completely reduce the runoff errors. As shown in Figure 6, the prediction error patterns of the SVM model and the ENN model are similar in the same period (both are overestimated or underestimated), thus the performance of the MMM model is in between the two models. In this study, it is reasonable for the prediction results of the MMM model to be between the best SVM model and the worst ENN model.

The monthly data structure-based model did not outperform the sequential data structure-based model, which is different from other scholars’ studies. According to Zhao et al. [32], the monthly predictive structure can effectively extract the instinctive hydrological information that is more easily learned by the predictive model than the traditional sequential predictive structure. The analysis may be due to differences in model types leading to different results. In Zhao’s study, the IGWO-GRU model was applied to the runoff prediction, while the SVM model is applied in this paper. In addition, the runoff data in this study are based on karst basins, and the subsurface geological conditions are extremely different from those in non-karst areas. Karst has a continuous influence on runoff. Therefore, under the sequential structure, this may make the SVM model more effective in capturing the influence pattern of karst. Based on these two points, it is also reasonable that the sequential data structure-based model is superior to the monthly data structure-based model in runoff prediction.

4.2. Policy Recommendations

The above studies show that the data structure can have a degree of impact on model performance. In this study, the traditional sequential structure model is superior to the monthly structure model. This result also broadens the ideas for runoff prediction. Water resource managers can consider choosing different data structures for runoff prediction and select the best model to manage water resources more scientifically and effectively.

4.3. Innovation, Limitations, and Further Research

A method for accurately quantifying and qualitatively measuring the stability of the model was developed. The stability of the model was measured for the first time by changing the data structure and introducing the quantitative calculations of IHA-RVA, which has also been used in previous studies, but has never been quantified for the stability of runoff prediction models and has only been applied to measure the degree of variability in hydrological conditions [33]. In this study, the overall variability of the model indicators is 10.52%, which is a low variability, meaning that the model is highly stable.

All three models showed negative values when predicting low values, and further model improvements are needed in this area to ensure that all data are positive.

Due to the data, the results and conclusions of this study cannot be directly transferred to other watersheds, and more watersheds are needed to obtain more generalized results. Furthermore, in future studies, replication and validation in more watersheds could be attempted.

The stability analysis of the model is the key to the preferred model. However, the factors affecting the stability of the model are complex and diverse. This study attempts to analyze the stability of the model only from the perspective of data structure, and the stability of the model performance needs to be carried out in depth. Due to the influence of climate change and human activities, the change of runoff becomes more complicated. However, the prediction of runoff by machine learning models cannot fully consider the influence of climate change and human activities. Therefore, a runoff prediction model that can fully combine the factors of climate and human activities needs to be further proposed.

5. Conclusions

In this study, the Chengbi River Karst Basin in Guangxi was used as the research object. The main findings of this study are as follows: The three models for runoff prediction in the karst areas are applicable, and there is room for improvement in prediction accuracy. Among the three models (SVM model, ENN model, and MMM model), the SVM model had the best runoff prediction, the MMM had the second-best runoff prediction, and the ENN model has the worst prediction. It is worth noting that although the data structure has an effect on the model, the degree of influence is limited, indicating that the SVM model has good reliability. By changing the data structure, it was found that the overall metrics of the SVM model changed by 10.52% and were highly stable. At the same time, different data structures can be considered for runoff prediction and simulation, to achieve better prediction results.

Based on these findings, further research direction is needed into, for example, exploring more machine learning models for runoff prediction in karst areas. Furthermore, in addition to the multi-model mean model, the performance of other coupled models can be further studied, and the relationship between the data structure and model performance is worth exploring in depth.