Monthly Runoff Forecasting Using Particle Swarm Optimization Coupled with Flower Pollination Algorithm-Based Deep Belief Networks: A Case Study in the Yalong River Basin

Abstract

Accuracy in monthly runoff forecasting is of great significance in the full utilization of flood and drought control and of water resources. Data-driven models have been proposed to improve monthly runoff forecasting in recent years. To effectively promote the prediction effect of monthly runoff, a novel hybrid data-driven model using particle swarm optimization coupled with flower pollination algorithm-based deep belief networks (PSO-FPA-DBNs) was proposed, which selected the optimal network depth via PSO and searched for the optimum hyper parameters (the number of neurons in the hidden layer and the learning rate of the RBMs) in the DBN using FPA. The methodology was divided into three steps: (i) the Comprehensive Basin Response (COM) was constructed and calculated to characterize the hydrological state of the basin, (ii) the information entropy algorithm was adopted to select the key factors, and (iii) the novel model was proposed for monthly runoff forecasting. We systematically compared the PSO-FPA-DBN model with the traditional prediction models (i.e., the backpropagation neural network (BPNN), support vector machines (SVM), deep belief networks (DBN)), and other improved models (DBN-PLSR, PSO-GA-DBN, and PSO-ACO-DBN) for monthly runoff forecasting by using an original dataset. Experimental results demonstrated that our PSO-FPA-DBN model outperformed the peer models, with a mean absolute percentage error (MAPE) of 18.23%, root mean squared error (RMSE) of 230.45 m³/s, coefficient of determination (DC) of 0.9389, and qualified rate (QR) of 64.2% for the data from the Yalong River Basin. Also, the stability of our PSO-FPA-DBN model was evaluated. The proposed model might adapt effectively to the nonlinear characteristics of monthly runoff forecasting; therefore, it could obtain accurate and reliable runoff forecasting results.

1. Introduction

Accuracy in monthly runoff forecasting is of great significance in the full utilization of flood and drought control and water resources [1,2,3]. At present, monthly runoff forecasting is in the stage of exploration and development. Moreover, the prediction accuracy cannot satisfy the actual needs of various production departments. Therefore, it is of high scientific significance and practical value to develop a precise and robust runoff prediction model to significantly promote the prediction effect of monthly runoff. However, runoff series are susceptible to many uncertainties and exhibit highly nonlinear characteristics. This hinders hydrology departments from predicting runoff [4,5,6].

The identification of key factors plays an important role in monthly runoff forecasting. However, it is difficult to determine key influencing factors closely related to runoff variation because of the highly nonlinear characteristics of the process of runoff changes, which seriously affects the accuracy and reliability of runoff forecasting [7]. In recent years, the antecedent runoff, rainfall, vegetation index, and other meteorological parameters, such as air pressure, wind speed, etc., have been found to be closely related to the runoff forecasting [8]. In addition, many teleconnection climate factors influencing runoff variation [9,10] have been considered as alternative candidate predictive factors, including the East Asian Trough Intensity Index, ENSO Modoki Index, and other indexes. The approaches used in key factor selection are principally the prior knowledge method [11], correlation coefficient method [12,13], principal component analysis (PCA) [14,15], mutual information (MI) [16], and partial mutual information (PMI) [17]. Among the various factor-selection methods, the prior knowledge method relies mainly on artificial experience, which is subjective and has certain limitations. The correlation coefficient method and PCA are generally linear methods that are difficult to adapt to the complex nonlinear characteristics of the monthly runoff process and have a certain scope of application. Information entropy methods, particularly the MI method, omit the distribution of variables and are suitable for linear and nonlinear correlations between alternative factors. The PMI method is an improvement over the MI method. It can effectively prevent the influence of selected factors and reduce redundant variables and computational complexity in monthly runoff prediction.

The selection of the prediction model is critical for monthly runoff forecasting [18,19,20,21]. Currently, the commonly used prediction methods are divided into two categories: process- and data-driven model methods. Process-driven models refer to the improvement in the structure of the conceptual rainfall–runoff model based on the hydrological theory, to predict runoff, such as the Tank [22] and Xin’anjiang model [23]. The process-driven models are widely used and have many achievements. However, many problems also need to be addressed. For example, the physical causes affecting long-term variations in the runoff sequence are complex, which hinders the determination of model parameters. Moreover, runoff variation processes with complex nonlinear characteristics limit the application of process-driven models. On the contrary, data-driven models omit the physical mechanisms of hydrological processes, demand less data, and can offer satisfactory forecast results. These alternatively utilize the relationship between the input and output patterns and have been widely used in monthly runoff forecasting. These include the backpropagation neural network (BPNN) [1], support vector machine (SVM) [24], extreme learning machine (ELM) [25,26], multilayer perceptron model for stochastic synthesis (MLPS) [27], and multiple combination models [28,29]. However, these shallow data-driven models have many problems (such as a relatively complex model structure and parameters that need to be initialized, optimized constantly, and adjusted in the training process) with relatively low efficiency. At present, deep-learning-based prediction models have attracted an increasing number of hydrologists [30,31]. In contrast to shallow networks, deep neural networks include multiple layers with nonlinear operational units, and the output of the upper layer serves as the input for the next layer. Through layer-by-layer transmission, the feature data in the higher layer can be distinguished from the original data in the lower layer to acquire better object representation. The advantages of deep learning can compensate for the shortcomings of existing monthly runoff forecasting methods and improve the forecasting speed, accuracy, and generalization capability [32,33]. For example, Ren et al. [34] adopted the RNN, LSTM, and GRU models for mid- to long-term runoff prediction and obtained good effects. Wang et al. [35] proposed the SMD-SE-WPD-LSTM hybrid forecasting model for annual runoff and achieved higher accuracy and consistency.

Despite these data-driven models performing well in runoff prediction, it should be noted that the prediction effect of data-driven models largely relies on the hyperparameters and model parameters [36]. Nevertheless, these parameters must be tuned in the training stage for better application of data-driven models. In general, the parameter optimization algorithms are principally the grid search and gradient decent-based algorithms. However, due to the complexity of the hyperparameters, the acquired trained models occasionally exhibit poor performance [37]. In addition, these parameter-optimization algorithms can introduce too much computation and lead to entrapment in the local optimum. In contrast, bio-inspired algorithms are more conducive to solving global optimization problems. As a result, integrating data-driven models with bio-inspired optimization algorithms may raise the computational speed, accuracy, and robustness of the models and thereby have been widely applied in hydrology, such as particle swarm optimization [38] (PSO), a genetic algorithm (GA) [39], an artificial bee colony (ABC) [40], a flower pollination algorithm (FPA) [41], and a firefly algorithm (FFA) [42]. For instance, Yaseen et al. [43] proposed a hybrid ANFIS-FFA model for monthly streamflow forecasting, and the proposed model performed better than the original ANFIS model. Yue et al. [3] developed a novel IPSO-ELM for mid-long-term runoff and found the hybrid IPSO-ELM model was superior to the BPNN, SVM, ELM, PSO-ELM, and other bio-inspired algorithm-based models. Although bio-inspired algorithms have been recently used to search for the optimal parameters of the data-driven models for monthly runoff forecasting, they mainly depend on traditional bio-inspired algorithms (e.g., GA and PSO) rather than current state-of-the-art algorithms, such as FPA. Therefore, this study intended to evaluate the runoff prediction accuracy based on the deep belief networks (DBNs) model optimized using PSO coupled with FPA: (i) selecting the optimal network depth of the DBN via PSO; (ii) searching for the optimum hyperparameters of the DBN using FPA.

In summary, data-driven models, especially the Deep Belief Networks, have rarely been integrated with PSO and FPA methods for monthly runoff forecasting. Therefore, to further promote the prediction effect of monthly runoff, a novel hybrid PSO-FPA-DBN model was developed in this paper. The objectives of this study were as follows: first, a Comprehensive Basin Response (COM) was constructed and calculated to characterize the hydrological state of the basin. Second, the information entropy algorithm was adopted to select key factors by calculating the correlation between the candidate factors and the COM for the inputs of the prediction model. Finally, a novel hybrid PSO-FPA-DBN model was proposed for monthly runoff forecasting using the deep belief networks (DBNs) optimized using PSO and FPA. With these effective components, our model achieved good runoff forecasting results. It outperformed state-of-the-art data-driven models. The main contributions of this study were as follows:

(1)

The COM was constructed to characterize the hydrological state of the entire basin.

(2)

Partial mutual information was applied to select the key factors and reduce redundant variables and the computational complexity.

(3)

The PSO-FPA-DBN model was proposed for monthly runoff forecasting. Highly accurate and reliable results were obtained.

2. Methodology

The methodology is divided into five parts: Comprehensive Basin Response, factor reduction using information entropy, PSO, FPA, and DBN. These aspects are discussed in detail in the following sections. The flowchart of the proposed methodology is presented in Figure 1.

2.1. Comprehensive Basin Response

The important factors affecting runoff over watersheds include rainfall, climate, vegetation conditions, and human activities. In this study, the lagged predictive factors of runoff, rainfall, and climate from the current time-step t until the previous 12 time-steps (one month) were considered in our method. Moreover, the corresponding data were collected at hydrological stations with a monthly temporal scale. To describe the weighted response and then characterize the hydrological state of the basin, we constructed the Comprehensive Basin Response. The following are its details.

As mentioned above, the weight of the ith hydrological station is defined as follows:

$$W_i=\frac{1/S_i^{}}{{\displaystyle \sum _{i=1}^{n}1/S_i^{}}}$$

(1)

where n is the number of hydrological stations and S_i is the controlling area of the ith hydrological station.

Consequently, the Comprehensive Basin Response for the jth month is obtained by

$$C_j={\displaystyle \sum _{i=1}^n{W}_i{C}_{ij}}$$

(2)

where C_ij denotes the monthly average runoff at the ith hydrological station in the jth month.

2.2. Factor Reduction Using Information Entropy

In recent years, factor selection methods based on information entropy, particularly MI methods, have been widely applied in hydrology. MI can calculate the linear and nonlinear correlations between predictive objects and measure the amount of information contained in a variable [44,45]. Thus, a few researchers have selected MI to reduce the candidate predictive factors influencing runoff variation. However, this strategy is controlled by the correlation between the input variables, which causes redundancy in the factors. To solve this problem, R.J. May [46,47] proposed the PMI method to remove the correlation between variables. It can raise the speed and accuracy of variable selection.

Assuming that the number of discrete samples is N, the PMI is defined in the following discrete form:

$$PMI=\frac{1}{N}{\displaystyle \sum _{i=1}^N\ln \frac{f_{X^{\prime },Y^{\prime }}(x_i{}^{\prime },y_i{}^{\prime })}{f_{X^{\prime }}(x_i{}^{\prime })f_{Y^{\prime }}(y_i{}^{\prime })}}$$

(3)

To present Equation (3) in a simple form, the PMI is calculated using the kernel density. It is a nonparametric probability density-estimation method with high accuracy. The sample probability density function is defined as follows:

$${\hat{f}}_X(x_i)=\frac{1}{N}{\displaystyle \sum _{j=1}^N\frac{1}{{(2\pi )}^{d/2}{\lambda }^d\det {(S)}^{1/2}}}\exp (-\frac{{(x_i-x_j)}^T{S}^{-1}(x_i-x_j)}{2{\lambda }^2})$$

(4)

where ${\hat{f}}_X(x_i)$ represents the estimated density function of the variable at x_i, d represents the dimension, S represents the covariance matrix, and $\lambda$ represents the window width [48]:

$$\lambda ={(\frac{1}{d+2})}^{1/(d+4)}{N}^{(-1/(d+4))}$$

(5)

2.3. Particle Swarm Optimization

Among evolutionary computation methods, PSO is widespread [49]. Assuming that N represents the dimension of the searching domain, the ith swarm particle at the time-step k is denoted with an N-dimensional vector $x_i^k={\left\{x_{i1}^k,x_{i2}^k,\dots ,x_{iN}^k\right\}}^T$. The particle velocity at the time-step k is denoted with the vector $v_i^k={\left\{v_{i1}^k,v_{i2}^k,\dots ,v_{iN}^k\right\}}^T$. The previously best-visited position of the ith particle at the time-step k is denoted with $p_i^k={\left\{p_{i1}^k,p_{i2}^k,\dots ,p_{iN}^k\right\}}^T$. g denotes the exponent of the optimum particle during the evolution process. The new velocity and position of the ith particle are calculated by Equations (6) and (7), respectively:

$$v_{id}^{(k+1)}=v_{id}^k+c_1{r}_1(p_{id}^k-x_{id}^k)+c_2{r}_2(p_{gd}^k-x_{id}^k)$$

(6)

$$x_{id}^{(k+1)}=x_{id}^k+v_{id}^{(k+1)}$$

(7)

where the vector dimension of the particle is denoted by d = 1, 2, …, N. The particle exponent is denoted by i = 1, 2, …, N_s. N_s is the number of the swarm. c₁ and c₂ are the acceleration factors. r₁ and r₂ are uniformly distributed random numbers that vary within the range [0, 1].

2.4. Flower Pollination Algorithm

This paper introduces the FPA. It is inspired by the pollination characteristics of flowering plants. For the FPA, the global search results are obtained by

$$x_i^{t+1}=x_i^t+\gamma L(\lambda )(g_{best}-x_i^t)$$

(8)

where $\gamma$ denotes the scaling parameter. $L(\lambda )$ denotes the random number vector of a Lévy distribution with the scaling parameter $\lambda$. $g_{best}$ is the optimum result provided by the selection mechanism. The $x_i^t$ at the current time step is modified by varying the step sizes. The local search result is obtained by

$$x_i^{t+1}=x_i^t+\epsilon (x_j^t-x_k^t)$$

(9)

Here, $\epsilon$ is the uniformly distributed random number. $x_j^t$ and $x_k^t$ are the solution representations given by flower patches.

2.5. Deep Belief Networks

In recent years, deep learning has yielded performance improvements in several studies. For example, DBN, convolutional neural networks (CNNs), and stacked autoencoders (SSAEs) have achieved significant success. In 2006, Hinton [50] proposed the DBN model, including many restricted Boltzmann machines (RBMs), BP neural networks, and softmax classifiers. A CNN [51] has a multilayer network structure that analyzes the spatial correlation of data and reduces the number of training parameters. These exhibit remarkable robustness and are widely used in face recognition, speech recognition, and many other fields. The SSAE is composed of many sparse auto-encoders and softmax classifiers [52]. These methods pre-train the network layer-by-layer to initialize the parameters of the SSAE in the unsupervised learning stage. Supervised learning is used after the pretraining.

2.5.1. Restricted Boltzmann Machine (RBM)

The RBM is an energy-based model that combines all the input parameters with energy functions to obtain the dependence between the input parameters. The probability of various combinations of parameters is inversely proportional to the energy. The corresponding energy value is minimized through continuous training and learning, as shown in Figure 2.

Assuming that the variable is $v$, the probability distribution can be expressed as

$$p(v)=\frac{e^{-E(v)}}{z}$$

(10)

In Equation (10), $z$ is the normalization factor. It can be expressed as

$$z={\displaystyle \sum _v{e}^{-E(v)}}$$

(11)

The RBM contains an input layer and a hidden layer. Herein, the number of neurons in the input layer is equal to the vector dimension of the training sample features. The dependence between each dimension of the input sample features is extracted by the hidden layer. The two layers are connected by undirected edges. Assuming that the weight is $w_{ij}$, the bias values of the input and hidden layers are $b_j$ and $c_i$, respectively. The internal neurons of the input or hidden layer are not connected by edges such that the internal neurons of each layer are independent of each other.

For a convenient calculation, the values of the hidden and input layers are limited to binary zero or one. After inputting the samples, a probability of zero or one can be determined for each hidden layer, and the expectation of the hidden layer is the feature output. The energy function of the RBM is defined as follows:

$$E(v,h)=-bv-ch-hWv$$

(12)

In Equation (12), $W$ represents the connection weight between the input and hidden layers of the RBM model; $b$ and $c$ represent the biases of the input and hidden layers; and $v$ and $h$ represent the state of elements in the input and hidden layers, respectively. According to the energy function, the joint probability distribution $(v,h)$ can be defined as follows:

$$p(v,h)=\frac{e^{-E(v,h)}}{z}$$

(13)

In Equation (13), $z={\displaystyle \sum _{v,h}{e}^{-E(v)}}$.

Because the elements within each layer of RBM are not connected,

$$p\left(h\left|v\right.\right)={\displaystyle \prod _i^{n}p\left(h_i\left|v\right.\right)}$$

(14)

$$p\left(v\left|h\right.\right)={\displaystyle \prod _j^{m}p\left(v_j\left|h\right.\right)}$$

(15)

When the state of the input elements is given, the activation probability of a point in the jth hidden layer is

$$p\left(h_i=1\left|v\right.\right)=\sigma \left(c_i+W_{i}v\right)$$

(16)

In Equation (16), $\sigma$ is the activation function, $\sigma (x)=\frac{1}{1+e^{-x}}$.

The activation probability of a unit in the input layer can be defined as

$$p\left(v_j=1\left|h\right.\right)=\sigma \left(b_j+W_j^{\prime }h\right)$$

(17)

The training process of the RBM is generally realized using a gradient descent algorithm or Gibbs sampling. However, these methods are inefficient when addressing high-dimensional data. The contrastive divergence algorithm of the RBM has the following advantages:

(1)

Training sample initialization, denoted as $v_0$.

(2)

Calculating the state values $h$ of the elements in all the hidden layers employing Formula (16).

(3)

Calculating the state values $v$ of the elements in the input layer and reconstructing these.

(4)

The weights are updated according to the error of the real and reconstructed values.

Thus, the updating algorithms of the model parameters $b_j$, $c_i$, and $W_{ij}$ are as follows:

$$\mathsf{\Delta }{W}_{ij}=\epsilon ({〈v_j{h}_i〉}_{data}-〈{v_j{h}_i〉}_{recon})$$

(18)

$$\mathsf{\Delta }{b}_j=\epsilon ({〈v_j〉}_{data}-〈{v_j〉}_{recon})$$

(19)

$$\mathsf{\Delta }{c}_i=\epsilon ({〈h_i〉}_{data}-〈{h_i〉}_{recon})$$

(20)

where $\epsilon$ is the learning rate, ${〈.〉}_{data}$ is the input sample for RBM, and $〈{.〉}_{recon}$ represents the reconstructed data.

2.5.2. The Architecture of the DBN

The typical architecture of the DBN is shown in Figure 3.

From Figure 3, the input layer formed the first RBM with the first hidden layer of the DBN. The data features extracted by the first RBM are inputted into the second RBM through the hidden layer of the first RBM. At this time, the hidden layer of the first RBM functions as the input layer of the second RBM. Meanwhile, the hidden layer of the second RBM is the second hidden layer of the DBN, which constitutes the DBN model with multiple RBMs.

Conventional supervised learning has an excessive number of hidden layers. This results in problems, such as long training times and low convergence speeds. However, this approach is not suitable for DBN. Hinton proposed a level-by-level unsupervised greedy training algorithm that has been widely used because of its unsupervised learning, fast parameter determination, and accurate feature extraction. Therefore, this algorithm was adopted in this study to train the DBN model. The algorithm includes two stages: (i) RBM pretraining (feature extraction from training samples and initialization parameters of the model) and (ii) fine-tuning (the BP algorithm is used for fine-tuning the model parameters to further optimize the DBN). The training process is shown in Figure 4. The specific training steps are as follows:

(1)

Based on the contrastive divergence algorithm, the input samples (original data) are trained in the first RBM.

(2)

The output of the hidden layers in the first RBM is regarded as the input of the second RBM, and the second RBM is trained with a contrastive divergence algorithm.

(3)

Training is continued according to the above method until all the RBMs are trained.

(4)

After training all the RBMs to obtain the appropriate model parameters through the above steps, supervised training with the BP algorithm is used for fine-tuning all the DBN parameters in the output layer at the end of the DBN. At this point, the BP algorithm should only search for the weights of the network in a local space. Thus, compared with the common BP algorithm, the training speed is improved significantly, the parameters converge more straightforwardly, and it is not straightforward to fall into the dilemma of a local extremum.

To overcome the shortage of local extrema caused by the random generation of initial weights of the model, the DBN is divided into multiple single RBMs for training layer-by-layer with a contrastive divergence algorithm. It optimizes the initial weights of the DBN and reduces the complexity. The above structure and training methods enable the DBN to outperform conventional shallow learning network models, such as BPNN and SVM.

2.6. Normalization of the Original Experimental Data

To improve the accuracy and reliability and avoid overfitting for monthly runoff forecasting, the min-max normalization method [53,54] was selected for standardization and normalization before the machine learning model development.

2.7. Evaluation Criteria

Five evaluation indicators, namely the mean absolute percentage error (MAPE), root mean squared error (RMSE), coefficient of determination (DC) [55], relative error (RE), and qualified rate (QR), were employed to verify the monthly runoff prediction effect of the data-driven models. Generally, the smaller MAPE, RMSE, and RE are and the higher the DC and QR are, the better the model performance [19].

The calculation formulas for each evaluation index are as follows:

$$MAPE=\frac{1}{n}{\displaystyle {\sum }_{t=1}^n\left|\frac{(y_a^t-y_p^t)}{y_a^t}\right|}$$

(21)

$$RMSE=\sqrt{\frac{1}{n}{\displaystyle {\sum }_{t=1}^n{(y_a^t-y_p^t)}^2}}$$

(22)

In Equations (21) and (22), $y_a^t$ denotes the actual value at the time-step $t$, and $y_p^t$ denotes the predicted value at the time-step $t$. $n$ is the number of sampled data. RMSE was used to evaluate the degree of variation in the data.

$$QR=\frac{m}{n_{QR}}\times 100\%$$

(23)

In Formula (23), m represents the number of qualified predictions, and $n_{QR}$ represents the total prediction time.

$$DC=1-\frac{{\displaystyle {\sum }_{t=1}^n{(y_a^t-y_p^t)}^2}}{{\displaystyle {\sum }_{t=1}^n{(y_a^t-\overline{y_a^t})}^2}}$$

(24)

$$RE=\left|\frac{(y_a^t-y_p^t)}{y_a^t}\right|\times 100\%$$

(25)

In Formula (24), $\overline{y_a^t}$ represents the average of the actual values, and the range of the DC variation is $[0,1]$. The closer DC is to one, the higher is the precision of the model. The RE was used to evaluate the reliability of the prediction model.

3. The Proposed PSO-FPA-DBN Model for Monthly Runoff Forecasting

Currently, hydrologists have performed research on runoff forecasting. The methods can be classified into four: cause analysis, statistical methods, intelligent-computing-based forecasting, and numerical-weather-forecast-based forecasting.

Runoff production is influenced by complex factors. The main influencing factors differ across regions, thereby exhibiting complex non-linear characteristics. To obtain highly accurate and reliable runoff prediction results, an improved DBN model (PSO-FPA-DBN) was developed for runoff prediction using a DBN optimized by PSO and the FPA. The network architecture, learning algorithms, and parameter optimization methods of the model were analyzed in detail.

3.1. Network Architecture

In this study, the hybrid PSO-FPA-DBN model was proposed for monthly runoff forecasting (Figure 5). The prediction model is divided into (L + 2) layers. The training of the prediction model included the following two stages:

(1)

Pre-training stage. Unsupervised learning was used to extract samples step-by-step. The specific steps included inputting and obtaining the parameters of RBM1 using a contrastive divergence algorithm. We then obtained the output of RBM1 by training it for the initial extraction of the feature vectors of the impact factors influencing monthly runoff. Similarly, after completing the training process of RBM2, RBM (L − 1), and RBML, the training process of the entire model for monthly runoff forecasting was completed.

(2)

Parameter fine-tuning stage. The parameters of the model were fine-tuned using supervised learning with the BP algorithm so that the model had a better fitting effect. When the prediction error was less than a given threshold, the training process was considered complete.

3.2. Learning Algorithms

The hybrid PSO-FPA-DBN model was divided into the following steps.

Step 1: Network initialization. Each layer of neurons in the RBM was randomly initialized as zero or one. The connection weight $W_{ij}$ between the different layers was set to a value in the range [0, 1].

Step 2: Steps 2.1 and 2.4 were repeated until the energy function $E(v,h)$ yielded the convergence state. The threshold value $\alpha$ was set to determine whether the energy function $E(v,h)$ converged.

$$E(v,h)=-{\displaystyle \sum _{i=1}^n{b}_i{v}_i}-{\displaystyle \sum _{j=1}^m{b}_j{h}_j}-{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^m{v}_i{h}_j{w}_{ij}}}$$

(26)

where $v_i$ was the binary state of input samples $X$ and $b_i$ was the bias of the input samples. $h_j$ was the binary state of the input sample features $j$, and $b_j$ was the bias of the input sample features. The energy function converged as

$$\left|\left.A^{\prime }(v,h)-A(v,h)\right|\right.<\alpha$$

(27)

$$A\left(v,h\right)={\displaystyle \sum _{a=k}^{K+k}\frac{E_a\left(v,h\right)}{K}}$$

(28)

$$A\prime \left(v,h\right)={\displaystyle \sum _{b=k-K}^{k-1}\frac{E_b\left(v,h\right)}{K}}$$

(29)

where $\alpha$ was the defined threshold, $k$ was the iteration number, and $K$ was a constant used to represent the number of variables influencing the monthly runoff forecasting in the basin.

Step 2.1 Assuming that the probability expectation of hidden layer $h_j$ was one.

$$p_j=\frac{1}{1+e^{-\mathsf{\Delta }{E}_j}}$$

(30)

$$\mathsf{\Delta }{E}_j={\displaystyle \sum _{i=1}^n{w}_{ij}{v}_i}+b_j$$

(31)

where $b_j$ was the bias of the hidden layer element $h_j$, $j=1,2,\dots ,m$.

Step 2.2 Assuming that the probability expectation of the input layer $v_i$ was one.

$$p_i=\frac{1}{1+e^{-\mathsf{\Delta }{E}_i}}$$

(32)

$$\mathsf{\Delta }{E}_i={\displaystyle \sum _{j=1}^m{w}_{ij}{h}_j}+b_i$$

(33)

where $b_i$ was the bias of hidden layer element $v_i$, $i=1,2,\dots ,n$.

Step 2.3 Calculating the expected value and reconstruction value of the RBM.

$$p_{ij}={〈v_i{h}_j〉}_{data}$$

(34)

$$p_{ij}^{\prime }={〈v_j{h}_i〉}_{recon}$$

(35)

Step 2.4 Updating the weights $W_{ij}$.

$$\mathsf{\Delta }{W}_{ij}=\epsilon (p_{ij}-p_{ij}{}^{\prime })$$

(36)

where $\epsilon$ represented the learning rate, and $0<\epsilon <1$.

Step 3: Considering the hidden layer of RBM1 as the input layer of RBM2, repeating steps 1 and 2. Similarly, considering the hidden layer of RBM2 as the input layer of RBM3, …, RBML, repeating steps 1 and 2.

Step 4: After training, the output of RBML was the preliminary predicted value $X(d)$.

Step 5: Calculating the mean square error (MSE) of $X(d)$ and the sample-labeled data $\stackrel{\wedge }{X}(d)$. In addition, fine-tuning the parameters of the PSO-FPA-DBN model with the BP algorithm so that the model had a better prediction effect.

Step 6: If the MSE was less than a predefined threshold, the training of the model was completed, and the convergence formula was defined as follows:

$$\frac{MSE(N-1)-MSE(N)}{MSE(N-1)}<\beta$$

(37)

where $\beta$ represented a pre-given threshold and $N$ represented the number of cycles for fine-tuning with the BP algorithm.

Step 7: After the training, a prediction model was constructed to predict the variation in monthly runoff in the basin.

These steps were the main learning algorithms for monthly runoff forecasting based on the PSO-FPA-DBN model. The parameters of the model obtained from the training of multiple RBMs were well-fitted to the optimal solution. This was followed by the adoption of supervised learning with a BPNN to fine-tune the parameters and finally achieve all the training rapidly. This helped the model extract features conveniently from massive, high-dimensional, and multi-factor data.

3.3. Determining the Network Depth of the Proposed Model Based on Particle Swarm Optimization

To construct the PSO-FPA-DBN model, the optimal network depth of the DBN was first solved. An ineffective network depth significantly affected the prediction model accuracy. At present, there are several methods for selecting the network depth, such as trial-and-error and parameter-optimization methods. Trial-and-error methods are used to search for the optimal network depth through many experiments. These increase the computational complexity. Thus, in this study, the PSO algorithm was selected to acquire the optimal network depth based on the network error reconstruction algorithm (Figure 6). The network error reconstruction was defined as

$$RE={\displaystyle \sum _{i=1}^n{(X-S_i)}^2}$$

(38)

where X denoted the ith batch matrix, S_i denoted the reconstruction results of the ith batch matrix, and RE denoted the reconstruction error.

3.4. Parameter Optimization Based on the Flower Pollination Algorithm

Conventional methods, such as the trial-and-error method, can identify better model parameters through repeated trial-and-error. However, these require many experiments and comparisons and have low generalization capability. To solve this problem, this study adopted a dynamic optimization algorithm based on the FPA to improve the accuracy and reliability of monthly runoff using a DBN. The FPA could prevent a local extremum by searching for optimization over a large range and ensured a global optimal solution. This increased the convergence speed of the network and improved the generalization capability of the model.

According to the training sample data, the FPA algorithm was used to dynamically determine the number of neurons in the hidden layer and the learning rate of the RBMs. Therefore, a group of optimal parameter combinations was constructed as a pollen position for iterative updating to solve the global optimization problems (Figure 7).

Assuming that M represented the number of neurons in the hidden layers, L represented the number of RBMs, ${\epsilon }_1$ represented the learning rate of RBM1,${\epsilon }_2$ represented that of RBM2, and ${\epsilon }_L$ represented that of RBML. In this study, we considered an (L + 1)-dimensional vector particle $y(m,{\epsilon }_1,{\epsilon }_2,\dots ,{\epsilon }_L)$, m = 1, 2, …, M, ${\epsilon }_1,{\epsilon }_2,\dots ,{\epsilon }_L\in (0,1)$. The algorithm was divided into the following steps:

Step 1: Parameter initialization

Assuming that $\left[x_i,y_i\right]$ represented the given training samples, $x_i\in R_n$, R_n represented the sample space with n-dimensional feature vectors, $i=1,2,\cdots ,Q$, and Q represented the number of training samples. The population size was set to P, and the maximum number of iterations was set to I.

Step 2: Fitness function selection

Using the root mean square error (RMSE) for model fitting.

Step 3: Iteration and updating

According to step 2, the fitness values were calculated and individuals were updated.

Step 4: Obtaining an optimal parameter combination for the PSO-FPA-DBN model.

Therefore, it was necessary to determine whether the termination condition had been attained. When the termination condition was attained, the optimal parameter combination for the PSO-FPA-DBN model was obtained. Otherwise, return to step 3.

4. Study Area and Data

The Yalong River (25°12–34°9 N, 96°47–102°42 E) is the largest tributary of the Jinsha River in the southern region of the Tibetan Plateau. It runs from the northwest to the southeast. Therefore, the research on monthly runoff forecasting is of high scientific significance and practical value. The geography of the Yalong River is shown in Figure 8.

In this study, the dataset was from the period from January 1960 to December 2011. It included the monthly mean runoff, rainfall, and climatic factors. The data from January 1960 to December 2001 were used for training, and those from January 2002 to December 2011 were used for validation. The original monthly runoff series in Lianghekou, Jinping, Guandi, and Ertan hydrological stations are shown in Figure 9. The runoff data were provided by the Hydrological Bureau of Changjiang Water Resources Commission of The Ministry of Water Resources, China (http://www.cjh.com.cn/ (accessed on 20 January 2023)). The atmospheric circulation factors and meteorological data were provided by the National Climate Center of China (http://data.cma.cn/ (accessed on 20 January 2023)).

5. Results

This section includes the COM selection results, factor selection results, and proposed PSO-FPA-DBN model. Moreover, the proposed PSO-FPA-DBN model was compared with the traditional prediction methods (namely, BPNN and SVM) and other improved DBN models to demonstrate its superiority according to the evaluation criteria in the Yalong River Basin.

5.1. COM Selection Results

According to Section 2.1, the COM was constructed by Lianghekou, Jinping, Guandi, and Ertan hydrological stations, and the weight calculation results are shown in

Table 1. The weight calculation results.

Station	Lianghekou	Jinping	Guandi	Ertan
Percentage of the controlled area	51%	81%	89%	97%
Weight normalization	0.37	0.23	0.21	0.19

.

5.2. Factor Selection Results

According to Section 2.2, this study selected the PMI method for factor selection from the measured COM, rainfall, and climatic factors to effectively promote the prediction effect of monthly runoff.

In terms of predictive variable selection, the predictive factors of the dataset included the measured values of 23 variables from the previous 12 months. Consequently, the candidate factors include COM (com(t-1), com(t-2), …, com(t-12)), area rainfall (ar(t-1), ar(t-2), …, ar(t-12)), and 21 climatic factors (tcf1(tcf1(t-1), tcf1(t-2), …, tcf1(t-12)), tcf2(tcf2(t-1), tcf2(t-2), …, tcf2(t-12)), …, tcf21(tcf21(t-1), tcf21(t-2), …, tcf21(t-12))). The total number of factors is 276.

As described in Section 2.2, the PMI method was employed to determine key influencing factors closely related to runoff variation according to the AIC criterion. The selection procedure was as follows:

(1)

The top 20 candidate variables are listed in

Table 2. The importance ranking of candidate factors in the top 20.

Importance Ranking	The Candidate Factors	Importance Ranking	The Candidate Factors
1	ar(t-1), 253th	2	com(t-12), 276th
3	ar(t-7), 259th	4	tcf1(t-1), 1st
5	ar(t-12), 264th	6	com(t-1), 265th
7	com(t-11), 275th	8	com(t-2), 266th
9	tcf15(t-6),174th	10	tcf3(t-7), 31th
11	tcf13(t-8), 152th	12	tcf16(t-1), 181th
13	tcf16(t-5), 185th	14	com(t-3), 267th
15	tcf5(t-9), 57th	16	tcf21(t-5), 245th
17	tcf3(t-3), 27th	18	tcf4(t-7), 43th
19	tcf14(t-8), 164th	20	tcf7(t-4), 76th

.

(2)

The key variable factors were selected from the above results and a new factor selection result was formed. As a result, the reduced factors were 13 in number, as shown in

Table 3. Factor selection results.

Influencing Factors	Selected Factors
COM	com(t-1), com(t-2), com(t-3), com(t-11), com(t-12)
Rainfall Factors	ar(t-1), ar(t-7), ar(t-12)
Climate Factors	tcf1(t-1), tcf3(t-7), tcf15(t-6), tcf13(t-8), tcf16(t-1)

.

5.3. Monthly Runoff Forecasting Based on PSO-FPA-DBN Model

5.3.1. Network Depth of the Proposed Model

As described in Section 3.3, the DBN was set as follows: 13 input elements, 1 output element, a learning rate of 0.01, and 800 iterations. The optimal network depth of the proposed model was selected from two to six based on the PSO algorithm. The selection process is shown in Figure 10.

As shown in Figure 10, when the network depth of the PSO-FPA-DBN model was four, the COM prediction results were the best. Thus, the optimal network depth of the PSO-FPA-DBN model was four layers.

5.3.2. Parameter Optimization of the Proposed Model

The PSO-FPA-DBN model was trained using 504 sets of sample data collected between January 1960 and December 2001. With the RMSE as the objective, the PSO-FPA-DBN model was used to determine the optimal combination of the number of neurons in the hidden layer and the learning rate of the RBMs. The parameters of the PSO-FPA-DBN model were set as follows:

(1)

The number of elements in the input layer was 13, the number of hidden layers was three, the learning rate for fine-tuning the BP algorithm was 0.01, and the number of training iterations was 600.

(2)

FPA: The population size was 90, the maximum number of iterations was 600, the transition probability was 0.8, the scaling parameter was one, the scaling parameter $\gamma$ was one, and the $\lambda$ was 1.5.

After iteration and updating, when the number of hidden layer neurons was 12, the learning rates of RBM1, RBM2, and RBM3 were 0.2, 0.4, and 0.5, respectively. The RMSE was minimal, and the model performance was optimal. Thus, the optimal parameter combination for the PSO-FPA-DBN model was obtained.

5.3.3. Comparison Models

This study used the MATLAB R2016a software modeling tools as the working platform on a personal computer (Windows 11 operation system; CPU: 12th Gen Intel (R) Core (TM) i5-12400F @ 2.50 GHz; RAM: 16 GB). The parameters of comparison models are set in

Table 4. Parameter setting of comparison models.

Models	Parameter Setting
BPNN	The hidden nodes = 12; the training function = “tansig”, learning function = “logsig”; the maximum training time = 600, learning rate = 0.1, momentum factor = 0.9, and expected error = 0.001; selecting the LM algorithm as the training algorithm.
SVM	The kernel function = “sigmoid”, and the parameters of SVM were optimized via the grid-search algorithm with cross-validation.
DBN-PLSR	The number of iterations of every RBM = 300, the enhancement coefficient of the learning rate = 1.4, the decrease coefficient of the learning rate = 0.7, and the limited value = 0.02.
PSO-GA-DBN	PSO: the population size = 90, the maximum number of iterations = 600, the learning rate = 0.1, and the expected error = 0.001. GA: the population size = 90, the maximum number of iterations = 600, the mutation probability rate = 0.01, and the crossover ratio = 0.7.
PSO-ACO-DBN	PSO: the population size = 90, the maximum number of iterations = 600, the learning rate = 0.1, and the expected error = 0.001. ACO: the ant colony size = 90, the maximum number of iterations = 600, the important factor of pheromone = 1, the importance factor of the heuristic function = 5, and the pheromone factor = 0.1.

Note: BPNN: backpropagation neural network; SVM: support vector machines; PLSR: partial least square regression; GA: genetic algorithm; ACO: artificial bee colony.

.

5.3.4. Runoff Forecasting

To sufficiently demonstrate the highly accurate and reliable runoff prediction results of the proposed PSO-FPA-DBN model, the BPNN, SVM, DBN, DBN-PLSR, PSO-GA-DBN, and PSO-ACO-DBN were selected for a comparison. The monthly runoff forecasting results are shown in Figure 11. The calibrations of the models are shown in Figure 12. The relative percent error based on the seven data-driven models are shown in Figure 13.

As shown in Figure 11, Figure 12 and Figure 13, we can see that the proposed novel PSO-FPA-DBN hybrid model may adapt effectively to the nonlinear characteristics of monthly runoff forecasting and obtain accurate and reliable runoff forecasting results. The monthly runoff forecasting effect of all data-driven models is relatively ideal, and the predicted and observed runoff are highly consistent. In addition, the forecasting effect of the data-driven models coupled with bio-inspired optimization algorithms (i.e., PSO-FPA-DBN, DBN-PLSR, PSO-GA-DBN, and PSO-ACO-DBN) are much better than other models.

6. Discussion

To effectively promote the prediction effect of monthly runoff, we proposed a novel PSO-FPA-DBN hybrid model, which selected the optimal network depth via PSO and searched for the optimum hyperparameters in the DBN using FPA. We also selected six data-driven models (i.e., BPNN, SVM, DBN, and other improved models (DBN-PLSR, PSO-GA-DBN, and PSO-ACO-DBN)) as benchmarks to investigate the performance comparison for monthly runoff prediction. In addition, the MAPE, RMSE, DC, RE, and QR were employed as evaluation indicators of point prediction results to evaluate the prediction accuracy of the above data-driven models. The performance comparison based on the seven data-driven models are shown in

Table 5. Performance comparison between the PSO-FPA-DBN model and other models.

Model	MAPE (%)	RMSE (m3·s−1)	DC	QR (%)
BPNN	24.75	326.29	0.8775	45.8
SVM	24.64	360.02	0.8508	51.7
DBN	41.00	277.50	0.9114	55.8
DBN-PLSR	19.98	229.70	0.9393	61.7
PSO-GA-DBN	18.77	240.20	0.9336	62.5
PSO-ACO-DBN	17.85	237.63	0.9350	63.3
PSO-FPA-DBN	18.23	230.45	0.9389	64.2

Note: QR: qualified rate.

.

As shown in Table 5, the proposed PSO-FPA-DBN model was better than the traditional prediction methods, i.e., BPNN and SVM, and other improved DBN models (DBN-PLSR, PSO-GA-DBN, and PSO-ACO-DBN). Moreover, it could adapt well to the highly nonlinear characteristics of monthly runoff forecasting. For the MAPE index, PSO-ACO-DBN, PSO-FPA-DBN, and PSO-GA-DBN models were better than other models (17.85%, 18.23%, and 18.77%, respectively). These indicated a marginal error via the prediction models based on the bio-inspired algorithms. For the RMSE index, the DBN-PLSR and PSO-FPA-DBN models were the best (229.70 m³/s and 230.45 m³/s, respectively), and the SVM model was the worst. In terms of the DC index, the improved DBN models were better than the other data-driven models (DBN-PLSR model: 0.9393, PSO-FPA-DBN model: 0.9389, PSO-ACO-DBN model: 0.9350, and PSO-GA-DBN model: 0.9336). This revealed that the improved DBN models were in good agreement with the related experimental data. For the QR index, the PSO-FPA-DBN model was the best (64.2%). The BPNN model had a lower score for this index.

To summarize, the methodology presented in this study consists of Comprehensive Basin Response, factor reduction using information entropy, DBN, PSO, and the FPA. Among these, the COM and PMI methods were selected to reduce the factors influencing runoff prediction. The PSO-FPA-DBN method was developed to acquire good runoff prediction results. In a case study of the Yalong River Basin, the comprehensive performance of the PSO-FPA-DBN model was better than those of the BPNN, SVM, and other improved DBN models in terms of the evaluation indicators. Also, the forecasting effect of the data-driven models coupled with bio-inspired optimization algorithms (i.e., PSO-FPA-DBN, DBN-PLSR, PSO-GA-DBN, and PSO-ACO-DBN) were much better than other models, because bio-inspired optimization algorithms could deduct the optimal solutions for the optimization problem and increasing the computational speed.

7. Conclusions

In general, it is difficult to characterize runoff trends and realize accurate and reliable monthly runoff forecasts. To overcome the drawbacks of other conventional data-driven models, a hybrid model using particle swarm optimization coupled with flower pollination algorithm-based deep belief networks (PSO-FPA-DBNs) was proposed. In contrast, the optimization-parameter-selection algorithms of the DBN were studied to obtain highly accurate and reliable results. The novelty of our proposed methodology lied in the Comprehensive Basin Response, PMI-based factor selection, and PSO-FPA-DBN model. Finally, we systematically compared the PSO-FPA-DBN model with the traditional prediction methods for monthly runoff forecasting (i.e., BPNN, SVM, DBN, and other improved models (DBN-PLSR, PSO-GA-DBN, and PSO-ACO-DBN)) using an original dataset containing monthly runoff series measured at the Lianghekou, Jinping, Guandi, and Ertan hydrological stations; rainfall data; and climate data from January 1960 to December 2011, in the Yalong River Basin, China. The experimental results demonstrated that the proposed PSO-FPA-DBN model could adapt effectively to the highly nonlinear characteristics of monthly runoff forecasting. Therefore, it could obtain highly accurate and reliable runoff prediction results.

We acknowledge that certain factors were not considered in this study, such as vegetation data and human activities. Moreover, because of the limit of conditions, more recent data were not included in the study. In the future, we would investigate more underlying surface conditions, human activities, and recent data that would be considered in runoff prediction. In addition, future work would also include investigations of additional data-driven models and their parameter-optimization algorithms. Firstly, some state-of-the-art data-driven models would be developed for monthly runoff forecasting, including a Convolutional Neural Network (CNN), Reinforcement Learning (RL), Long-Short Term Memory network (LSTM), machine learning-based hybrid models, or ensemble approaches integrating physics-based models with deep learning algorithms. Secondly, the development of more efficient multi-parameter optimization methods is based on bio-inspired algorithms for determining the optimal parameters of the data-driven models.