A Short-Term Load Forecasting Model Based on Crisscross Grey Wolf Optimizer and Dual-Stage Attention Mechanism

Abstract

Accurate short-term load forecasting is of great significance to the safe and stable operation of power systems and the development of the power market. Most existing studies apply deep learning models to make predictions considering only one feature or temporal relationship in load time series. Therefore, to obtain an accurate and reliable prediction result, a hybrid prediction model combining a dual-stage attention mechanism (DA), crisscross grey wolf optimizer (CS-GWO) and bidirectional gated recurrent unit (BiGRU) is proposed in this paper. DA is introduced on the input side of the model to improve the sensitivity of the model to key features and information at key time points simultaneously. CS-GWO is formed by combining the horizontal and vertical crossover operators, to enhance the global search ability and the diversity of the population of GWO. Meanwhile, BiGRU is optimized by CS-GWO to accelerate the convergence of the model. Finally, a collected load dataset, four evaluation metrics and parametric and non-parametric testing manners are used to evaluate the proposed CS-GWO-DA-BiGRU short-term load prediction model. The experimental results show that the RMSE, MAE and SMAPE are reduced respectively by 3.86%, 1.37% and 0.30% of those of the second-best performing CSO-DA-BiGRU model, which demonstrates that the proposed model can better fit the load data and achieve better prediction results.

1. Introduction

Electric load forecasting plays an important role in the modernization of power system management and has become the research focus of current power enterprises [1]. It can be divided into long-term, medium-term and short-term forecasting according to its different purposes [2]. Among them, short-term load forecasting can ensure the safe and stable operation of the power system and improve social benefits [3]. Furthermore, it can accelerate the development of the power market and improve economic benefits [4]. Therefore, it is of great significance to design an efficient and accurate short-term load forecasting method.

The early short-term load methods mainly use are the exponential smoothing method [5] and hidden Markov model [6], but the ability of these methods to extract nonlinear characteristics of load is weak [7]. With the rapid increase in the installation of smart meters [8] and the development of artificial intelligence technology [9], short-term load forecasting based on big data analysis has become a current research hotspot, such as the BP neural network [10], extreme learning machine [11], support vector machine [12], etc. In addition, in order to avoid falling into local minima, some scholars use swarm intelligence optimization algorithms to optimize the artificial intelligence model. For example, Niu and Dai [13] proposed a short-term load forecasting model based on modified particle swarm optimization, in which the parameters of least squares supporting a vector machine are optimized. The experimental results show that the regression accuracy and generalization ability of the model have been improved by the proposed algorithm. To address the problem of the ease in which long short-term memory neural networks fall into local minima, a whale optimization algorithm (WOA) is used to optimize the network [14]. Li et al. [15] use grey wolf optimization (GWO) to optimize the parameters of every single kernel in an extreme learning machine to improve its forecasting ability. However, GWO quickly falls into the optima trap or fails to find the global optimal solution [16].

Moreover, the above-mentioned shallow learning method is applicable in the scenario where only the historical load is used for forecasting. If it is necessary to extract deep hidden features in massive load data, a deep learning model is needed. For example, Khan et al. [17] use a convolution neural network to extract the coupling relationship of the input features. Muzaffar and Afshari [18] use a long short-term memory (LSTM) neural network to learn the temporal correlation contained in the load time series data. In addition, a gated recurrent unit (GRU) with simpler structures is also used for short-term load prediction [19], with a high-efficiency ability of feature extraction. To extract the implicit coupling relationship between features and temporal dependency in load time series, the combination of CNN and LSTM [20], or its improved variants (e.g., CNN-GRU [21], GRU-TCN [22] and RCNN-ML-LSTM [23]) are used in load prediction. However, the above deep learning models have the problems of gradient disappearance and gradient explosion [24]. Therefore, it is of great significance to avoid these problems and accelerate the convergence speed of the model, so as to improve the accuracy of load forecasting.

The above-mentioned short-term load forecasting models based on artificial intelligence techniques do not consider the importance of input features, making important features disappear with the increase of step size [25]. Feature selection is the commonly used technique to select out the most appropriate input features in forecasting problems [26]. Kong et al. [27] utilize principal component analysis to determine the major factors affecting wind speed, reducing the dimension of relative features and improving the generation of model. Li et al. [28] develop a feature selection method to choose competitive input features. The above feature selection methods reduce the number of input features only once before prediction by simple correlation analysis, which causes the potentially important input variable be discarded [29]. To address this problem, an attention mechanism is proposed [30], with the advantage of making the model handle the dependency of long time series more easily. For example, Wang et al. [31] proposed a short-term load forecasting model based on a feature attention mechanism (FA), in which the effective characteristics of the input variables are highlighted, leading to improved prediction accuracy. In Ref. [32], a temporal attention mechanism (TA) is applied in short-term load prediction to capture the high-impact time steps of load sequence, so as to further reduce the prediction errors. However, there are few research studies that focus on combining FA with TA to propose a multi-stage attention mechanism, capturing the feature and temporal relationship in load time series.

In view of the shortcomings of existing forecasting models, this paper proposes a short-term load forecasting model (DA-CS-GWO-BiGRU) based on a dual-stage attention mechanism (DA) and crisscross grey wolf optimizer algorithm (CS-GWO). The contributions of this paper are presented as follows:

• Combining the advantages of a feature and temporal attention mechanism, a dual-stage attention mechanism (DA) is introduced in this paper. DA is utilized at the input side of the forecasting model to comprehensively capture the correlation relationship between various variables and temporal dependency in the load time series.
• To address the deficiency of GWO, a novel crisscross grey wolf optimizer algorithm is firstly applied in a short-term load forecasting problem. By introducing horizontal and vertical crossover operators, the global search ability and community diversity of CS-GWO are improved.
• The proposed DA-CS-GWO-BiGRU model is verified by using the real load data set collected in a certain area. The experimental results show that the proposed model has higher forecasting accuracy than other comparison models, and has good application prospects.

The organization of the remainder of this paper is as follows. Section 2 introduces the basic principle of the deep learning model involved in this paper. Section 3 presents the methodology of the proposed DA-CS-GWO-BiGRU short-term load forecasting model. Section 4 introduces the metrics of evaluating predictions. Section 5 focuses on the details of the experiments, and the results are analyzed and discussed. Section 6 points out the limitations of and indicates the subsequent work that could follow this paper. Finally, Section 7 summarizes this paper.

2. Principle of Deep Learning Model

2.1. BiGRU Neural Network

A recurrent neural network (RNN) can only remember short-term dependencies of a time series and is often accompanied by the problem of gradient explosion or disappearance in the training process [33], leading to its limited use in practice. By modifying the calculation method of the hidden state of RNN, GRU and LSTM can effectively strengthen the long-term dependence of time series [19]. The network structure of GRU is shown in Figure 1.

It can be seen from Figure 1 that the GRU network calculates the combination degree of the current input and the previous status information by the reset gate r_t. The calculation process is shown in Equation (1).

$${\mathit{r}}_t=\sigma \left({\mathit{x}}_t{W}_r+{\mathit{h}}_{t-1}{U}_r+b_r\right)$$

(1)

where ${\mathit{x}}_t$ is the input data in t-th time step, ${\mathit{h}}_{t-1}$ is the output of the previous time step, $W_r$ and $U_r$ are the weight metrices of the reset gate, $b_r$ is the bias of the reset gate and $\sigma$ is the sigmoid activation function.

In addition, the GRU network controls the retention of the previous state information ${\mathit{h}}_{t-1}$ in the current state by update gate z_t, and its calculation process is shown in Equation (2).

$${\mathit{z}}_t=\sigma \left({\mathit{x}}_t{W}_z+{\mathit{h}}_{t-1}{U}_z+b_z\right)$$

(2)

where $W_z$ and $U_z$ are the weight metrices of the update gate and $b_z$ is the bias of the update gate.

Next, GRU obtains candidate hidden states through the reset gate based on the updating mechanism of RNN [34], as shown in Equation (3).

$${\tilde{\mathit{h}}}_t=\tanh \left({\mathit{x}}_t{W}_h+\left({\mathit{r}}_t\odot {\mathit{h}}_{t-1}\right)U_h+b_h\right)$$

(3)

where $W_h$ and $U_h$ are the weight metrices of candidate output and $b_h$ is the bias of the candidate output.

Finally, GRU obtains new hidden state ${\mathit{h}}_t$ by considering the previous hidden state ${\mathit{h}}_{t-1}$ and candidate hidden state ${\tilde{\mathit{h}}}_t$ as well as z_t. The calculation process is shown in Equation (4).

$${\mathit{h}}_t={\mathit{z}}_t\odot {\mathit{h}}_{t-1}+\left(1-{\mathit{z}}_t\right)\odot {\tilde{\mathit{h}}}_{t-1}$$

(4)

It can be seen from Equations (1)–(4) that GRU only considers the influence factors on the current time, and lacks consideration of future influence factors [35]. Meanwhile, a recently proposed neural network named bidirectional gated recurrent unit (BiGRU) can effectively make up for the deficiency of GRU. BiGRU can fully excavate the influence relationship hidden in the time series before and after through its unique forward and backward propagation network structure [36]. The network structure of BiGRU is shown in Figure 2, and its calculation process is shown in Equations (5)–(7).

$${\overrightarrow{\mathit{h}}}_t=G({\mathit{x}}_t,{\overrightarrow{\mathit{h}}}_{t-1})$$

(5)

$${\overleftarrow{\mathit{h}}}_t=G({\mathit{x}}_t,{\overleftarrow{\mathit{h}}}_{t-1})$$

(6)

$${\mathit{h}}_t={\overrightarrow{\mathit{w}}}_t{\overrightarrow{\mathit{c}}}_t+{\overleftarrow{\mathit{w}}}_t{\overleftarrow{\mathit{c}}}_t+{\mathit{b}}_t$$

(7)

where ${\overrightarrow{\mathit{h}}}_t$ is the state information of forward propagation, ${\overleftarrow{\mathit{h}}}_t$ is the state information of backward propagation, ${\overrightarrow{\mathit{w}}}_t$ is the weight metrices of the hidden layer in forward propagation, ${\overleftarrow{\mathit{w}}}_t$ is the weight metrices of the hidden layer in backward propagation, ${\mathit{b}}_t$ is the bias of the hidden layer and $G(\cdot )$ is the calculation process of GRU as shown in Equations (1)–(4).

As can be seen from Figure 2, BiGRU can exploit information both from the past and the future. Therefore, this paper uses BiGRU for timing analysis.

2.2. Attention Mechanism

By simulating human visual behavior, the attention mechanism adaptively assigns different attention weights to the input features of the model to highlight the more critical influence factors [37], helping the model predict better.

The attention mechanism is mainly composed of three parts, namely, attention weight calculation, weight normalization and intermediate semantic vector calculation. Firstly, the attention weight e of different features in the model input x or at t-th time step is calculated by using a multi-layer perceptron or neural network. Then, in order to meet the requirement that the sum of attention weights is 1, the attention weight e is normalized to find $\mathit{\alpha }$. Finally, the intermediate semantic vector can be obtained by considering x and $\mathit{\alpha }$ as shown in Equation (8).

$$\mathit{c}=\mathit{\alpha }\mathit{x}$$

(8)

Therefore, this paper expects to use the attention mechanism to capture the coupling relationship between each feature and the impact of information both from the past and the future on the forecasted load value.

3. DA-CS-GWO-BiGRU Short-Term Load Forecasting Model

3.1. Mathematical Model

Set ${\mathit{L}}^{(i)}=\left(L(24\cdot i-23),\dots ,L(24\cdot i)\right)$ as the electric load time series of the previous day, ${\mathit{t}}^{(i)}=\left(t_{\max }(i),t_{\max }(i+1),t_{\min }(i),t_{\min }(i+1)\right)$ as the highest and lowest temperature of the previous day and the current day, ${\mathit{r}}^{(i)}=\left(r(i),r(i+1)\right)$ as the rainfall of the previous day and the current day and ${\mathit{d}}^{(i)}=\left(d(i),d(i+1)\right)$ as the weather day type of the previous day and the current day. The short-term load forecasting problem can be regarded as using the load information of the previous day ${\mathit{L}}^{(i)}$, and combining its relevant characteristics ${\mathit{t}}^{(i)}$, ${\mathit{r}}^{(i)}$ and ${\mathit{d}}^{(i)}$ to make a prediction of the electric load values in the current day. Let the function map of the model be $F_{\theta }$, and the prediction process is shown in Equation (9).

$${\widehat{\mathit{Y}}}^{(i)}=F_{\theta }({\mathit{X}}^{(i)})$$

(9)

where ${\mathit{X}}^{(i)}=\left[{\mathit{L}}^{(i)},{\mathit{t}}^{(i)},{\mathit{r}}^{(i)},{\mathit{d}}^{(i)}\right]=\left(x_1^{(i)},x_2^{(i)},\dots ,x_T^{(i)}\right)\in {ℝ}^T$, and ${\widehat{\mathit{Y}}}^{(i)}$ represents the predicted load values of the current day.

3.2. Dual-Stage Attention Mechanism

As shown in Section 3.1, the prediction model takes historical load time series ${\mathit{L}}^{(i)}$, temperature ${\mathit{t}}^{(i)}$, rainfall ${\mathit{r}}^{(i)}$ and weather day type ${\mathit{d}}^{(i)}$ as inputs. According to [38], different time steps and different features of the same time step have unequal effects on the output. In order to simultaneously enhance the sensitivity of features and the temporal dimension while making short-term load predictions, this paper proposes a novel attention mechanism named dual-stage attention mechanism (DA) that combines a feature attention (FA) and a temporal attention mechanism (TA). Combining the advantages of TA and FA, DA can fully capture the relationship between variables and temporal dependence in load time series, so as to provide data support for an efficient forecasting model.

3.2.1. Feature Attention Mechanism

To highlight the more critical influence features of the input, FA is introduced. As the successor of the attention mechanism, FA also has three parts, and the calculation of the attention weight of the input features is realized through the neural network in this paper. The realization of FA is demonstrated in Figure 3 and as below.

(1) Attention weight calculation: Set $\mathit{X}=\left({\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_T\right)\in {ℝ}^{N\times T}$ as the input vector of the prediction model, where ${\mathit{x}}_j={\left\{x_j^{(i)}\right\}}_{i=1}^N\in {ℝ}^N(j\in \left\{1,2,\dots ,n_f\right\})$ and N is the number of samples. The quantization process of the corresponding weight of each feature is shown in Equation (10).

$$\mathit{e}=\sigma (\mathit{X}{\mathit{W}}_e+{\mathit{b}}_e)$$

(10)

Among them, $\mathit{e}=\left(e_1,e_2,\dots ,e_T\right)$ is the unnormalized attention weight, ${\mathit{W}}_e\in {ℝ}^{T\times T}$ is the trainable coefficient matrix and ${\mathit{b}}_e\in {ℝ}^T$ is the bias.

(2) Weight normalization: In order to make the attention weight satisfy the probability distribution whose sum is 1, $\mathit{e}$ is normalized by the softmax function, as shown in Equation (11).

$${\alpha }_j=\mathrm{softmax}(e_j)=\exp (e_j)/{\displaystyle \sum _{i=1}^T\exp (e_i)}$$

(11)

(3) Intermediate semantic vector calculation: The normalized weight ${\alpha }_j$ is multiplied by the corresponding feature vector ${\mathit{x}}_j$ to achieve the purpose of enhancing or reducing the expression of ${\mathit{x}}_j$. Finally, the adaptively optimized feature vector ${\mathit{X}}^{\mathrm{ATT}}$ can be obtained as shown in Equation (12).

$${\mathit{X}}^{\mathrm{ATT}}=\left({\alpha }_1{\mathit{x}}_1,{\alpha }_2{\mathit{x}}_2,\dots ,{\alpha }_T{\mathit{x}}_T\right)$$

(12)

It is worth noting that the attention weights are dynamically changed during the training process of the model, and the weights are determined only when the iterations converge.

3.2.2. Temporal Attention Mechanism

In order to capture the temporal correlation relationship between each time step in ${\mathit{X}}^{\mathrm{ATT}}$ and the current prediction results, TA is introduced. The adaptive extraction of features at important moments is realized by integrating TA and the BiGRU network. The implementation is shown in Figure 4 and as below.

TA also has three parts, which are the same as FA.

(1) Attention weight calculation: Taking the vector ${\mathit{X}}^{\mathrm{ATT}}$ containing the feature association relationship and the hidden state ${\mathit{h}}_{t-1}$ at the previous time step of BiGRU as the input of TA, the attention weights at t-th time step in the iterative process are quantified, as shown in Equation (13).

$${\mathit{f}}_t=\sigma (\left[{\mathit{X}}^{\mathrm{ATT}};{\mathit{h}}_{t-1}\right]{\mathit{W}}_f+{\mathit{b}}_f)$$

(13)

where ${\mathit{W}}_f\in {ℝ}^{(2T)\times n_p}$ is the trainable coefficient matrix, ${\mathit{b}}_f\in {ℝ}^{n_p}$ is the bias and $n_p$ is the number of hidden elements in the last layer of BiGRU.

(2) Weight normalization: In addition, ${\mathit{f}}_t$ is normalized using the softmax function, as shown in Equation (14).

$${\beta }_t^j=\exp (f_t^j)/{\displaystyle \sum _{i=1}^T\exp (f_t^i)}$$

(14)

(3) Intermediate semantic vector calculation: In order to obtain the implicit temporal correlation relationship at t-th time step, ${\beta }_t^j$ and ${\alpha }_t{\mathit{x}}_t$ are weighted and summed to obtain an intermediate semantic vector ${\mathit{d}}_t$, which contains features and temporal-related information, as shown in Equation (15).

$${\mathit{d}}_t={\displaystyle \sum _{j=1}^T{\beta }_t^j{\alpha }_j{\mathit{x}}_j}$$

(15)

Once the iteration of BiGRU terminates, the final hidden state ${\mathit{h}}_T$ and the intermediate semantic vector ${\mathit{d}}_T$ are obtained. The final prediction is taken by a single-layer feedforward network utilizing ${\mathit{h}}_T$ and ${\mathit{d}}_T$, as shown in Equation (16).

$${\mathit{Y}}^{\prime }=\left[{\mathit{h}}_T;{\mathit{d}}_T\right]{\mathit{W}}_y+{\mathit{b}}_y$$

(16)

where ${\mathit{W}}_y\in {ℝ}^{(2T)\times n_p}$ is the weight metrices of the feedforward network and ${\mathit{b}}_y\in {ℝ}^{n_p}$ is the bias.

Assuming that all parameters of the DA-BiGRU model are $\mathit{\theta }$, its loss function is shown in Equation (17). With the goal of minimizing this loss function, the final DA-BiGRU prediction model is obtained when the training is over.

$$J({\mathit{Y}}^{\prime },\mathit{Y},\mathit{\theta })=\frac{1}{N}{\displaystyle \sum _{i=1}^N{({{\mathit{Y}}^{\prime }}^{(i)},{\mathit{Y}}^{(i)})}^2}$$

(17)

3.3. CS-GWO Optimization Algorithm

Facts have proved that the prediction model trained with an Adam optimizer can achieve rapid convergence in the early stage, but the learning rate is too low in the later stage of training, which may affect the effective convergence of the model and cause generalization problems [39]. Swarm intelligence optimization algorithms (e.g., PSO, GWO) are used to address the aforementioned problems; however, this leads to another problem of failing to find global optimal solutions [16]. In Ref. [40], an improved GWO algorithm created by incorporating a crisscross optimization algorithm (CSO) [41] is proposed for solving the optimal power flow problem, effectively avoiding falling into the local optimum and preventing the premature convergence. Inspired by this, this paper applies a crisscross grey wolf optimizer (CS-GWO) to optimize the DA-BiGRU model in the early stage of training, so as to further accelerate the convergence and improve the generation of the short-term load prediction model. Compared with GWO, CS-GWO achieves better global search ability and group diversity by introducing horizontal crossover and vertical crossover operators of CSO.

The implementation of CS-GWO is mainly composed of five parts, which are parameter initialization, hunting, attacking prey, horizontal crossover and vertical crossover. The implementation process is described in detail as follows.

3.3.1. Parameter Initialization

Set the population of grey wolf as $\mathit{\Phi }={\left[{\mathit{\theta }}_1,{\mathit{\theta }}_2,\dots ,{\mathit{\theta }}_{n_M}\right]}^T\in {ℝ}^{n_M\times D}$, where $n_M$ is the population size and D is the population dimension. Select the individual with the best fitness value in population $\mathit{\Phi }$ as grey wolf $\alpha$, the individual with the second-best fitness value in population $\mathit{\Phi }$ as grey wolf $\beta$, the individual with the third-best fitness value in population $\mathit{\Phi }$ as grey wolf $\delta$, and the rest of the population $\mathit{\Phi }$ as grey wolf $\omega$.

3.3.2. Hunting

Since the optimal hunting position of the wolves is unknown in the abstract search space, it is necessary to set three wolves with the strongest hunting ability to guide the hunting of the wolves. Assume that the three wolves are $\alpha$, $\beta$ and $\omega$, respectively, and the other wolf $\omega$ updates its position according to the positions of the above three wolves during the iteration process, as shown in Equations (18) and (19).

$$\left\{\begin{array}{c}{\mathit{\theta }}_{\alpha }^{\prime }={\mathit{\theta }}_{\alpha }(t)-A_1\cdot \left|C_1{\mathit{\theta }}_{\alpha }(t)-{\mathit{\theta }}_{\omega }(t)\right|\\ {\mathit{\theta }}_{\beta }^{\prime }={\mathit{\theta }}_{\beta }(t)-A_2\cdot \left|C_2{\mathit{\theta }}_{\beta }(t)-{\mathit{\theta }}_{\omega }(t)\right|\\ {\mathit{\theta }}_{\delta }^{\prime }={\mathit{\theta }}_{\delta }(t)-A_3\cdot \left|C_3{\mathit{\theta }}_{\delta }(t)-{\mathit{\theta }}_{\omega }(t)\right|\end{array}\right.$$

(18)

$${\mathit{\theta }}_{\omega }(t+1)=\frac{{\mathit{\theta }}_{\alpha }^{\prime }+{\mathit{\theta }}_{\beta }^{\prime }+{\mathit{\theta }}_{\delta }^{\prime }}{3}$$

(19)

where ${\mathit{\theta }}_{\alpha }(t)$, ${\mathit{\theta }}_{\beta }(t)$, ${\mathit{\theta }}_{\delta }(t)$ and ${\mathit{\theta }}_{\omega }(t)$ are the positions of grey wolves $\alpha$, $\beta$, $\delta$ and $\omega$ at t-th iteration, respectively, and $A_1$, $A_2$, $A_3$, $C_1$, $C_2$ and $C_3$ are synergy coefficients, which are calculated as Equations (20) and (21).

3.3.3. Attack Prey

Once the prey is at rest, grey wolves stop searching and attack the prey. To simulate this process, GWO designs a synergy coefficient A, as shown in Equation (20).

$$A=2a\cdot r_1-a$$

(20)

where $r_1$ is a random number in the range of [0, 1], and a linearly decreases from 2 to 0 during the entire iteration process.

It can be seen from Equation (20) that GWO simulates the attack process of grey wolves. When $\left|A\right|<1$, the wolves attack the prey; when $\left|A\right|>1$, the wolves leave the prey alone, hoping to find better prey.

In addition, in order to avoid local optimum, GWO also designs another synergy coefficient C, as shown in Equation (21).

$$C=2r_2$$

(21)

where $r_2$ is a random number in the range of [0, 1].

3.3.4. Horizontal Crossover

In order to improve the global search ability of GWO, horizontal crossover (HC) is used to perform arithmetic crossover operations between two different individuals in all dimensions. Assuming that the i-th parent ${\mathit{\theta }}_i$ and the j-th parent ${\mathit{\theta }}_j\left(i,j\in \left\{1,2,\dots ,n_M\right\}\right)$ perform HC operations on the d-th dimension, respectively, their offspring can be expressed as:

$$\left\{\begin{array}{l}{\mathit{\theta }}_{i,d}^{HC}=r_3\times {\mathit{\theta }}_{i,d}+\left(1-r_3\right)\times {\mathit{\theta }}_{j,d}+C_4\times \left({\mathit{\theta }}_{i,d}-{\mathit{\theta }}_{j,d}\right)\\ {\mathit{\theta }}_{j,d}^{HC}=r_4\times {\mathit{\theta }}_{j,d}+\left(1-r_4\right)\times {\mathit{\theta }}_{i,d}+C_5\times \left({\mathit{\theta }}_{j,d}-{\mathit{\theta }}_{i,d}\right)\end{array}\right.$$

(22)

where r₃ and r₄ are uniformly distributed random values in the range of [0, 1], and C₄ and C₅ are uniformly distributed random values in the range of [−1, 1]. Once the HC is over, the new population ${\mathit{\Phi }}^{HC}={\left[{\mathit{\theta }}_1^{HC},{\mathit{\theta }}_2^{HC},\dots ,{\mathit{\theta }}_{n_M}^{HC}\right]}^T\in {ℝ}^{n_M\times D}$ can be obtained.

3.3.5. Vertical Crossover

In order to improve the population diversity of GWO, vertical crossover (VC) is used to perform arithmetic crossover operations for all individuals between two different dimensions to generate offspring. Assuming that the $d_1$-th dimension and $d_2$-th dimension of the individual perform VC operations, the offspring can be expressed as:

$${\mathit{\theta }}_{i,d_1}^{VC}=r\times {\mathit{\theta }}_{i,d_1}+\left(1-r\right)\times {\mathit{\theta }}_{i,d_2}$$

(23)

where r is a random value uniformly distributed in the range of [0, 1]. Once the VC is over, a new population ${\mathit{\Phi }}^{VC}={\left[{\mathit{\theta }}_1^{VC},{\mathit{\theta }}_2^{VC},\dots ,{\mathit{\theta }}_{n_M}^{VC}\right]}^T\in {ℝ}^{n_M\times D}$ is obtained.

3.3.6. The Detailed Implementation Steps of CS-GWO

To solve the problems of gradient disappearance and gradient explosion problems in deep neural networks [42], the CS-GWO is used to optimize weights and bias $\mathit{\theta }$ of the DA-GRU model in the early stage of training, aiming to improve the generalization performance of the model. The flow chart of CS-GWO is shown in Figure 5, and the detailed implementation steps of the CS-GWO algorithm are as follows:

(1) Initialize parameters: Set the number of grey wolf populations $n_M$, the maximum number of iterations T and the population dimension D (the number of weights and bias of the DA-BiGRU model), and initialize the population $\mathit{\Phi }$.

(2) Set the fitness function: Take Equation (17) as the fitness function.

(3) Determine wolves  $\alpha$, $\beta$ and $\delta$: Use Equation (17) to calculate the fitness of each individual; the individual with the best fitness value is grey wolf $\alpha$, the second-best fitness value is grey wolf $\beta$ and the third-best fitness value is grey wolf $\omega$.

(4) Update position and synergy coefficient: Firstly, update the position of grey wolf $\omega$ according to Equations (18) and (19), and then update A and C according to Equations (20) and (21).

(5) Horizontal crossover: According to Equation (22), perform horizontal crossover on the parent population $\mathit{\Phi }$ to obtain the offspring population ${\mathit{\Phi }}^{HC}$, and use Equation (17) to calculate the fitness of each individual. If the fitness of individual ${\mathit{\theta }}_k$$\left(k\in \left\{1,2,\dots ,n_M\right\}\right)$ in $\mathit{\Phi }$ is worse than that of individual ${\mathit{\theta }}_k^{HC}$ in ${\mathit{\Phi }}^{HC}$, replace ${\mathit{\theta }}_k$ with ${\mathit{\theta }}_k^{HC}$; otherwise, do not replace.

(6) Vertical crossover: According to Equation (23), carry out horizontal crossover on the parent population $\mathit{\Phi }$ to obtain the offspring population ${\mathit{\Phi }}^{VC}$, and use Equation (17) to calculate the fitness of each individual. If the fitness of individual ${\mathit{\theta }}_k$ in $\mathit{\Phi }$ is worse than that of individual ${\mathit{\theta }}_k^{VC}$ in ${\mathit{\Phi }}^{VC}$, replace ${\mathit{\theta }}_k$ with ${\mathit{\theta }}_k^{VC}$; otherwise, do not replace.

(7) Iteration termination: If the number of iterations reaches T, the position of grey wolf $\alpha$ is used as the initial weight and threshold ${\mathit{\theta }}_{\alpha }$ of the DA-GRU model; otherwise, return to step (3) and continue the iteration.

4. Evaluation Index

In order to evaluate the effectiveness of the proposed prediction model, this paper uses root mean square error (RMSE), mean absolute error (MAE), symmetric mean absolute percentage error (SMAPE) and decision coefficient (R²) to evaluate the prediction results. The definitions of the four evaluation indicators are shown in Equations (24)–(27), where RMSE, MAE and SMAPE are indicators used to describe the error between the predicted value and the real value. The smaller the value, the more accurate the prediction result. Furthermore, R² is the indicator used to assess the linear relationship between the input and output values. The larger the value, the higher the prediction accuracy.

$$\mathrm{RMSE}=\sqrt{\frac{1}{n_{test}}{\displaystyle \sum _{n_{test}}{\left({\mathit{Y}}_{test}-{\widehat{\mathit{Y}}}_{test}\right)}^2}}$$

(24)

$$\mathrm{MAE}=\frac{1}{n_{test}}{\displaystyle \sum _{n_{test}}\left|{\mathit{Y}}_{test}-{\widehat{\mathit{Y}}}_{test}\right|}$$

(25)

$$\mathrm{SMAPE}=\frac{1}{n_{test}}{\displaystyle \sum _{n_{test}}\frac{\left|{\mathit{Y}}_{test}-{\widehat{\mathit{Y}}}_{test}\right|}{\left({\mathit{Y}}_{test}+{\widehat{\mathit{Y}}}_{test}\right)/2}}$$

(26)

$${\mathrm{R}}^2=1-\frac{{\displaystyle \sum _{n_{test}}{\left({\mathit{Y}}_{test}+{\widehat{\mathit{Y}}}_{test}\right)}^2}}{{\displaystyle \sum _{n_{test}}{\left({\mathit{Y}}_{test}-{\overline{\mathit{Y}}}_{test}\right)}^2}}$$

(27)

where ${\mathit{Y}}_{test}$ and ${\widehat{\mathit{Y}}}_{test}$ are the actual and predicted load values in the testing dataset, respectively, ${\overline{\mathit{Y}}}_{test}$ is the average value of the actual load value and $n_{test}$ is the sample number of the testing dataset.

5. Experiment and Analysis

This section will verify the effectiveness of the proposed DA-CS-GWO-BiGRU short-term load forecasting model through four evaluation indicators (RMSE, MAE, SMAPE and R²) and two experiments. In addition, in order to reduce the errors caused by the experimental operation, both experiments were performed 20 times, and the average value was taken as the final experimental result. Both experiments are based on Python 3.8 and the Keras deep learning library. The core configuration of the used computer is Intel (R) Core (TM) i5-9600K 6-core processor, 3.70 GHz operating frequency, 8 GB memory capacity and Windows 10 operating system.

Particularly, the load data used in this paper is the real sample data of a region in 2018. These sample data have a total of 365 pieces, and their time resolution is 24. In order to reduce the influence of data distribution on the experimental results, the data is randomly sorted; 300 samples are selected as the training dataset, 30 samples are used as the validation dataset and 35 samples are used as the testing dataset. These datasets are depicted in Figure 6.

5.1. Parameter Settings

The attention mechanism is realized by a single-layer fully connected neural network, the number of neurons is 32 and the activation function is softmax.

The number of neurons in the hidden layer of the prediction model based on a BP neural network is 32, the activation function is ReLU, the number of neurons in the output layer is 24 and the activation function is linear. The unit number of the models based on GRU and BiGRU is 32, the number of neurons in the output layer is 24 and the activation function is linear. The BP, GRU, BiGRU, FA-BiGRU, TA-BiGRU, and DA-BiGRU models all use the Adam optimizer, and their hyperparameters β₁, β₂ and ε are set to 0.9, 0.999 and 1 × 10⁻⁸, respectively. In addition, MSE is used as the loss function, and the number of iterations of these models is set to 500.

5.2. Case 1: The Effectiveness of the BiGRU Model and Dual-Stage Mechanism

In order to better evaluate the effectiveness of the DA-BiGRU prediction model proposed in this paper, this section verifies the superiority of the BiGRU model and effectiveness of the dual-stage attention mechanism from the aspect of short-term load prediction. Persistence, BP, GRU, BiGRU, feature-attention-mechanism-based BiGRU (FA-BiGRU) and temporal-attention-mechanism-based BiGRU (TA-BiGRU) models are compared with the DA-BiGRU model in this case. The experimental results are shown in

Table 1. The experiment results of case study 1.

Prediction Model	RMSE/MW	MAE/MW	SMAPE	R2
persistence	36.679	29.258	4.810	0.900
BP	34.671	27.650	4.399	0.911
GRU	32.141	25.128	3.892	0.924
BiGRU	31.009	25.051	3.868	0.929
FA-BiGRU	29.583	23.068	3.591	0.935
TA-BiGRU	29.840	24.239	3.830	0.930
DA-BiGRU	29.053	22.897	3.566	0.937

and Figure 7, where Figure 7 is a comparison chart between the prediction results of different models and the real values on 29–31 December 2018.

As shown in Table 1 and Figure 7, the following conclusions can be drawn:

(1) The advantages of BiGRU model:

The prediction performance of the deep learning model is the best among the single prediction models (i.e., persistence, BP, GRU and BiGRU models), and the prediction accuracy of the BiGRU model is the highest. For example, compared with the classic baseline model persistence, the RMSE, MAE and SMAPE values of the BiGRU model are reduced by 15.46%, 14.38% and 0.942%, respectively, and the R² value is increased by 1.67%. Compared with the shallow neural network BP model, the RMSE, MAE and SMAPE values of the BiGRU model are reduced by 10.56%, 9.40% and 0.531%, respectively, and the R² value is increased by 1.98%. In addition, compared with the GRU model, the BiGRU model has the best RMSE, MAE, SMAPE and R² values.

The reasons are as follows: Firstly, the machine learning model uses a large amount of historical load data for training, which can effectively capture the nonlinear relationship of load time series, and the prediction performance is improved compared with the persistence model. Secondly, the BiGRU model has a unique bidirectional propagation structure, which can link the past and future influencing factors with the current load time series so as to improve the accuracy of short-term load forecasting.

(2) The effectiveness of the dual-stage attention mechanism:

This model combined with the feature attention mechanism can automatically extract the correlation between each feature, which has the ability of reducing the prediction error of the model. Compared with the BiGRU model, the RMSE, MAE and SMAPE values of the FA-BiGRU model decreased by 4.60%, 7.92% and 7.16%, respectively, and the R² value increased by 0.65%.

This model combined with the temporal attention mechanism realizes the adaptive extraction of features at important moments, which improves the prediction stability of the model. Compared with the BiGRU model, the RMSE, MAE and SMAPE values of the TA-BiGRU model decreased by 3.77%, 3.24% and 0.98%, respectively, and the R² value increased by 0.11%.

In addition, compared with other comparison models in this case study, the proposed DA-BiGRU model has the best RMSE, MAE, MAPE and R² values. This is because this model combines feature and temporal attention mechanisms, which can improve the sensitivity of the model to key features and key time steps, and finally achieve the purpose of improving prediction accuracy.

5.3. Case 2: The Effectiveness of the CS-GWO Algorithm

The popular suite of benchmark functions in validating optimization performance, i.e., CEC 2017 [43], is utilized in this subsection to conduct extensive optimization experiments. The CEC 2017 test suite has 30 functions, which can be divided into four categories: unimodal functions (F₁–F₃), multimodal functions (F₄–F₁₀), hybrid functions (F₁₁–F₂₀) and composition functions (F₂₁–F₃₀). The ideal optimal value of each benchmark functions is 0.

Moreover, the well-known optimization algorithms (i.e., PSO, WOA, GWO and CSO) are compared with CS-GWO to evaluate the effectiveness of the CS-GWO algorithm from various perspectives, including accuracy, the Wilcoxon signed-rank test and a paired samples t-test.

5.3.1. The Setting of the Numerical Experiments

The dimension of benchmark functions is uniformly set to 30 in this subsection. For a fair comparison, the number of iterations of the swarm intelligence optimization algorithm and the number of individuals is set to 3000 and 30, respectively. The position of PSO is set in the range of [−1, 1], and the limit of speed is set in the range of [−0.5, 0.5] [44]. For WOA [45], GWO [46] and CS-GWO, the limit of individual is set in the range of [−1, 1]. Among them, the vertical crossover probability of CSO and CS-GWO is set to 60%, and the horizontal intersection crossover is set to 100% [47]. To reduce statistical errors, all the reported results in this subsection are based on 30 independent runs.

5.3.2. The Comparison of Optimization Accuracy

The above-mentioned algorithms are evaluated using the CEC 2017 test suite, and the experimental results are shown in

Table 2. Comparison of CS-GWO with well-known algorithms for CEC 2017 test functions.

Functions	Metrics	PSO	WOA	GWO	CSO	CS-GWO
F1	Mean	3.169 × 1011	2.408 × 103	3.781 × 1010	2.275 × 1010	4.875 × 103
	Min	1.137 × 1011	1.070 × 102	5.121 × 109	4.985 × 109	1.000 × 102
	Max	5.806 × 1011	9.847 × 103	8.496 × 1010	6.467 × 1010	1.771 × 104
	Std	9.145 × 1010	2.219 × 103	2.197 × 1010	1.430 × 1010	5.283 × 103
	Rank	5	1	4	3	2
F2	Mean	8.880 × 102	7.076 × 102	6.260 × 102	6.012 × 102	5.550 × 102
	Min	8.253 × 102	6.184 × 102	5.892 × 102	5.705 × 102	5.129 × 102
	Max	9.519 × 102	8.124 × 102	7.649 × 102	6.601 × 102	6.350 × 102
	Std	3.825 × 101	4.662 × 101	3.205 × 101	2.046 × 101	3.437 × 101
	Rank	5	4	3	2	1
F3	Mean	1.406 × 105	5.016 × 104	4.718 × 104	3.481 × 104	3.001 × 102
	Min	8.673 × 104	2.929 × 104	3.286 × 104	1.132 × 104	3.000 × 102
	Max	3.292 × 105	7.088 × 104	6.894 × 104	6.353 × 104	3.003 × 102
	Std	6.473 × 104	1.017 × 104	9.881 × 103	1.028 × 104	6.049 × 10−2
	Rank	5	4	3	2	1
F4	Mean	6.990 × 103	4.687 × 102	6.504 × 102	5.645 × 102	4.937 × 102
	Min	2.951 × 103	4.001 × 102	5.166 × 102	4.833 × 102	4.641 × 102
	Max	1.422 × 104	4.911 × 102	1.218 × 103	7.337 × 102	5.187 × 102
	Std	2.613 × 103	1.995 × 101	1.457 × 102	6.179 × 101	1.520 × 101
	Rank	5	1	4	3	2
F5	Mean	8.966 × 102	7.087 × 102	6.214 × 102	5.966 × 102	5.774 × 102
	Min	8.469 × 102	6.323 × 102	5.671 × 102	5.681 × 102	5.202 × 102
	Max	9.739 × 102	7.816 × 102	6.944 × 102	6.708 × 102	6.720 × 102
	Std	3.825 × 101	4.073 × 101	2.883 × 101	2.248 × 101	4.910 × 101
	Rank	5	4	3	2	1
F6	Mean	7.009 × 102	6.730 × 102	6.270 × 102	6.196 × 102	6.003 × 102
	Min	6.795 × 102	6.482 × 102	6.126 × 102	6.102 × 102	6.000 × 102
	Max	7.331 × 102	7.250 × 102	6.460 × 102	6.366 × 102	6.017 × 102
	Std	1.221 × 101	1.364 × 101	8.497 × 100	6.550 × 100	4.418 × 10−1
	Rank	5	4	3	2	1
F7	Mean	1.397 × 103	1.140 × 103	8.841 × 102	8.559 × 102	8.710 × 102
	Min	1.218 × 103	1.022 × 103	8.019 × 102	7.960 × 102	7.698 × 102
	Max	1.525 × 103	1.324 × 103	1.076 × 103	9.970 × 102	8.977 × 102
	Std	6.925 × 101	7.795 × 101	6.307 × 101	5.190 × 101	3.069 × 101
	Rank	5	4	3	1	2
F8	Mean	1.103 × 103	9.403 × 102	9.039 × 102	8.979 × 102	8.742 × 102
	Min	1.002 × 103	9.114 × 102	8.541 × 102	8.479 × 102	8.129 × 102
	Max	1.164 × 103	9.910 × 102	9.464 × 102	1.032 × 103	9.867 × 102
	Std	3.661 × 101	2.327 × 101	2.433 × 101	3.537 × 101	4.819 × 101
	Rank	5	4	3	2	1
F9	Mean	9.729 × 103	7.439 × 103	2.287 × 103	2.092 × 103	9.002 × 102
	Min	5.733 × 103	3.310 × 103	1.410 × 103	1.035 × 103	9.000 × 102
	Max	1.324 × 104	1.376 × 104	4.608 × 103	3.691 × 103	9.029 × 102
	Std	1.659 × 103	3.347 × 103	7.803 × 102	7.556 × 102	5.347 × 10−1
	Rank	5	4	3	2	1
F10	Mean	8.429 × 103	6.333 × 103	5.139 × 103	4.483 × 103	7.671 × 103
	Min	6.359 × 103	4.002 × 103	2.849 × 103	3.091 × 103	6.807 × 103
	Max	1.005 × 104	9.761 × 103	8.679 × 103	7.903 × 103	8.508 × 103
	Std	8.850 × 102	1.699 × 103	1.674 × 103	1.270 × 103	4.446 × 102
	Rank	5	3	2	1	4
F11	Mean	8.717 × 103	1.220 × 103	2.183 × 103	1.786 × 103	1.159 × 103
	Min	4.894 × 103	1.153 × 103	1.389 × 103	1.276 × 103	1.108 × 103
	Max	1.418 × 104	1.298 × 103	4.783 × 103	3.937 × 103	1.217 × 103
	Std	2.371 × 103	3.880 × 101	9.575 × 102	7.822 × 102	3.555 × 101
	Rank	5	2	4	3	1
F12	Mean	2.599 × 1010	1.764 × 105	3.831 × 108	5.773 × 108	3.160 × 105
	Min	2.572 × 109	1.191 × 104	2.075 × 107	2.392 × 107	2.261 × 104
	Max	2.031 × 1011	8.030 × 105	1.507 × 109	3.799 × 109	1.485 × 106
	Std	3.719 × 1010	1.616 × 105	3.689 × 108	8.210 × 108	3.465 × 105
	Rank	5	1	4	3	2
F13	Mean	1.224 × 1010	1.733 × 104	1.497 × 108	4.750 × 107	1.410 × 104
	Min	2.027 × 108	3.680 × 103	3.990 × 104	4.008 × 104	1.416 × 103
	Max	2.020 × 1011	4.611 × 104	1.498 × 109	1.405 × 109	4.460 × 104
	Std	3.616 × 1010	1.109 × 104	4.094 × 108	2.564 × 108	1.138 × 104
	Rank	5	2	4	3	1
F14	Mean	4.157 × 106	1.219 × 104	4.015 × 105	1.794 × 105	4.376 × 104
	Min	1.956 × 104	1.746 × 103	2.666 × 104	2.325 × 103	4.285 × 103
	Max	4.485 × 107	1.370 × 105	1.337 × 106	9.272 × 105	2.532 × 105
	Std	8.338 × 106	2.417 × 104	4.196 × 105	2.868 × 105	5.106 × 104
	Rank	5	1	4	3	2
F15	Mean	1.424 × 109	8.116 × 103	4.410 × 106	4.594 × 106	3.994 × 103
	Min	2.520 × 105	1.731 × 103	2.667 × 104	1.355 × 104	1.607 × 103
	Max	1.273 × 1010	3.219 × 104	3.595 × 107	9.453 × 107	2.100 × 104
	Std	2.717 × 109	8.011 × 103	9.902 × 106	1.740 × 107	4.244 × 103
	Rank	5	2	4	3	1
F16	Mean	5.003 × 103	2.990 × 103	2.559 × 103	2.455 × 103	2.489 × 103
	Min	3.196 × 103	2.365 × 103	2.069 × 103	2.033 × 103	1.700 × 103
	Max	1.215 × 104	3.871 × 103	3.254 × 103	3.319 × 103	3.017 × 103
	Std	2.024 × 103	3.759 × 102	3.002 × 102	3.224 × 102	3.925 × 102
	Rank	5	4	3	1	2
F17	Mean	5.052 × 103	3.017 × 103	2.460 × 103	2.392 × 103	2.126 × 103
	Min	3.917 × 103	2.269 × 103	2.102 × 103	1.990 × 103	1.612 × 103
	Max	1.022 × 104	3.507 × 103	3.213 × 103	3.419 × 103	3.014 × 103
	Std	1.165 × 103	3.432 × 102	2.835 × 102	3.141 × 102	3.478 × 102
	Rank	5	4	3	2	1
F18	Mean	1.149 × 108	7.835 × 105	1.858 × 106	1.463 × 106	1.632 × 105
	Min	2.326 × 106	8.942 × 104	5.083 × 104	8.757 × 104	4.111 × 104
	Max	7.292 × 108	2.771 × 106	2.160 × 107	8.673 × 106	4.130 × 105
	Std	2.234 × 108	6.988 × 105	4.064 × 106	1.773 × 106	8.817 × 104
	Rank	5	4	3	2	1
F19	Mean	1.516 × 109	9.880 × 103	1.345 × 107	2.830 × 106	7.321 × 103
	Min	8.062 × 106	1.979 × 103	3.550 × 104	6.985 × 103	1.991 × 103
	Max	2.216 × 1010	4.385 × 104	3.380 × 108	1.373 × 107	3.052 × 104
	Std	4.402 × 109	9.797 × 103	6.135 × 107	3.282 × 106	7.050 × 103
	Rank	5	2	4	3	1
F20	Mean	9.231 × 103	6.463 × 103	6.165 × 103	5.747 × 103	4.678 × 103
	Min	5.555 × 103	2.300 × 103	4.389 × 103	2.475 × 103	2.300 × 103
	Max	1.129 × 104	9.968 × 103	1.015 × 104	9.513 × 103	9.283 × 103
	Std	1.071 × 103	1.958 × 103	1.440 × 103	1.854 × 103	3.017 × 103
	Rank	5	4	3	2	1
F21	Mean	2.697 × 103	2.520 × 103	2.408 × 103	2.388 × 103	2.403 × 103
	Min	2.583 × 103	2.411 × 103	2.373 × 103	2.341 × 103	2.319 × 103
	Max	2.838 × 103	2.650 × 103	2.523 × 103	2.436 × 103	2.469 × 103
	Std	5.997 × 101	6.193 × 101	2.773 × 101	2.040 × 101	3.968 × 101
	Rank	5	4	2	1	3
F22	Mean	9.247 × 103	6.023 × 103	6.002 × 103	6.264 × 103	3.598 × 103
	Min	6.356 × 103	2.300 × 103	2.731 × 103	2.672 × 103	2.300 × 103
	Max	1.083 × 104	1.204 × 104	1.041 × 104	9.926 × 103	9.689 × 103
	Std	1.019 × 103	2.548 × 103	1.493 × 103	2.164 × 103	2.650 × 103
	Rank	5	4	2	3	1
F23	Mean	3.363 × 103	3.479 × 103	2.792 × 103	2.757 × 103	2.708 × 103
	Min	3.098 × 103	3.095 × 103	2.726 × 103	2.701 × 103	2.674 × 103
	Max	3.869 × 103	3.882 × 103	2.946 × 103	2.895 × 103	2.767 × 103
	Std	1.838 × 102	1.772 × 102	5.363 × 101	3.990 × 101	2.849 × 101
	Rank	4	5	3	2	1
F24	Mean	3.533 × 103	3.531 × 103	2.993 × 103	2.947 × 103	2.970 × 103
	Min	3.281 × 103	3.233 × 103	2.899 × 103	2.881 × 103	2.865 × 103
	Max	3.856 × 103	3.882 × 103	3.095 × 103	3.092 × 103	3.011 × 103
	Std	1.543 × 102	1.353 × 102	5.745 × 101	5.943 × 101	3.716 × 101
	Rank	4	5	3	1	2
F25	Mean	4.003 × 103	2.915 × 103	3.005 × 103	2.960 × 103	2.888 × 103
	Min	3.581 × 103	2.884 × 103	2.930 × 103	2.906 × 103	2.883 × 103
	Max	5.042 × 103	2.948 × 103	3.220 × 103	3.044 × 103	2.910 × 103
	Std	3.285 × 102	2.514 × 101	7.924 × 101	3.339 × 101	4.344 × 100
	Rank	5	2	4	3	1
F26	Mean	1.005 × 104	6.454 × 103	4.605 × 103	4.554 × 103	4.105 × 103
	Min	7.873 × 103	2.800 × 103	4.114 × 103	4.051 × 103	3.698 × 103
	Max	1.259 × 104	1.021 × 104	5.272 × 103	5.723 × 103	4.894 × 103
	Std	1.211 × 103	2.659 × 103	3.458 × 102	3.358 × 102	2.572 × 102
	Rank	5	4	2	3	1
F27	Mean	3.833 × 103	4.218 × 103	3.200 × 103	3.200 × 103	3.211 × 103
	Min	3.410 × 103	3.644 × 103	3.200 × 103	3.200 × 103	3.201 × 103
	Max	5.619 × 103	4.903 × 103	3.200 × 103	3.200 × 103	3.221 × 103
	Std	4.177 × 102	3.348 × 102	2.205 × 10−4	3.106 × 10−4	5.428 × 100
	Rank	4	5	2	1	3
F28	Mean	5.399 × 103	3.157 × 103	3.315 × 103	3.317 × 103	3.210 × 103
	Min	4.195 × 103	3.100 × 103	3.296 × 103	3.296 × 103	3.100 × 103
	Max	6.761 × 103	3.265 × 103	3.474 × 103	3.465 × 103	3.267 × 103
	Std	6.843 × 102	6.460 × 101	4.611 × 101	4.549 × 101	3.247 × 101
	Rank	5	1	3	4	2
F29	Mean	3.507 × 103	3.498 × 103	2.987 × 103	2.963 × 103	2.953 × 103
	Min	3.197 × 103	3.312 × 103	2.878 × 103	2.877 × 103	2.860 × 103
	Max	3.771 × 103	3.695 × 103	3.111 × 103	3.074 × 103	2.992 × 103
	Std	1.371 × 102	9.597 × 101	6.437 × 101	6.320 × 101	3.351 × 101
	Rank	4	5	3	2	1
F30	Mean	4.622 × 109	1.995 × 104	1.353 × 107	3.037 × 107	8.942 × 103
	Min	6.778 × 107	7.843 × 103	3.555 × 104	1.607 × 104	5.375 × 103
	Max	7.022 × 1010	4.219 × 104	2.789 × 108	3.440 × 108	1.593 × 104
	Std	1.505 × 1010	7.370 × 103	5.074 × 107	7.612 × 107	3.023 × 103
	Rank	5	2	3	4	1
Mean rank		4.883	3.183	3.117	2.316	1.500
Final rank		5	4	3	2	1

. The reported values in Table 2 are based on the errors between the terminated values of the optimization process and the target values of the benchmark functions. To intuitively quantify the optimization ability of the metaheuristics, mean values (Mean), minimum values (Min), maximum values (Max), standard deviation (Std) and ranks (Rank) are used to evaluate the accuracy. Mean, Min and Max reveal the optimization accuracy of the algorithm, Std reveals the optimization stability and Rank is based on the Friedman test [48] to rank the optimization performance of the algorithm from the aspect of statistics. Moreover, the minimum values of Mean in each benchmark function are shown in bold.

For the unimodal and functions (i.e., F₁–F₁₀), CS-GWO achieves the best results six times, and WOA and CSO share the remaining four best results. This reveals that WOA performs well for simple low-dimensional optimization problems. For the 10 hybrid functions (i.e., F₁₁–F₂₀), CS-GWO achieves the best performance seven times. Although WOA has the best values for F₁₂ and F₁₄, it gets the second-worst values in 4 out of 10 cases. This reveals that WOA has unstable performance in solving complex problems [49,50]. For the 10 composition functions (i.e., F₂₁–F₃₀), CS-GWO achieves the best performance six times, followed by CSO’s three times. The worst rank of CSO in composition functions is three with F₂₁ and F₂₇, indicating that CSO has the ability to escape local optimum when applied in complex optimization problems [47].

Particularly, GWO never dominates in optimization of all benchmark functions, but it ranks third overall. This reveals that GWO has stable performance in solving optimization problems but easily falls into the optima trap [51]. Comprehensively speaking, CS-GWO ranks first overall among the optimization algorithms and obtains the best performance in 19 out of 30 functions, whether from the aspect of the optimization accuracy or stability.

From the above analysis, we can conclude that CS-GWO performs the best with the 30 dimensional optimization problems among all the compared algorithms. This is because CS-GWO combines the stable optimization performance of GWO and outstanding ability in finding global optima in the problem-solving space of CSO.

5.3.3. Wilcoxon Signed-Rank Test and Paired Samples t-Test

In order to further prove the validity of the CS-GWO algorithm, the parametric and non-parametric tests, which are the paired samples t-test (PSTT) [52] and Wilcoxon signed-rank test (WSRT) [53], are adopted to evaluate the difference of optimization performance between CS-GWO and the comparison algorithms in 30 benchmark functions. The null hypothesis of PSTT and WSRT are that there is no difference between two compared samples. If the difference approximately obeyed the normal distribution, PSTT is used. When this premise is not satisfied, WSRT can be selected. The results of PSTT and WSRT are shown in

Table 3. The paired samples t-test results.

Functions	CS-GWO–PSO		CS-GWO–WOA		CS-GWO–GWO		CS-GWO–CSO
	t-Value	Sig. (2-Tailed)	t-Value	Sig. (2-Tailed)	t-Value	Sig. (2-Tailed)	t-Value	Sig. (2-Tailed)
F1	−1.898 × 101	6.748 × 10−18	2.413 × 100	2.236 × 10−2	−8.712 × 100	1.367 × 10−9	−9.427 × 100	2.472 × 10−10
F2	−3.355 × 101	9.349 × 10−25	−1.603 × 101	6.012 × 10−16	−7.118 × 100	7.828 × 10−8	−7.771 × 100	1.434 × 10−8
F3	−3.733 × 101	4.521 × 10−26	−1.447 × 101	8.498 × 10−15	−2.535 × 100	1.688 × 10−2	−4.732 × 100	5.343 × 10−5
F4	−1.361 × 101	4.016 × 10−14	5.103 × 100	1.907 × 10−5	−6.128 × 100	1.122 × 10−6	−5.836 × 100	2.499 × 10−6
F5	−2.913 × 101	5.005 × 10−23	−1.207 × 101	7.880 × 10−13	−2.016 × 100	5.320 × 10−2	−4.119 × 100	2.892 × 10−4
F6	−4.482 × 101	2.469 × 10−28	−2.925 × 101	4.471 × 10−23	−1.638 × 101	3.381 × 10−16	−1.711 × 101	1.078 × 10−16
F7	−4.000 × 101	6.363 × 10−27	−1.717 × 101	9.852 × 10−17	−1.211 × 100	2.358 × 10−1	−1.954 × 100	6.035 × 10−2
F8	−1.977 × 101	2.240 × 10−18	−6.441 × 100	4.791 × 10−7	−2.143 × 100	4.061 × 10−2	−3.119 × 100	4.073 × 10−3
F9	−2.915 × 101	4.917 × 10−23	−1.070 × 101	1.390 × 10−11	−8.638 × 100	1.639 × 10−9	−9.730 × 100	1.224 × 10−10
F10	−4.268 × 100	1.926 × 10−4	3.982 × 100	4.199 × 10−4	1.263 × 101	2.564 × 10−13	7.920 × 100	9.815 × 10−9
F11	−1.744 × 101	6.491 × 10−17	−6.545 × 100	3.613 × 10−7	−4.355 × 100	1.517 × 10−4	−5.824 × 100	2.584 × 10−6
F12	−3.827 × 100	6.381 × 10−4	2.057 × 100	4.877 × 10−2	−3.849 × 100	6.013 × 10−4	−5.683 × 100	3.811 × 10−6
F13	−1.855 × 100	7.384 × 10−2	−1.045 × 100	3.045 × 10−1	−1.014 × 100	3.188 × 10−1	−2.002 × 100	5.470 × 10−2
F14	−2.698 × 100	1.150 × 10−2	2.856 × 100	7.859 × 10−3	−2.508 × 100	1.800 × 10−2	−4.504 × 100	1.004 × 10−4
F15	−2.870 × 100	7.579 × 10−3	−2.481 × 100	1.915 × 10−2	−1.445 × 100	1.592 × 10−1	−2.437 × 100	2.118 × 10−2
F16	−6.619 × 100	2.964 × 10−7	−5.076 × 100	2.056 × 10−5	−3.658 × 10−1	7.172 × 10−1	−1.120 × 100	2.720 × 10−1
F17	−1.250 × 101	3.371 × 10−13	−9.608 × 100	1.622 × 10−10	−2.962 × 100	6.043 × 10−3	−4.194 × 100	2.354 × 10−4
F18	−1.801 × 101	2.752 × 10−17	−1.777 × 101	3.930 × 10−17	−4.058 × 100	3.412 × 10−4	−4.287 × 100	1.828 × 10−4
F19	−1.886 × 100	6.940 × 10−2	−1.182 × 100	2.468 × 10−1	−4.709 × 100	5.685 × 10−5	−1.200 × 100	2.399 × 10−1
F20	−7.822 × 100	1.258 × 10−8	−2.627 × 100	1.363 × 10−2	−1.677 × 100	1.042 × 10−1	−2.327 × 100	2.715 × 10−2
F21	−2.256 × 101	6.096 × 10−20	−8.681 × 100	1.476 × 10−9	1.939 × 100	6.226 × 10−2	−6.014 × 10−1	5.523 × 10−1
F22	−1.092 × 101	8.593 × 10−12	−3.382 × 100	2.076 × 10−3	−4.182 × 100	2.434 × 10−4	−4.385 × 100	1.395 × 10−4
F23	−1.887 × 101	7.868 × 10−18	−2.395 × 101	1.174 × 10−20	−5.113 × 100	1.853 × 10−5	−7.663 × 100	1.895 × 10−8
F24	−1.967 × 101	2.579 × 10−18	−2.128 × 101	3.048 × 10−19	2.003 × 100	5.460 × 10−2	−2.219 × 100	3.446 × 10−2
F25	−1.859 × 101	1.176 × 10−17	−5.979 × 100	1.688 × 10−6	−1.154 × 101	2.322 × 10−12	−8.043 × 100	7.194 × 10−9
F26	−2.662 × 101	6.231 × 10−22	−4.748 × 100	5.111 × 10−5	−6.544 × 100	3.618 × 10−7	−6.443 × 100	4.764 × 10−7
F27	−8.182 × 100	5.069 × 10−9	−1.640 × 101	3.285 × 10−16	1.129 × 101	3.900 × 10−12	1.129 × 101	3.902 × 10−12
F28	−1.764 × 101	4.810 × 10−17	4.099 × 100	3.058 × 10−4	−1.060 × 101	1.737 × 10−11	−9.310 × 100	3.254 × 10−10
F29	−2.047 × 101	8.731 × 10−19	−2.666 × 101	6.021 × 10−22	−6.827 × 10−1	5.002 × 10−1	−2.368 × 100	2.476 × 10−2
F30	−1.682 × 100	1.033 × 10−1	−7.369 × 100	4.050 × 10−8	−2.184 × 100	3.718 × 10−2	−1.460 × 100	1.550 × 10−1

and

Table 4. Wilcoxon signed-rank test results of CEC 2017.

Functions	CS-GWO vs. PSO				CS-GWO vs. WOA
	p-Value	R+	R−	Winner	p-Value	R+	R−	Winner
F1	1.734 × 10−6	0	465	+	6.564 × 10−2	322	143	=
F2	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F3	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F4	1.734 × 10−6	0	465	+	2.163 × 10−5	439	26	−
F5	1.734 × 10−6	0	465	+	1.921 × 10−6	1	464	+
F6	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F7	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F8	1.734 × 10−6	0	465	+	2.843 × 10−5	29	436	+
F9	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F10	5.287 × 10−4	64	401	+	9.627 × 10−4	393	72	−
F11	1.734 × 10−6	0	465	+	1.238 × 10−5	20	445	+
F12	1.734 × 10−6	0	465	+	8.972 × 10−2	315	150	+
F13	1.734 × 10−6	0	465	+	1.589 × 10−1	164	301	=
F14	3.182 × 10−6	6	459	+	1.150 × 10−4	420	45	−
F15	1.734 × 10−6	0	465	+	3.609 × 10−3	91	374	+
F16	1.734 × 10−6	0	465	+	5.307 × 10−5	36	429	+
F17	1.734 × 10−6	0	465	+	2.879 × 10−6	5	460	+
F18	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F19	1.734 × 10−6	0	465	+	1.470 × 10−1	162	303	=
F20	5.216 × 10−6	11	454	+	1.480 × 10−2	114	351	+
F21	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F22	2.353 × 10−6	3	462	+	3.379 × 10−3	90	375	+
F23	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F24	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F25	1.734 × 10−6	0	465	+	1.359 × 10−4	47	418	+
F26	1.734 × 10−6	0	465	+	2.613 × 10−4	55	410	+
F27	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F28	1.734 × 10−6	0	465	+	6.639 × 10−4	398	67	−
F29	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F30	1.734 × 10−6	0	465	+	4.286 × 10−6	9	456	+
+/=/−				30/0/0				23/3/4
Functions	CS-GWO vs. GWO				CS-GWO vs. CSO
	p-Value	R+	R−	winner	p-Value	R+	R−	winner
F1	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F2	3.182 × 10−6	6	459	+	1.127 × 10−5	19	446	+
F3	1.359 × 10−4	47	418	+	3.327 × 10−2	129	336	+
F4	1.921 × 10−6	1	464	+	3.882 × 10−6	8	457	+
F5	4.196 × 10−4	61	404	+	7.190 × 10−2	145	320	=
F6	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F7	1.470 × 10−1	162	303	=	4.779 × 10−1	198	267	=
F8	8.217 × 10−3	104	361	+	3.872 × 10−2	132	333	+
F9	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F10	1.238 × 10−5	445	20	−	2.879 × 10−6	460	5	−
F11	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F12	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F13	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F14	6.892 × 10−5	39	426	+	7.865 × 10−2	147	318	=
F15	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F16	3.709 × 10−1	189	276	=	7.813 × 10−1	219	246	=
F17	3.065 × 10−4	57	408	+	1.319 × 10−2	112	353	+
F18	6.156 × 10−4	66	399	+	3.589 × 10−4	59	406	+
F19	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F20	2.564 × 10−2	124	341	+	1.986 × 10−1	170	295	=
F21	7.971 × 10−1	245	220	=	4.950 × 10−2	328	137	−
F22	6.156 × 10−4	66	399	+	1.484 × 10−3	78	387	+
F23	2.879 × 10−6	5	460	+	3.405 × 10−5	31	434	+
F24	5.984 × 10−2	141	324	=	3.872 × 10−2	333	132	−
F25	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F26	3.112 × 10−5	30	435	+	4.286 × 10−6	9	456	+
F27	1.734 × 10−6	465	0	−	1.734 × 10−6	465	0	−
F28	1.734 × 10−6	0	465	+	1.734 × 10−6	0	465	+
F29	7.190 × 10−2	145	320	=	8.130 × 10−1	221	244	=
F30	1.734 × 10−6	465	0	−	1.734 × 10−6	465	0	−
+/=/−				22/5/3	+/=/−			19/6/5

, respectively.

Two indicators including t-value and Sig. (2-tailed) can be obtained from PSTT. If the Sig. (2-tailed) is less than 0.05, it can be concluded that the CS-GWO algorithm is better than the compared algorithm. In addition, four indicators including p-value, R+, R− and winner can be obtained from WSRT. If the p-value is less than 0.05, the null hypothesis can be rejected at 5% significance level. R+ represents a mean error of the CS-GWO algorithm that is higher than that of the compared one. R− represents a mean error of the CS-GWO algorithm that is lower than that of the compared one. Finally, winner indicates whether the CS-GWO algorithm is superior to the compared algorithm, “+” indicates that the CS-GWO algorithm is better than the compared algorithm, “−” indicates that the CS-GWO algorithm is worse than the compared algorithm and “=” indicates that the performance of the two algorithms display no obvious difference.

It can be seen that most of the Sig. (2-tailed) values in Table 3 are less than 0.05, which means that there is an obvious difference between the CS-GWO algorithm and the comparison algorithms, further indicating that the CS-GWO algorithm is superior to the involved algorithms. In Table 4, most of the combinations (p-Value, R+, R, winner) are (1.734 × 10⁻⁶, 0, 465, +), revealing that the CS-GWO algorithm outperforms the comparison algorithms. Furthermore, the results of ‘+/=/−’ is 94/14/12, indicating that the CS-GWO algorithm is better than the compared algorithm in 94 out of 120 cases.

From a statistical perspective, it can be concluded that CS-GWO dominates the other compared algorithms in optimization problems with CEC 2017 test functions.

5.4. Case 3: The Effectiveness of the CS-GWO-DA-BiGRU Model

In order to verify the effectiveness of the combination of the crisscross grey wolf optimization algorithm and the DA-BiGRU model in short-term load forecasting, PSO-DA-BiGRU, WOA-DA-BiGRU and GWO-DA-BiGRU models are compared with the combined CS-GWO-DA-BiGRU model in this case. In this subsection, the number of iterations of the swarm intelligence optimization algorithm is uniformly set to 200, and the number of individuals is set to 20.

The experimental results are shown in

Table 5. The experiment results of case study 2.

Prediction Model	RMSE/MW	MAE/MW	SMAPE	R2
DA-BiGRU	29.053	22.897	3.566	0.937
PSO-DA-BiGRU	28.546	22.285	3.519	0.939
WOA-DA-BiGRU	28.209	22.221	3.471	0.941
GWO-DA-BiGRU	27.895	22.162	3.545	0.942
CSO-DA-BiGRU	27.194	21.255	3.347	0.945
CS-GWO-DA-BiGRU	26.144	20.963	3.337	0.949

and Figure 8, where Figure 8 is a comparison chart between the prediction results of different models and the real values on 29–31 December 2018.

As shown in Table 2 and Figure 8, the following conclusions can be drawn:

(1) The effectiveness of the swarm intelligence optimization algorithm:

The prediction model combined with the swarm intelligence optimization algorithm has better prediction performance than the single prediction model (i.e., DA-BiGRU). For example, compared with the DA-BiGRU model, the RMSE, MAE and SMAPE values of the PSO-DA-BiGRU and GWO-DA-BiGRU models are reduced by 1.75% and 3.99%, 2.67% and 3.21% and 1.32% and 0.59%, respectively, and the R² values are increased by 0.21% and 0.53%, respectively. This is because the weights and the bias of the DA-BiGRU model are optimized by the swarm intelligence optimization algorithm in the initial stage of training, which can effectively avoid the problems of gradient disappearance and gradient explosion, and further improve the accuracy of load forecasting.

(2) The superiority of the CS-GWO algorithm:

Among all the comparison forecasting models, the proposed CS-GWO-DA-BiGRU short-term load forecasting model has the highest forecasting accuracy. For example, the RMSE, MAE and SMAPE are reduced by 3.86%, 1.37% and 0.30% from those of the second-best performing CSO-DA-BiGRU model, respectively. From the aspect of the R² value, the CS-GWO-DA-BiGRU model has an increase of 0.42% compared with the CSO-DA-BiGRU model. Therefore, the CS-GWO algorithm combined with horizontal crossover and vertical crossover operators can improve the global search ability and enhance the diversity of the population, making a great contribution to improving short-term load forecasting.

6. Discussion

In this paper, a high-precision model called CS-GWO-DA-BiGRU is presented in short-term load forecasting problems. However, the proposed model still has some shortcomings that need to be improved. The limitations and future research can be summarized as follows.

(1) The CS-GWO algorithm only focuses on improving the accuracy of short-term load prediction while ignoring the prediction stability, leading to unstable prediction when extending to new data. In the future, we plan to upgrade CS-GWO to a multi-object CS-GWO algorithm to improve the accuracy and stability of short-term load prediction simultaneously.

(2) At present, the intelligent big data platform is valuable for the improvement of the prediction model. In future work, the proposed CS-GWO-DA-BiGRU prediction model will be embedded into the intelligent big data platform to construct an intelligent load forecasting system.

7. Conclusions

Short-term load prediction is essential for the stable operation and safety management of power systems. Therefore, this paper proposes a hybrid model for short-term load prediction, named CS-GWO-DA-BiGRU, which consists of a dual-stage attention mechanism, crisscross grey wolf optimization algorithm and bidirectional gated recurrent unit. The main contributions of this paper can be concluded as follows:

(1) Different from the conventional feature mechanism applied in short-term load forecasting, this paper proposes a dual-stage attention mechanism by combining feature and temporal attention mechanisms. Based on case 1, compared with FA-BiGRU, the RMSE, MAE and SMAPE values of the DA-BiGRU model are reduced by 1.79%, 0.74% and 0.70%, respectively, and the R² value is increased by 0.21%. Therefore, DA can effectively capture the correlation relationship of input feature and temporal dependence in load time series simultaneously.

(2) By combining horizontal and vertical crossover operators, the global search ability and population diversity of GWO are enhanced. Based on the Friedman test in case 2, CS-GWO ranks first among the well-known algorithms and achieves the best results for 19 out of 30 functions in CEC 2017. In addition, CS-GWO outperforms the compared algorithms in 94 out of 120 cases based on the Wilcoxon signed-rank test. Furthermore, for the proposed CS-GWO-DA-BiGRU model in case 3, which is based on CS-GWO, the R² value has an increase of 0.42% compared with the CSO-DA-BiGRU model and has the best forecasting performance.