Forecasting of Short-Term Load Using the MFF-SAM-GCN Model

Abstract

Short-term load forecasting plays a significant role in the operation of power systems. Recently, deep learning has been generally employed in short-term load forecasting, primarily in the extraction of the characteristics of digital information in a single dimension without taking into account of the impact of external variables, particularly non-digital elements on load characteristics. In this paper, we propose a joint MFF-SAM-GCN to realize short-term load forecasting. First, we utilize a Bi-directional Long Short-Term Memory (Bi-LSTM) network and One-Dimensional Convolutional Neural Network (1D-CNN) in parallel connection to form a multi-feature fusion (MFF) framework, which can extract spatiotemporal correlation features of the load data. In addition, we introduce a Self-Attention Mechanism (SAM) to further enhance the feature extraction capability of the 1D-CNN network. Then with the deployment of a Graph Convolutional Network (GCN), the external non-digital features such as weather, strength, and direction of wind, etc., are extracted. Moreover, the generated weight matrices are incorporated into the load features to enhance feature recognition ability. Finally, we exploit Bayesian Optimization (BO) to find the optimal hyperparameters of the model to further improve the prediction accuracy. The simulation is taken from our proposed model and six benchmark schemes by using the bus load dataset of the Shandong Open Data Network, China. The results show that the RMSE of our proposed MFF-SAM-GCN model is 0.0284, while the SMAPE is 9.453%,the MBE is 0.025, and R-squared is 0.989, which is better than the selected three traditional machine learning methods and the three deep learning models.

1. Introduction

In 2021, China proposed the construction of a new power system with new energy as the primary supply, which ensures energy and power security as its fundamental premise and sets the power needs of economic and social development as its primary goal. Supported by the interaction between storage and multi-energy complementarity, the new power system has the characteristics of being clean, low-carbon, open, and interactive. Taking a strong smart grid as the hub platform, it is also controllable, efficient, intelligent, and flexible [1]. The new power system must carry out real-time monitoring and precise regulation of the entire power generation, transmission, distribution, and consumption chain. However, power load prediction is not only the basis of the new power system’s flexibility, efficiency, and open-accessibility, but also the basis for formulating power grid construction planning, and optimizing Demand Side Response (DSR). It is the foundation for users to improve energy efficiency, which has an important impact on improving quality and efficiency for the department of power production.

Based on various time scales, electricity load forecasting may be split into three categories [2,3,4]: short-term load forecasting (SLF), medium-term load forecasting (MLF), and long-term load forecasting (LLF). SLF refers to the forecast of power load in the next few days, hours, or even minutes. MLF is the forecasting of power load in the period of one month to one quarter, and LLF is the forecasting of power over more than one year. SLF gives an accurate insight into the consumption behaviors of energy customers and enables better flexibility in demand-side management, which is essential for economical operation and short-term maintenance planning. Furthermore, aggregating short-term load projections provides the basis for medium and long-term load forecasting.

The main features used for load forecasting include historical load data, economic indicators of various industries in the whole society, weather (temperature, humidity, wind, and rainfall, etc.), and electricity price policies, etc. Due to the fact that people in different regions and industries have various power consumption modes, SLF is more challengable than MLF and LLF. The SLF approaches are now divided into two categories, which are the classical techniques and the machine learning methods. The first category includes time series analysis [5], multiple linear regression [6], and Kalman filtering [7], etc. Although these models are easy to conduct, it is difficult to extract complex features of energy consumption patterns from them, and their robustness is poor. The other is the machine learning based methods, such as clustering [8], decision trees [9], random forests [10], support vector machines [11], fuzzy logic [12], and neural networks [13], etc. Owing to its superior feature extraction capabilities, deep learning has been one of the most appealing approaches for SLF in recent years. Recurrent Neural Networks (RNN) [14] are the most commonly used time series prediction models in deep learning, accomplishing series prediction by examining the temporal correlation between time series. Long Short Term Memory (LSTM) networks [15] are one of the most effective implementations of RNNs, solving the problem of long-term dependence of time series and are the most commonly used model in SLF. Convolutional Neural Networks (CNN) [16] are another popular deep learning model effective at extracting spatial data. One-Dimensional Convolutional Neural Networks (1D-CNN) [17] can identify basic patterns in time series data and collect local information, which can be used with RNN to boost prediction accuracy even further. One of the current hot issues in time series research is selecting and extracting distinct features and boosting the prediction potential of the model via feature fusion [18]. External characteristics such as geography, industry, economics, and weather may boost the feature richness and forecasting capabilities of the model in SLF. External features, on the other hand, are usually irregular, and CNN cannot handle the unstructured input and has limited capacity to extract irregular features. Thus, the Graph Convolutional Network (GCN) [19] was proposed. Not only can a GCN recover irregular features, but it can also capture the association between distinct characteristics.

In this paper, we offer a model for SLF that combines with multi-feature fusion (MFF), the Self-Attention Mechanism (SAM), and GCN. To extract the spatiotemporal correlation features of load data, our proposed model employs 1D-CNN and bi-directional Long Short-Term Memory (Bi-LSTM) networks in parallel to form an MFF framework. Then we introduce an SAM to improve the feature extraction capability of 1D-CNN. In addition, GCN is used to extract external non-numerical features such as weather, wind force, and direction. Moreover, the generated weight matrices are incorporated into the load features to enhance feature recognition ability. Finally, we exploit the Bayesian Optimization (BO) to find the optimal hyperparameters of the model to further improve the prediction accuracy. The following are the primary contributions of this study.

(1)

An MFF and GCN based SLF model is suggested, in which MFF is used to extract spatiotemporal correlation characteristics of load data and GCN is to elicit external non-numerical information.

(2)

A novel MFF structure is developed, which employs the 1D-CNN based on the SAM to extract spatially relevant load data features. Moreover, the use of Bi-LSTM network can help to extract temporally relevant load data features.

(3)

A BO algorithm is used to find the optimal model parameters, and the proposed prediction model is experimentally compared with currently popular prediction models to investigate the role of each module and the effect of hyperparameter settings and busbars on prediction performance via ablation experiments.

The rest of the paper is structured as follows: Section 2 presents current SLF methodologies, Section 3 describes the MFF-SAM-GCN framework presented in this research, and Section 4 is dedicated to experimentation and analysis. The suggested framework is contrasted with frequently used approaches in the experiments, and ablation experiments are performed; Section 5 is the conclusion, which summarizes this paper and suggests the direction of future research.

2. Related Work

SLF based on machine learning has been extensively studied. The European Network for Intelligent Technologies (EUNITE) held a worldwide competition on power demand forecasting in 2001, in which Chen et al. [20] used Support Vector Regression (SVR) technique to win the event; Rafi et al. [21] proposed a SLF method based on CNN and LSTM networks for SLF in Bangladesh’s national electricity system; Goh et al. [22] proposed a hybrid method combining multiple independent 1D-CNN and LSTM networks for SLF in the Irish national grid; Farsi et al. [23] proposed a hybrid parallel CNN-LSTM network model, which consists of two paths, CNN and LSTM, and the outputs of the two paths are combined to form a fully connected path as the final output, using the hourly load dataset in Malaysia and the daily load dataset in Germany to forecast the model; Wang et al. [24] proposed an SLF model based on time convolution network (TCN) and LightGBM algorithm, and validated the model using datasets from different industries in China, Australia, and Ireland.

Because of its unique capacity to extract significant information, the Attention Mechanism (AM) has been integrated with deep learning and has been employed in a variety of industries. For SLF, Fahim et al. [25] designed a hybrid predictive model using a LSTM network combined with a SAM for monitoring educational reform in Morocco; Miao et al. [26] suggested a hybrid model of CNN-Bi-LSTM network coupled with BO and AM; Azam et al. [27] proposed a hybrid deep learning architecture based on Bi-LSTM and interpretable multi-headed attention mechanism, combining deep learning with Ensemble Empirical Mode Decomposition (EEMD) algorithm, which was evaluated using ISO-NE load and price dataset; Zang et al. [28] combined LSTM networks and SAM to design a two-channel hybrid model that combined feature engineering to pre-process the raw data and tested it using residential customer electricity load datasets from different countries and regions; Shang et al. [29] suggested a Multivariate Multistep CNN-LSTM (MMCNN-LSTM) hybrid model estimate 24-h power demands using Particle Swarm Optimization (PSO) and Kernel Fuzzy C Means (KFCM), and the model’s performance was validated by three prediction trials.

The GCN has unique advantages in extracting spatial features and has been applied to traffic flow prediction. Zhao et al. [30] proposed a traffic flow prediction method using GCN and Gated Recurrent Unit (GRU) to form a Temporal Graph Convolutional Network (T-GCN) model, which was validated on an SZ-taxi dataset and Los-loop dataset; Wang et al. [31] created a global relational reasoning GCN model for human posture assessment and included it into the early phases of HRNet and SimpleBaseline. The findings suggest that the model can successfully learn key point correlations.

3. Method

3.1. Overall Network

The framework of MFF-SAM-GCN prediction model is shown at Figure 1. The input of the model adopts both numerical and textual features. The basic module is a MFF framework consisting of Bi-LSTM and 1D-CNN, with Bi-LSTM extracting time-related load data features and 1D-CNN extracting spatially related load data features. The two models are fused to obtain multi-dimensional features. SAM is applied to the 1D-CNN to improve the efficiency of feature extraction by increasing the significance of target features and reducing irrelevant detail information. Furthermore, GCN is employed as a feature improvement module to collect external non-numerical characteristics such as weather, wind force, and wind direction, which are then reorganized with global information.

3.2. Multi-Feature Fusion Framework

The commonly used deep learning models for data sequence feature extraction are LSTM and CNN, where LSTM is a special type of RNN with time series as input and a cyclic structure that can learn long-term dependency information between sequences, and CNN can extract spatial features that can recognize simple patterns of sequences and use these simple patterns to generate complex patterns at a high level. 1D-CNN can extract effective features directly. In order to obtain rich dimensional information, this paper uses one Bi-LSTM module and two 1D-CNN modules in parallel to form a MFF framework to extract spatiotemporally correlated features of load data in parallel and adds SAM between the two 1D-CNNs to enhance the feature extraction capability of the 1D-CNN.

The load data matrix is represented in the MFF framework by $X\in {\mathrm{R}}^{N\times C}$, where N is the number of data collection points obtained for a specific period (e.g., hours, days), C is the number of periods. $x^t=\left[x_1^t,\cdots ,x_C^t\right],t=1,\cdots N$ is the load data vector for the same collection points of different periods, and $x_i=[x_i^1,\cdots ,x_i^N],i=1,\cdots ,C$ is the load data vector for different collection points for the same period.

3.2.1. Bi-Directional Long Short-Term Memory Network

The LSTM network is a special kind of RNN that substitutes the RNN’s implicit function with a memory neuron, which can recall more information than an RNN alone and is appropriate for processing sequences with longer gaps or delays. Figure 2 [32] depicts the structure of the memory neuron structure in the LSTM network, which primarily consists of the forget gate, input gate, information update, and output gate, where ${\mathit{x}}^t,{\widehat{\mathit{y}}}^t$ denote the input and output information at moment t, respectively, which $s^t$ represents the state vector, which is utilized by the LSTM network to recall the history information; ${\tilde{\mathit{s}}}^t$ represents the candidate vector, which is used to update the state vector, ${\mathit{h}}^t$ sames as output message ${\widehat{\mathit{y}}}^t$, and it is delivered to the next memory neuron as the activation vector. ${\mathit{o}}_f^t,{\mathit{o}}_u^t,{\mathit{o}}_o^t$ denote the forget gate, input gate, output gate, respectively; $\sigma$ and $tanh$ are both activation operations, $\sigma$ designates the sigmoid function, $\mathrm{sigmoid}\left(x\right)=1/\left(1+e^{-x}\right)$ and $tanh$ denotes the hyperbolic tangent function, $tanh\left(x\right)=(e^x-e^{-x}\left)/\right(e^x+e^{-x})$.

(1)

Forget gate: The forget gate utilizes the sigmoid function to pick some of the information from the output ${\mathit{h}}^{t-1}$ at the moment of $t-1$ and the input $x^t$ at the moment of t. The output of the forgotten gate is given by:

$${\mathit{o}}_{\mathit{f}}^{\mathit{t}}=\mathrm{sigmoid}({\mathit{w}}_{\mathit{f}}[{\mathit{h}}^{\mathit{t}-\mathbf{1}},{\mathit{x}}^{\mathit{t}}]+{\mathit{b}}_{\mathit{f}})$$

(1)

(2)

Input gate: The $sigmoid$ and $tanh$ functions are used to choose certain information from the output $h^{t-1}$ at the moment of $t-1$ and the input $x^t$ at the moment of t, respectively. Two outputs of input gate are:

$$\begin{array}{c}\hfill {\mathit{o}}_{\mathit{u}}^{\mathit{t}}=\mathrm{sigmoid}({\mathit{w}}_{\mathit{u}}[{\mathit{h}}^{\mathit{t}-\mathbf{1}},{\mathit{x}}^{\mathit{t}}]+{\mathit{b}}_{\mathit{u}})\\ \hfill {\tilde{\mathit{s}}}^{\mathit{t}}=tanh({\mathit{w}}_{\mathit{s}}[{\mathit{h}}^{\mathit{t}-\mathbf{1}},{\mathit{x}}^{\mathit{t}}]+{\mathit{b}}_{\mathit{s}})\end{array}$$

(2)

(3)

Information update: The output $o_f^t$ of the forget gate at the moment of t is multiplied by the state vector $s^{t-1}$ at the moment of $t-1$ and added to the product of the two output values of the input gate. The updated information is represented as:

$${\mathit{s}}^{\mathit{t}}={\mathit{o}}_{\mathit{f}}^{\mathit{t}}\odot {\mathit{s}}^{\mathit{t}-\mathbf{1}}+{\mathit{o}}_{\mathit{u}}^{\mathit{t}}\odot {\tilde{\mathit{s}}}^{\mathit{t}}$$

(3)

where ⊙ denotes the Hadamard product operation, which is defined as the product of two vectors with the same number of elements at the same position.

(4)

Output gate: on the one hand, it utilizes the $sigmoid$ function to pick some information from the output ${\mathit{h}}^{t-1}$ at the moment of $t-1$ and the input ${\mathit{x}}^t$ at the moment of t. Use the $tanh$ function, on the other hand, to filter the updated state vector ${\mathit{s}}^t$ selection and then multiply the two to obtain the output of the present instant that is:

$$\begin{array}{c}\hfill {\mathit{o}}_o^t=sigmoid({\mathit{w}}_o[{\mathit{h}}^{t-1},{\mathit{x}}^{\mathit{t}}]+{\mathit{b}}_o)\\ \hfill {\mathit{h}}^t={\mathit{o}}_o^t\odot tanh\left({\mathit{s}}^t\right)\end{array}$$

(4)

In the above description, the calculated weight matrix ${\mathit{w}}_f,{\mathit{w}}_u,{\mathit{w}}_o,{\mathit{w}}_s$, and bias vector ${\mathit{b}}_u,{\mathit{b}}_o,{\mathit{b}}_s$, as described previously, are the parameters that the LSTM network must learn during the training phase.

Although they can manage and anticipate events with large gaps and delays in a time series, LSTM networks can only utilize the past data. The Bi-LSTM network is made up of two LSTM networks with opposing time orientations, one starting at the beginning and ending at the end of the time series and the other starting at the end and ending at the beginning. The Bi-LSTM network has much higher accuracy than the LSTM network.

Figure 3 [33] depicts the Bi-LSTM network model, where $\mathit{x}=[x^1,\cdots ,x^N],\widehat{\mathit{y}}=[{\widehat{y}}^1,\cdots ,{\widehat{y}}^N]$ represents the input and output information. $\overrightarrow{\mathit{h}}$ and $\overleftarrow{\mathit{h}}$ denote the output vectors of the forward and reverse LSTM networks respectively. The Bi-LSTM network model may be expressed as follows:

$$\begin{array}{c}\hfill \overrightarrow{\mathit{h}}=f(\overrightarrow{\mathit{w}}\left[{\left(\overrightarrow{\mathit{h}}\right)}^{pre},\mathit{x}\right]+\overrightarrow{\mathit{b}})\\ \hfill \overleftarrow{\mathit{h}}=f(\overleftarrow{\mathit{w}}\left[{\left(\overleftarrow{\mathit{h}}\right)}^{next},\mathit{x}\right]+\overleftarrow{\mathit{b}})\\ \hfill \widehat{\mathit{y}}=F({\mathit{w}}_y\left[\overrightarrow{\mathit{h}},\overleftarrow{\mathit{h}}\right]+{\mathit{b}}_y)\end{array}$$

(5)

where $f\left(·\right)$ denotes the activation function that incorporates the operations of Equations (1)–(4), $F\left(·\right)$ denotes the activation function that performs weighted splicing on the outputs $\overrightarrow{\mathit{h}}$ and $\overleftarrow{\mathit{h}}$ of the forward and inverse LSTM networks, ${\mathit{w}}_y,{\mathit{b}}_y$ are the weight matrices and bias vectors for calculating the forward LSTM network’s output $\widehat{\mathit{y}}$, respectively, and $\overrightarrow{\mathit{w}},\overrightarrow{\mathit{b}}$ are the weight matrices and bias vectors for generating the forward LSTM network’s output $\overrightarrow{\mathit{h}}$, and ${\left(\overrightarrow{\mathit{h}}\right)}^{\mathrm{pre}}$ is the output of the previous memory neuron in the inverse LSTM network’s output. ${\left(\overrightarrow{\mathit{h}}\right)}^{\mathrm{next}}$ is the output of the previous memory neuron in the forward LSTM network, which is set to 0 if the present memory neuron is the first, and is the output of the latter memory neuron in the reverse LSTM network, which is set to 0 if the present memory neuron is the last.

3.2.2. One-Dimensional Convolutional Neural Network

The 1D-CNN extracts local characteristics of data sequences using a certain size window segmentation, which recovers spatially meaningful features from the load data. Two 1D-CNN cascades are created in this research, and the SAM is introduced between them to improve the 1D- CNN’s extraction capacity. The result ${\widehat{\mathit{x}}}_{\mathit{i}}$ of the first 1D-CNN on the load data ${\mathit{x}}_{\mathit{i}}$ is:

$${\widehat{\mathit{x}}}_{\mathit{i}}=\sigma ({\mathit{w}}_{\mathit{i}}\otimes {\mathit{x}}_{\mathit{i}}+{\mathit{b}}_{\mathit{i}})$$

(6)

where ⊗ represents the convolution procedure, $\sigma \left(·\right)$ denotes the activation function, ${\mathit{w}}_{\mathit{i}},{\mathit{b}}_{\mathit{i}}$ are the weight matrix and bias vector of ${\widehat{\mathit{x}}}_{\mathit{i}}$, respectively.

3.2.3. Self-Attention Mechanism

The AM improves the convolutional kernel’s perceptual field, while the SAM is a version [34] of the AM that lowers reliance on external input and is better at capturing the internal importance of data or features. The SAM is employed in this paper to improve the capture of target feature dependencies, and its implementation framework is presented in Figure 4.

The primary premise of the SAM is to identify the most significant aspects and improve them by calculating data scores at various places in the data stream. Three one-dimensional convolutional extraction load data features are employed in this study, labeled as $\mathit{a}\left(\mathit{x}\right),\mathit{b}\left(\mathit{x}\right),\mathit{c}\left(\mathit{x}\right)$.

$$\begin{array}{c}\hfill a\left({\widehat{\mathit{x}}}_{\mathit{i}}\right)={\mathit{w}}_{\mathit{a}}\otimes {\widehat{\mathit{x}}}_{\mathit{i}}\\ \hfill b\left({\widehat{\mathit{x}}}_{\mathit{i}}\right)={\mathit{w}}_{\mathit{b}}\otimes {\widehat{\mathit{x}}}_{\mathit{i}}\\ \hfill c\left({\widehat{\mathit{x}}}_{\mathit{i}}\right)={\mathit{w}}_{\mathit{c}}\otimes {\widehat{\mathit{x}}}_{\mathit{i}}\end{array}$$

(7)

where ${\mathit{w}}_{\mathit{a}},{\mathit{w}}_{\mathit{b}},{\mathit{w}}_{\mathit{c}}$ are the weight vectors for each of the three one-dimensional convolutions.

The softmax weights ${\alpha }_i^{mn},m,n=1,\cdots ,N$ between $a\left(x\right),b\left(x\right)$ has been calculated while using ${\widehat{\mathit{x}}}_i^m,{\widehat{\mathit{x}}}_i^n$ to represent the two data points in the output sequence ${\widehat{\mathit{x}}}_{\mathit{i}}$ after the first 1D-CNN.

$${\alpha }_{\mathit{i}}^{\mathit{mn}}=\mathrm{soft}max{\left(\right(a\left({\widehat{\mathit{x}}}_{\mathit{i}}^{\mathit{m}}\right))}^T\times b\left({\widehat{\mathit{x}}}_{\mathit{i}}^{\mathit{n}}\right))$$

(8)

where $\mathit{c}\left(\mathit{x}\right)$ and ${\alpha }_{\mathit{i}}^{\mathit{mn}},m,n=1,\cdots ,N$ are used to calculate self-attention scores for data ${\widehat{\mathit{x}}}_i^m$.

$${\beta }_{\mathit{i}}^{\mathit{m}}=\sum _{n=1}^N{\alpha }_{\mathit{i}}^{\mathit{mn}}c\left({\widehat{\mathit{x}}}_i^n\right)$$

(9)

The first 1D-CNN output sequence ${\widehat{\mathit{x}}}_i^m$ is concatenated with the weighted residuals of the SAM score, and the result is utilized as the input of the second 1D-CNN, i.e.:

$${\tilde{\mathit{x}}}_{\mathit{i}}={[{\tilde{\mathit{x}}}_{\mathit{i}}^{\mathit{T}}+\gamma {\beta }_{\mathit{i}}^{\mathit{T}}]}^T$$

(10)

where $\gamma$ is the trainable weight coefficient and ${\beta }_{\mathit{i}}=[{\beta }_i^1,\cdots ,{\beta }_i^N]$ is the SAM score vector for each data point. The output ${\widehat{\mathit{y}}}_i$ of ${\tilde{\mathit{x}}}_{\mathit{i}}$ after the second 1D-CNN is:

$${\widehat{\mathit{y}}}_{\mathit{i}}=\sigma ({\mathit{w}}_{\mathit{i}}\otimes {\tilde{\mathit{x}}}_{\mathit{i}}+{\mathit{b}}_{\mathit{i}})$$

(11)

The output information $\tilde{\mathit{y}}=[{\tilde{\mathit{y}}}^1,\cdots ,{\tilde{\mathit{y}}}^{\mathrm{N}}]$ is acquired by conducting the same operations and processing on a matrix of load data $\mathit{X}={\left[{\mathit{x}}_1,\cdots {\mathit{x}}_{\mathrm{C}}\right]}^T$ splited by a specified time in a sequential manner, and the feature $\mathit{y}$ is obtained by sewing $\widehat{\mathit{y}}$ and $\tilde{\mathit{y}}$ together, resulting in multi-feature fusion.

$$\mathit{y}=[\widehat{\mathit{y}},\tilde{\mathit{y}}]$$

(12)

3.3. Graph Convolutional Network

GCN is a feature extractor that works similarly to the CNN, except that a CNN extracts tensor data features, while a GCN extracts graph-structured data features. In this paper, we use GCN to extract external non-numerical features such as weather, wind force and wind direction, and reorganize the extracted features with global information. The GCN is made up of T vertices and linked edges between them, marked as $\mathit{G}=(\mathit{V},\mathit{A})$, where $\mathit{V}$ indicates the set of vertices and $A$ denotes the adjacency matrix of dimension $T\times T$ with each vertex having a feature dimension A and all vertices constituting a feature matrix $X$ of dimension $T\times D$.

GCN is essentially concerned with determining a learnable kernel weight that applies to the graph. The GCN propagates across layers as a deep learning model in the following manner.

$${\mathit{H}}^{(l+1)}=\sigma \left({\widehat{\mathit{D}}}^{-\frac{1}{2}}\widehat{\mathit{A}}{\widehat{\mathit{D}}}^{-\frac{1}{2}}{\mathit{H}}^{\left(l\right)}{\mathit{W}}^{\left(l\right)}\right)$$

(13)

where $\widehat{\mathit{A}}=\mathit{A}+\mathit{I}$, $\mathit{I}$ is the unit matrix, which is the same as adding a self-loop to each vertex. $\widehat{\mathit{D}}$ is the connectivity diagonal matrix characterizing the connectivity of $\mathit{G}$, ${\mathit{H}}^{\left(l\right)}$ is the feature matrix of the layer l, and ${\mathit{W}}^{\left(l\right)}$ represents the trainable weight of the layer l, $\sigma \left(·\right)$ represents the activation function.

External factors such as weather, wind force, and wind direction are non-numerical data that are connected in load forecasting, and their correlation matrix may be created using conditional probabilities [35]. The matrix can be obtained by assuming that the number of times $C_i$ an external feature label appears in the training dataset, and another external feature label appearing in the training dataset is $C_j$, and both appearing in the training dataset is $C_{ij};i,j=1,\cdots ,T$, and T indicate the dimensionality of the external feature label in the dataset. As a result, $\mathit{C}\in {\mathrm{R}}^{T\times T}$ can be obtained. The probability matrix of external characteristics, that is, the adjacency matrix $\mathit{A}\in {\mathrm{R}}^{T\times T}$, may be computed, and its matrix elements $a_{ij}$ can be identified.

$$a_{ij}=\frac{C_{ij}}{C_i};i,j=1,\cdots ,T$$

(14)

where $a_{ij}$ represents the likelihood that $C_j$ occurs when $C_i$ occurs. The correlation between external features is extracted using a multi-layer GCN, which means that the external features are quantized and encoded first, then mapped into a high-dimensional word embedding vector by the word embedding method. Finally, the word embedding vector and adjacency matrix are fed into the GCN for feature extraction to obtain the output $G_o$. As seen below, $G_o$ is multiplied by $y$ get the feature vector improved by GCN.

$$\mathit{S}={\mathit{G}}_o\times \mathit{y}$$

(15)

A fully linked layer then S-reduces the anticipated values to the predictive dimension.

3.4. Bayesian Optimization

The deep learning optimization issue is a black-box optimization problem, with the main notion being to assess each set of hyperparameter values in the set of hyperparameters $\mathit{Z}=\{z_1,\cdots ,z_n\}$, and the evaluation result may be stated as $f\left(z\right)$. The best hyperparameter $z^{*}$ is discovered throughout the optimization process, i.e.,:

$$z^{*}=\arg \underset{z\subseteq Z}{min}f\left(z\right)$$

(16)

BO [36] discovers parameter values that minimize the objective function by estimating probabilistic proxy models (substitution functions) which is based on previous assessments of the objective function. In contrast of the random and grid searches, BO considers prior evaluations when deciding on the next hyperparameter value. BO is utilized in this research to alter the hyperparameters and get the value of the hyperparameter which is best for the probabilistic agent model. Compared with the grid search method, BO requires fewer iterations. The following are the steps in the BO method.

Step 1: Determine the goal function $f\left(z\right)$ and the defining domain of the hyperparameter z. The independent variable $z$ in the optimization is the hyperparameter space, and the objective function is often a loss function. In reality, the formulation of the loss function is known, while the only thing unknown is the exact legislation.

Step 2: Determine the observed values. n parameter values $\mathit{Z}=\{z_1,\cdots ,z_n\}$ are retrieved from the hyperparameters’ designated domain, and the objective function value $\left\{f\left(z_i\right)\right|i=1,\cdots ,n\}$ corresponding to each parameter value is determined.

Step 3: Develop a probabilistic proxy model. The goal function’s distribution model is selected based on the data, often using a Gaussian or a mixed Gaussian process.

Step 4: Choose a collection function. The collection function is used to choose the next hyperparameter value and to assess the influence of the chosen hyperparameter value on the fitted results, often using the probability of improvement, expectation improvement, upper confidence bound and entropy, etc.

Repeat steps 2–4 until the distribution model’s goal value meets a predefined condition or the computing resources are depleted (e.g., the maximum number of observations or maximum allowed running time).

4. Experiments and Evaluation

4.1. Training Planning

4.1.1. Information Sources

The data is derived from the Shandong Data Open Network, China - Grid Bus Load Dataset and includes monitoring data for various buses corresponding to 5-min sample intervals from 1 January 2019 to 2 March 2020, with around 20,000 data points for each bus. Furthermore, the weather data for distinct area IDs comprise a total of 142 data points for weather (e.g., sunny/cloudy), 78 data points for wind force, and wind direction (e.g., north wind force 1–2/north wind force 3–4), and 83 data points for substation and bus information. The number of buses used in these trials is 10, with a total data volume of roughly 200,000. Figure 5 depicts non-digital feature data such as wind force, wind direction, and weather.

4.1.2. Pre-Processing of Data

The pre-processing of load data, as well as weather, wind force, and wind direction parameters consists primarily of data cleaning and data transformation. Because the busbar load monitoring data is acquired every 5 min and does not fluctuate significantly, the average value of 10 min before and after (i.e., four monitoring data) is utilized instead for the load data. Different categories of weather, wind force, and wind direction were initially searched for, then weather, wind force, and wind direction were transformed into numerical characteristics based on the categories. The adjacency matrix and feature matrix dimension with the most weather, wind force, and wind direction categories, 142, is selected, and the adjacency matrix dimension is $T\times T=142\times 142$. The label vector is transferred to high-dimensional space using a word embedding to create the dimensioned feature matrix with the dimension of $T\times D=142\times 300$.

The bus monitoring data every 5 min for 24 h (288 total data) is used to estimate the load data every 5 min for the following hour (12 total data) therefore the input has a load data matrix dimension of $N\times C=24\times 12$, and the output dimension is 12.

4.1.3. Establishment of a Training Environment

There is a link between the deep learning training environment, computational power, and training version. All models in the following trials were trained in the same software environment to control for factors. The operating system is Windows 10, all models were developed in TensorFlow using TensorFlow version 2.5.0, the CUDA version needed for GPU training was 11.3, and the cuDNN version was 8.2, and the GPU was an NVIDIA RTX 3090.

The hyperparameters are set up as follows during the model training phase: the total number of training epochs is set to 100, and the Adam optimizer is employed. The learning rate was initially set at 0.1. The learning rate was doubled by 0.8 per 10 generations. The Bi- LSTM’s output dimension was 32. The CNN had two layers, the output dimension was 16 and 32, the convolutional kernel had three layers, the stride was set to SAME, The GCN had two layers, the output feature dimension was 32 and 64, and the batch size was 28,000.

The first $70\%$ of monitoring data from each bus (a total of 140,000 data points) was utilized to create the training data set, and the final output was the projected outcome of the 60,000 individual monitoring data points selected for the test. Five identical experiments were averaged to improve experiment reliability. As indicated in Figure 6, the loss function was the Mean Square Error (MSE), MSE defined as follows:

$$\mathrm{MSE}=\frac{1}{N}\sum _{i=1}^n\left(y_i-{\widehat{y}}_i\right)$$

(17)

where $y_i$ is the actual load data, ${\widehat{y}}_i$ is the anticipated load data, and N is the sample size.

As shown in the graph, as the number of iterations increases, the loss value of the training data set and the loss value of the test data set continue to decrease until they are near zero, showing that the training can be completed correctly and excellent results may be obtained.

4.1.4. Indicators of Evaluation

As assessment indicators, Symmetric Mean Absolute Percentage Error (SMAPE), Root Mean Square Error (RMSE), Mean Bias Error (MBE) and coefficient of determination (R-squared) were used, with SMAPE defined as follows:

$$\mathrm{SMAPE}=\frac{100\%}{N}\sum _{i=1}^N\frac{|y_i-{\widehat{y}}_i|}{\left(\right|y_i\left|\right)/2+|{\widehat{y}}_i|}$$

(18)

where $y_i$ is the actual load data, ${\widehat{y}}_i$ is the anticipated load data, and N is the sample size.

The RMSE is a measure of the degree of variance in data and it is defined as:

$$\mathrm{RMSE}=\sqrt{\frac{1}{N}{\sum }_{i=1}^N(y_i-{\widehat{y}}_i{)}^2}$$

(19)

The MBE does not use absolute values so that positive and negative errors can cancel each other out. It determines whether there is a positive or negative bias in the model. MBE is defined as follows:

$$\mathrm{MBE}=\frac{{\sum }_{i=1}^N\left(y_i-{\widehat{y}}_i\right)}{N}$$

(20)

The R-squared is used to evaluate the goodness of fit, with the denominator being the dispersion of the original data and the numerator being the error between the predicted and true data, the two being divided to eliminate the effect of the original data dispersion, defined as:

$$R^2=1-\frac{{\sum }_{i=1}^N{(y_i-{\widehat{y}}_i)}^2}{{\sum }_{i=1}^N{(y_i-\overline{y})}^2}$$

(21)

where $\overline{y}$ is the mean of the actual load data

4.2. Test Results Comparison

Machine learning methods such as Linear Regression (LR), K-Nearest Neighbor (KNN), Decision Tree (DT), Bi-LSTM, CNN, and CNN-BILSTM (S2S), and other deep learning models were used as comparison models to compare and assess the prediction performance of various prediction models. CNN-Bi-LSTM (S2S) is distinct from the MFF-SAM-GCN model suggested in this research, which employs CNN first for initial feature extraction and then Bi-LSTM for subsequent feature extraction. To maintain consistency with the previous parameters, the optimizer was set to Adam, and the learning rate was set to 0.1. The highest number of iterations was 100.

Table 1. Prediction performance of different models.

Method	RMSE	SMAPE	MBE	$R^2$	Runtimes (Company: s)
LR	0.037	12.561%	0.036	0.952	189.431
KNN	0.042	15.243%	0.040	0.947	159.563
DT	0.045	16.655%	0.042	0.939	209.767
BiLSTM	0.034	11.046%	0.033	0.964	317.269
CNN	0.035	11.784%	0.033	0.957	306.436
CNN-LSTM(S2S)	0.032	10.542%	0.031	0.974	324.584
GCN-MTF	0.028	9.453%	0.025	0.989	337.063

displays the prediction results of each model. The RMSE, SMAPE and MBE of the proposed model in this work are lower, and the R-squared is greater, showing that the model’s prediction accuracy is better than the comparison model.

After 100 iterations, the final predicted load data for every 5 min in 1 hour is shown in Figure 7.

4.3. Experiments with Ablation

This section performs independent ablation tests for the SAM and GCN in order to assess the influence of these two modules on the experiments.

SAM module effect: Remove the SAM in the midst of the two 1D-CNNs to see the effect of the SAM. The rest of the options are the same.

GCN module effect: Remove GCN and disregard the effect of weather and external characteristics of wind force, and wind direction on the electrical load.

Table 2. Outcomes of the abliation trials.

Basic	Attention	GCN	RMSE	SMAPE	MBE	$R^2$	Runtimes (Company: s)
	-	-	0.037	11.485%	0.035	0.965	316.453
CNN-LSTM	✓	-	0.035	11.236%	0.032	0.972	322.536
(MFF)	-	✓	0.032	10.256%	0.031	0.979	326.461
	✓	✓	0.028	9.453%	0.025	0.989	337.063

displays the outcomes of the ablation trials. The findings reveal that eliminating the SAM and GCN from the MFF-SAM-GCN prediction model framework increases the RMSE, SMAPE, and MBE, reduces the R-squared, and makes the model less effective, demonstrating that SAM and GCN may enhance the model’s performance.

4.4. Extensive Research

4.4.1. The Effect of Hyperparameter Values

BO is used to discover the ideal hyperparameters in order to investigate and test the influence of hyperparameter settings on the model’s prediction performance. The objective function is the model’s loss function on the test dataset when the set of hyperparameters is used. The goal function in this research is the Mean Absolute Error (MAE) and MAE is defined as follows:

$$\mathrm{MAE}=\frac{1}{N}\sum _{i=1}^N\left|\left(y_i-{\widehat{y}}_i\right)\right|$$

(22)

where $y_i$ is the actual load data, ${\widehat{y}}_i$ is the anticipated load data, and N is the sample size.

The domain space is represented as the range of values chosen for each hyperparameter, and the alternative parameters in the domain space are picked based on empirical data as shown in

Table 3. Alternative parameters of domain space.

Hyperparameter	Search Scope
drop_out_rate	(0.1, 0.2, 0.3)
learning_rate	(0.1, 0.2, 0.3)
Bi-LSTM_hidden size	(32–128)
CNN1_number of filter	(16, 32, 64)
CNN2_number of filter	(16, 32, 64)
CNN1_kernel size	(2, 3, 4)
CNN1_kernel size	(2, 3, 4)
epochs	(50, 100, 200)
Batch size	(15,000, 20,000, 25,000, 30,000)

.

The domain space is searched 20 times, with the least MAE after each search shown in Figure 8. The smallest MAE after the 18th search, according to the graph, is 0.01928 with parameters [0.2, 0.1, 32, 16, 32, 2, 3, 200, 25,000].

4.4.2. The Impact of the Bus

Ten buses were used in the experiment, and three sub-experiments were carried out. In the first experiment, one input was added. The buses were embedded as words, and the buses’ features were extracted using LSTM networks, and fused with the MFF-SAM-GCN model to obtain the fused features, and then the fused features were enhanced using GCN; in the second experiment, the MFF-SAM-GCN model was trained using all of the experimental data without taking into account the influence of buses on the prediction results; the third method is to train the respective models for different buses and make predictions, the final predicted value is the average of all model predictions.

Table 4. Model performance comparison under different strategies.

Method	RMSE	SMAPE	MBE	$R^2$	Runtimes (Company: s)
method 1	0.028	9.286%	0.025	0.990	353.985
method 2	0.028	9.453%	0.025	0.989	337.063
method 3	0.026	9.232%	0.023	0.993	2980.231

displays the outcomes of the tests. As shown in the table, the RMSE, SMAPE, MBE, and R-squared obtained from the three experiments do not differ significantly, and the highest accuracy is obtained by training ten models without taking into account the systematic errors generated during the training process, but the time consumed increases as well.

5. Conclusions and Future Work

We propose an SLF model combined with MFF, SAM, and GCN, employing a Bi-LSTM network and SAM based 1D-CNN to extract spatiotemporal correlation features of load data and conduct feature fusion. We use the correlation between the weather, wind force, and wind direction to construct a GCN model, which not only captures textual information but also effectively enhances the concentration of load features and improves the accuracy of predictions.The proposed MFF-SAM-GCN forecasting model has been tested on the bus load dataset of Shandong Open Data Network, China. The simulation is among our proposed model and six benchmark models, i.e., three machine learning models (LR, KNN, and DT) and three deep learning models (Bi-LSTM, CNN, and CNN-BILSTM (S2S)). The simulation results indicate that the MFF-SAM-GCN prediction model has a SMAPE of 9.453%, an RMSE of 0.028, a MBE of 0.025 and an R-squared of 0.989, with an overall better prediction performance than the benchmark models. In addition, ablation tests were carried out in this paper to investigate the function of the SAM and GCN. Finally, extensive tests have been carried out to investigate and assess the impacts of hyperparameter settings and buses on prediction performance.

Finally, the MFF-SAM-GCN model described in this article is proven to accept a variety of data inputs. MFF can efficiently extract time-related and spatially-related features from load data, expanding the feature dimension of load prediction. Meanwhile, GCN can effectively extract textual features and capture external characteristics that have an influence on load prediction. In the future, different datasets will be sought to verify the generality of the model. New optimization algorithms or the integration of multiple optimization algorithms will be investigated to optimize the forecasting model, while more external such as day types, economic indicators factors [37] that may have an impact on load forecasting will be investigated to further improve forecasting accuracy and develop a practical product.