Hessian with Mini-Batches for Electrical Demand Prediction

Abstract

The steepest descent method is frequently used for neural network tuning. Mini-batches are commonly used to get better tuning of the steepest descent in the neural network. Nevertheless, steepest descent with mini-batches could be delayed in reaching a minimum. The Hessian could be quicker than the steepest descent in reaching a minimum, and it is easier to achieve this goal by using the Hessian with mini-batches. In this article, the Hessian is combined with mini-batches for neural network tuning. The discussed algorithm is applied for electrical demand prediction.

1. Introduction

Networks have many applications like detection [1,2], recognition [3,4], classification [5,6], and prediction [7,8]. Steepest descent is a supervised algorithm that is frequently used for neural network tuning, wherein the value of the scale parameters is adjusted according to the cost map. Steepest descent evaluates the first-order partial derivatives of the cost map with respect to the scale parameters in the neural network.

Mini-batches are commonly used to get better tuning of the steepest descent in a neural network; the training data are divided in mini-batches, and the training of the steepest descent is applied to all the mini-batches taking into account one tuning of the scale parameters for each mini-batch. One tuning of all the mini-batches is one epoch.

There are several applications for mini-batches. In [9,10,11,12], mini-batches were employed for tuning. In [13,14], mini-batches were used for clustering. In [15,16], mini-batches were utilized for optimization. Since mini-batches have been used in several applications, they could be a good alternative to get better tuning using steepest descent.

Steepest descent with mini-batches is used during tuning with a search for each mini-batch. Nevertheless, steepest descent with mini-batches could be delayed in reaching a minimum. The Hessian has been used as an alternative for neural network tuning, wherein the Hessian evaluates the second-order partial derivatives of the cost map with respect to the scale parameters.

The Hessian has the same form as the steepest descent. However, steepest descent takes into account constant values in its tuning rate and momentum, while the Hessian takes into account the second-order partial derivatives of the cost map with respect to the scale parameters in its tuning rate and momentum. This is the main reason why the steepest descent method may be delayed in reaching a minimum, while the Hessian could be quicker in reaching a minimum, and it is easier to reach this minimum by using the Hessian with mini-batches.

There have been several applications of the Hessian. In [17,18,19,20], the Hessian was used for tuning. In [21,22,23,24], the Hessian was employed for segmentation. In [25,26,27,28], the Hessian was utilized for optimization. In [29,30,31,32], the Hessian was used for pattern recognition. In [33,34], the Hessian was utilized for modeling. In [35,36], the Hessian was employed for identification. In [37,38], the Hessian was used for control. Since the Hessian has been used in several applications, it could be a good alternative for neural network tuning.

In this article, the Hessian is combined with mini-batches for neural network tuning. The full training data are divided in mini-batches, and we take into account each mini-batch to get better tuning of the Hessian in a neural network. One tuning of the mini-batches is known as one epoch.

Finally, we compare the tuning of the neural network using steepest descent, steepest descent with mini-batches, the Hessian, and the Hessian with mini-batches for electrical demand prediction, based on data provided by the International Organization for Standardization (ISO) of Great Britain.

The remainder of this article is organized as follows: Section 2 presents neural network tuning via the Hessian. Better tuning of the Hessian using mini-batches is explained in Section 3. Section 4 shows the comparison results of steepest descent, steepest descent with mini-batches, Hessian, and Hessian with mini-batches for electrical demand prediction. Conclusions and future work are presented in Section 5.

2. The Hessian for Neural Network Tuning

The algorithms for neural network tuning frequently evaluate the first derivatives of the cost map with respect to the scale parameters. Nevertheless, there are several cases where it is interesting to get the second derivatives of the cost map with respect to the scale parameters. The second derivatives of the cost map with respect to the scale parameters are known as the Hessian.

2.1. Design of the Hessian

In this article, we utilize a neural network with only one hidden layer. It could be expanded to a multilayer neural network, but in this article, we utilize a small neural network. This neural network utilizes sigmoid maps in the hidden layer and a linear map in the output layer. We express the neural network as

$$q_l={\displaystyle \sum }_j{\phi }_{lj}g({\displaystyle \sum }_i{\theta }_{ji}{b}_i)$$

(1)

where ${\theta }_{ji}$ is the scale parameter of the hidden layer, ${\phi }_{lj}$ is the scale parameter of the output layer, $g$ is the activation map, $b_i$ are the inputs, and $q_l$ are the outputs.

We take into account the neural network in Figure 1 where $n$ is the input layer, $l$ is the hidden layer, and $m$ is the output layer. We express the scale parameters from the input layer to the hidden layer as ${\theta }_{ji}$ and the scale parameters from the hidden layer to the output layer as ${\theta }_{lm}$.

We express the cost map as

$$E=\frac{1}{2}{\displaystyle \sum }_{l=1}^{L_T}{\left(q_l-t_l\right)}^2$$

(2)

where $q_l$ is the output of the neural network and $t_l$ is the target, $L_T$ is the total output number. We express the forward propagation as

$$\begin{array}{c}{z}_j={\displaystyle \sum }_i{\theta }_{ji}{b}_i\\ o_j=g(z_j)\\ x_l={\displaystyle \sum }_j{\phi }_{lj}{o}_j\\ q_l=f(x_l)=x_l\end{array}$$

(3)

where $b_i$ is the input, $q_l$ is the output of the neural network, ${\theta }_{ji}$ are hidden layer scale parameters, and ${\phi }_{lj}$ are output layer scale parameters.

We take into account the activation map in the hidden layer as

$$g(z_j)=\frac{1}{1+e^{-z_j}}.$$

(4)

The first and second derivatives of the sigmoid map (4) are

$$\begin{array}{c}{g}^{/}(z_j)=g(z_j)\left(1-g(z_j)\right),\\ g^{//}(z_j)=g(z_j)\left(1-g(z_j)\right)\left(1-2g(z_j)\right).\end{array}$$

(5)

We take into account the activation map of the output layer as the linear form

$$f(x_l)=x_l.$$

(6)

The first and second derivatives of the linear map (6) are

$$\begin{array}{c}{f}^{/}(x_l)=1,\\ f^{//}(x_l)=0.\end{array}$$

(7)

The first and second derivatives of the cost map (2) are

$$\begin{array}{c}\frac{\partial E}{\partial q_l}=\left(q_l-t_l\right),\\ \frac{{\partial }^{2}E}{\partial q_l^2}=1.\end{array}$$

(8)

Using the cost map (2) and $f(x_l)=x_l$ (6), we express the propagation of the output layer as

$$\begin{array}{c}\frac{\partial E}{\partial {\phi }_{lj}}=\frac{\partial E}{\partial q_l}\frac{\partial q_l}{\partial x_l}\frac{\partial x_l}{\partial {\phi }_{lj}}\\ =\left(q_l-t_l\right)\frac{\partial f(x_l)}{\partial x_l}{o}_j\\ =\left(q_l-t_l\right)\frac{\partial x_l}{\partial x_l}{o}_j\\ =\left(q_l-t_l\right)(1)g(z_j),\\ \Rightarrow \frac{\partial E}{\partial {\phi }_{lj}}=\left(q_l-t_l\right)g(z_j).\end{array}$$

(9)

Using the cost map (2) and $g^{/}(z_j)=\frac{\partial o_j}{\partial z_j}=\frac{\partial g(z_j)}{\partial z_j}=g(z_j)\left(1-g(z_j)\right)$ (5), we express the propagation of the hidden layer as

$$\begin{array}{c}\frac{\partial E}{\partial {\theta }_{ji}}=\frac{\partial E}{\partial q_l}\frac{\partial q_l}{\partial x_l}\frac{\partial x_l}{\partial o_j}\frac{\partial o_j}{\partial z_j}\frac{\partial z_j}{\partial {\theta }_{ji}}\\ =\left(q_l-t_l\right)(1){\phi }_{lj}{g}^{/}(z_j)b_i,\\ \Rightarrow \frac{\partial E}{\partial {\theta }_{ji}}=\left(q_l-t_l\right){\phi }_{lj}g(z_j)\left(1-g(z_j)\right)b_i.\end{array}$$

(10)

We express the second derivative of $E$ as the Hessian $H$:

$$H=\nabla \nabla E=\frac{{\partial }^{2}E}{\partial {\theta }^2}=\left[\begin{array}{cc}\frac{{\partial }^{2}E}{\partial {\theta }_{ji}^2}& \frac{{\partial }^{2}E}{\partial {\theta }_{ji}\partial {\phi }_{lj}}\\ \frac{{\partial }^{2}E}{\partial {\theta }_{ji}\partial {\phi }_{lj}}& \frac{{\partial }^{2}E}{\partial \theta {\phi }_{lj}^2}\end{array}\right]$$

(11)

and the Hessian is symmetrical:

$$\frac{{\partial }^{2}E}{\partial {\theta }_{ji}\partial {\phi }_{jl}}=\frac{{\partial }^{2}E}{\partial {\phi }_{jl}\partial {\theta }_{ji}}.$$

(12)

The Hessian terms are

$$\begin{array}{c}\frac{{\partial }^{2}E}{\partial {\theta }_{ji}^2}=b_i{b}_i^T{\phi }_{lj}\left[g^{//}(z_j){\sigma }_i+g^{/}{(z_j)}^2{\phi }_{lj}{S}_i\right]\\ \frac{{\partial }^{2}E}{\partial {\theta }_{ji}\partial {\phi }_{lj}}=b_i{g}^{/}(z_j)\left[{\sigma }_i+o_j{\phi }_{lj}{S}_i\right]\\ \frac{{\partial }^{2}E}{\partial {\phi }_{lj}^2}=o_j{o}_j^T\left[f^{//}(x_l){\sigma }_i+f^{/}{(x_l)}^2{S}_i\right]\end{array}$$

(13)

and

$$\begin{array}{c}{S}_i=\frac{{\partial }^{2}E}{\partial q_l^2}=1,\\ g^{/}(z_j)=g(z_j)\left(1-g(z_j)\right), f^{/}(x_l)=1,\\ g^{//}(z_j)=g(z_j)\left(1-g(z_j)\right)\left(1-2g(z_j)\right), f^{//}(x_l)=0,\\ o_j=\frac{\partial x_l}{\partial {\phi }_{lj}}=g(z_j), b_i=\frac{\partial z_j}{\partial {\theta }_{ji}}=\mathrm{inputs},\\ g(z_j)=\frac{1}{1+e^{-z_j}}, f(x_l)=x_l,\\ {\sigma }_i=\left(q_l-t_l\right).\end{array}$$

We substitute (13) and (11); then the Hessian is

$$\begin{array}{c}H=\nabla \nabla E=\frac{{\partial }^{2}E}{\partial {\theta }^2}=\left[\begin{array}{cc}\frac{{\partial }^{2}E}{\partial {\theta }_{ji}^2}& \frac{{\partial }^{2}E}{\partial {\theta }_{ji}\partial {\phi }_{lj}}\\ \frac{{\partial }^{2}E}{\partial {\theta }_{ji}\partial {\phi }_{lj}}& \frac{{\partial }^{2}E}{\partial {\phi }_{lj}^2}\end{array}\right]\\ \frac{{\partial }^{2}E}{\partial {\theta }_{ji}^2}=b_i{b}_i^T{\phi }_{lj}[g(z_j)\left(1-g(z_j)\right)\left(1-2g(z_j)\right)\left(q_l-t_l\right)\\ +g{(z_j)}^2{\left(1-g(z_j)\right)}^2{\phi }_{lj}]\\ \frac{{\partial }^{2}E}{\partial {\theta }_{ji}\partial {\phi }_{lj}}=b_{i}g(z_j)\left(1-g(z_j)\right)\left[\left(q_l-t_l\right)+g(z_j){\phi }_{lj}\right]\\ \frac{{\partial }^{2}E}{\partial {\phi }_{lj}^2}=g(z_j)g{(z_j)}^T\end{array}$$

(14)

where $b_i$ are the inputs, $q_l$ are the outputs, $g(z_j)=\frac{1}{1+e^{-z_j}}$ are the activation maps, $t_l$ are the targets, $z_j={\theta }_{ji}{b}_i$ are the hidden layer outputs, and ${\phi }_{lj}$ are the scale parameters of the hidden layer.

In the next step, we evaluate the Hessian using the Newton method.

2.2. Design of the Newton Method

It is necessary to express a method to tune the scale parameters of the Hessian. The Newton method is one alternative to tune the scale parameters of the Hessian. We express the basic tuning of the Newton method as follows:

$$\begin{array}{c}\left[\begin{array}{c}{\theta }_{ji,k+1}\\ {\phi }_{lj,k+1}\end{array}\right]=\left[\begin{array}{c}{\theta }_{ji,k}\\ {\phi }_{lj,k}\end{array}\right]-\alpha H_k^{-1}\left[\begin{array}{c}\frac{\partial E_k}{\partial {\theta }_{ji,k}}\\ \frac{\partial E_k}{\partial {\phi }_{lj,k}}\end{array}\right]\\ H_k=\left[\begin{array}{cc}\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}& \frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}\\ \frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}& \frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}\end{array}\right]\end{array}$$

(15)

where $\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}$, $\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}$, $\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}$ are as in (14) for each $k$; $\frac{\partial E_k}{\partial {\phi }_{lj,k}}$, $\frac{\partial E_k}{\partial {\theta }_{ji,k}}$ are as in (9), (10); ${\theta }_{ji,k}$, ${\phi }_{lj,k}$ are the scale parameters; and $\alpha$ is the tuning factor. The Newton method can quickly reach a minimum. The Newton method requires the existence of the inverse of the Hessian ($H_k^{-1}$).

Now, we express the Newton method of (15) by terms. First, from (15), we obtain the inverse of $H_k$ as

$$\begin{array}{c}{H}_k^{-1}={\left[\begin{array}{cc}\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}& \frac{{\partial }^2{E}_k}{\partial {\theta }_{j,k}\partial {\phi }_{lj,k}}\\ \frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}& \frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}\end{array}\right]}^{-1}\\ =\{\frac{1}{\left(\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}\right)\left(\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}\right)-{\left(\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}\right)}^2}\\ *\left[\begin{array}{cc}\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}& -\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}\\ -\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}& \frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}\end{array}\right]\}.\end{array}$$

(16)

We substitute $H_k^{-1}$ of (16) and ${\theta }_k$, $\frac{\partial E_k}{\partial {\theta }_k}$ of (16) into ${\theta }_{k+1}$ of (16) as follows:

$$\begin{array}{c}\left[\begin{array}{c}{\theta }_{ji,k+1}\\ {\phi }_{lj,k+1}\end{array}\right]=\left[\begin{array}{c}{\theta }_{ji,k}\\ {\phi }_{lj,k}\end{array}\right]\\ -\{\alpha \frac{1}{\left(\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}\right)\left(\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}\right)-{\left(\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}\right)}^2}\\ *\left[\begin{array}{cc}\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}& -\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}\\ -\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}& \frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}\end{array}\right]\left[\begin{array}{c}\frac{\partial E_k}{\partial {\theta }_{ji,k}}\\ \frac{\partial E_k}{\partial {\phi }_{lj,k}}\end{array}\right]\}.\end{array}$$

(17)

We express (17) by terms as

$$\begin{array}{c}{\theta }_{ji,k+1}={\theta }_{ji,k}-{\beta }_{Hji,k}\frac{\partial E_k}{\partial {\theta }_{ji,k}}+{\gamma }_{H,k}\frac{\partial E_k}{\partial {\phi }_{lj,k}}\\ {\phi }_{lj,k+1}={\phi }_{lj,k}-{\beta }_{Hlj,k}\frac{\partial E_k}{\partial {\phi }_{lj,k}}+{\gamma }_{H,k}\frac{\partial E_k}{\partial {\theta }_{ji,k}}\\ {\beta }_{Hji,k}=\alpha \frac{\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}}{\left(\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}\right)\left(\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}\right)-{\left(\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}\right)}^2}\\ {\beta }_{Hlj,k}=\alpha \frac{\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}}{\left(\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}\right)\left(\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}\right)-{\left(\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}\right)}^2}\\ {\gamma }_{H,k}=\alpha \frac{\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}}{\left(\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}\right)\left(\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}\right)-{\left(\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}\right)}^2}\end{array}$$

(18)

where $\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}$, $\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}$, $\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}$ are as in (14) for each $k$; $\frac{\partial E_k}{\partial {\phi }_{lj,k}}$, $\frac{\partial E_k}{\partial {\theta }_{ji,k}}$ are as in (9), (10) for each $k$; ${\theta }_{ji,k}$, ${\phi }_{lj,k}$ are the scale parameters for each $k$; and $\alpha$ is the tuning factor. Thus, (18) is the Newton method by terms.

For comparison, we express the steepest descent method as

$$\begin{array}{c}{\theta }_{ji,k+1}={\theta }_{ji,k}-{\beta }_{Gji,k}\frac{\partial E_k}{\partial {\theta }_{ji,k}}+{\gamma }_{G,k}\frac{\partial E_k}{\partial {\phi }_{lj,k}}\\ {\phi }_{lj,k+1}={\phi }_{lj,k}-{\beta }_{Glj,k}\frac{\partial E_k}{\partial {\phi }_{lj,k}}+{\gamma }_{G,k}\frac{\partial E_k}{\partial {\theta }_{ji,k}}\\ {\beta }_{Gji,k}=\alpha \\ {\beta }_{Glj,k}=\alpha \\ {\gamma }_{G,k}=0\end{array}$$

(19)

where $\frac{\partial E_k}{\partial {\phi }_{lj,k}}$, $\frac{\partial E_k}{\partial {\theta }_{ji,k}}$ are as in (9), (10) for each $k$; ${\theta }_{ji,k}$, ${\phi }_{lj,k}$ are the scale parameters for each $k$; and $\alpha$ is the tuning factor. Thus, (19) is the steepest descent method.

It can be seen that the Newton method by terms (Hessian) (18) has the same form as the steepest descent (19). However, the steepest descent (19) takes into account constant values in its tuning rate ${\beta }_{Gij,k}$, ${\beta }_{Gji,k}$ and momentum ${\gamma }_{G,k}$, while the Hessian (18) takes into account the second-order partial derivatives of the cost map with respect to the scale parameters in its tuning rate ${\beta }_{Hij,k}$, ${\beta }_{Hji,k}$ and momentum ${\gamma }_{H,k}$.

We express the mini-batches in the next section to get better tuning of the Hessian.

3. Mini-Batches to Get Better Tuning of the Hessian

The form to update the scale parameters of the neural networks is that each neuron assigns information to the next neuron and it receives information from the previous neuron. We need training for successful neural network tuning. The training is developed from one epoch to the next until the scale parameters reach constant values and the cost map reaches a minimum. In addition, we need the training data to be tuned in a random form with the goal to quickly reach a minimum.

In the training stage, the neural network computes its outputs each time to obtain a result, and we compare the outputs with targets; in this way, the cost map of the neural network decreases. The scale parameters take random initial values, and these scale parameters are tuned through time.

We use other testing data as the basic method to evaluate the neural network efficacy. This consists of taking $80\%$ of the data for training and taking $20\%$ of the data for testing. The first stage is the training, and the second stage is the testing.

In the tuning, the training and testing stages are important.

Design of the Mini-Batches

We take into account training data $v$ and $u$ characteristics:

$$B_{uv}=\underset{b_{1,uv}}{\left[b_1,b_2,b_3,\dots ,b_v\right]},$$

(20)

$$Q_v=\underset{q_{1,v}}{\left[q_1,q_2,q_3,\dots ,q_v\right]}.$$

(21)

In the mini-batches, we divide the training data $v$ into $w$-many mini-batches of size $y$, with the goal to quickly reach a minimum.

$$B_{uv}=\left[|\underset{b_{1,uy}}{b_{u1},\dots ,b_{uy}}|\underset{b_{2,uy}}{b_{uy+1},\dots ,b_{u2y}}|\underset{b_{w,uy}}{\dots ,b_{uwy}|}\right]$$

(22)

$$Q_v=\left[\underset{q_{1,y}}{|q_1,\dots ,q_y}|\underset{q_{2,y}}{q_{y+1},\dots ,q_{2y}|},\dots \underset{q_{w,y}}{,q_{wy}|}\right]$$

(23)

We express the Hessian with mini-batches as (we divide the training data $v$ in w-many mini-batches of size $y$):

(1)

For each epoch.

(2)

Evaluate the mini-batches and tune each of the mini-batches $c=1,2,\dots ,w$ with (24). These values are expressed in (15), (18).

(3)

Repeat for the next epoch.

$${\theta }_{k+1}={\theta }_k-\alpha {\displaystyle \sum }_{d=1}^y{H}_{k,d}^{-1}\frac{\partial E_{k,d}}{\partial {\theta }_k}$$

(24)

We express the properties of the mini-batches below:

• Most of the time, we do not need to utilize all data to reach an acceptable descent direction. A small number of mini-batches could be sufficient to estimate the target.
• Obtaining the Hessian using all the training data could have high computational cost.

The neural network tuning is performed using the Hessian with mini-batches (15), (18) where $\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}^2}$, $\frac{{\partial }^2{E}_k}{\partial {\phi }_{lj,k}^2}$, $\frac{{\partial }^2{E}_k}{\partial {\theta }_{ji,k}\partial {\phi }_{lj,k}}$ are as in (14); $\frac{\partial E_k}{\partial {\phi }_{lj,k}}$, $\frac{\partial E_k}{\partial {\theta }_{ji,k}}$ are as in (9), (10); and ${\theta }_{ji,k}$, ${\phi }_{lj,k}$ are the scale parameters. We tune the neural network with a tuning factor of $\alpha$ and $l$ neurons in the hidden layer, and we use $e$ epochs. In this kind of tuning, we divide the training data into mini-batches as in (22,23).

4. Comparisons

In this section, we compare steepest descent (SD), steepest descent with mini-batches (SDMB) from [9,10,11,12], the Hessian (H) from [17,18,19,20], and the Hessian with mini-batches (HMB) from this investigation for electrical demand prediction. The goal of these algorithms is that the neural network output $q_l$ must reach the target $t_l$ as soon as possible.

Efficient electrical demand prediction is critical for acceptable operations and planning with the intention of achieving profits. The load forecast influences a series of decisions, including the generators to be used for a given period, and influences the wholesale prices and the market prices in the electrical sector.

The training data used were a table with the history of electrical demand for each hour and temperature observations provided by the International Organization for Standardization (ISO) of Great Britain. The meteorological information includes the dry bulb temperature and the dew point. We took into account the data of the hourly electrical demand.

For the electrical demand prediction, we took into account eight characteristics to tune the neural network:

• The dry bulb temperature;
• The dew point;
• Hour of the day;
• Day of the week;
• A mark indicating if this is a free or a weekend day;
• Medium load of the past day;
• The load of the same hour, in the past day;
• Load of the same hour, the same day of the past week.

Further, we utilized the load of the same day as the target.

• Using the training data ($34800\times 8$), we trained the neural network for electrical demand prediction. After the training stage of the neural network, we used $8770$ datapoints for the testing for each characteristic, yielding a matrix with dimensions ($8770\times 8$)
• The neural network had three layers—one input layer, one hidden layer, and one output layer. The input layer had eight neurons, the hidden layer had six neurons, and the output layer had one neuron.

We obtained neural network tuning by using the Hessian with the following steps:

• We initialized the scale parameters with random values between $0$ and $1$;
• We obtained the forward propagation;
• We obtained the cost map;
• We obtained the back propagation;
• We utilized the Hessian tuning.

To evaluate the tuning of the neural network, we employed the determination coefficient ($R^2$), the mean absolute error ($MAE$), and the mean absolute percent error ($MAPE$), determined as follows:

$$\begin{array}{c}{R}^2=1-\frac{{\displaystyle {\displaystyle \sum }_{l=1}^{L_T}}{\left(q_l-\overline{t_l}\right)}^2}{{\displaystyle {\displaystyle \sum }_{l=1}^{L_T}}{\left(t_l-\overline{t_l}\right)}^2}\\ MAE={\displaystyle \sum }_{l=1}^{L_T}\left|q_l-t_l\right|\\ MAPE=\frac{100}{L_T}{\displaystyle \sum }_{l=1}^{L_T}\left|q_l-t_l\right|\end{array}$$

(25)

where $q_l$ is the neural network output, $t_l$ is the target, and $\overline{t_l}$ is the mean of the target. $R^2$ generates values from $0$ to $1$; $L_T$ is the total output number. If a method provides good tuning, it has $R^2$ values near to $1$, and if a method provides bad tuning, it has $R^2$ values near to $0$. If a method provides good tuning, it has $MAE$ values near to $0$ MWh, and if a method provides good tuning, it has $MAPE$ values near to $0\%$. We also used the cost map $E$ (2) to evaluate the tuning of the neural network. If a method provides good tuning, it has $E$ values near to $0$.

Results of the Comparison

It should be noted that the neural network trained by steepest descent (SD) (19) had $l=6$ neurons in its hidden layer, a tuning factor of $\alpha =0.0004$, and a number of epochs of $e=40$.

It should be noted that the neural network trained by steepest descent with mini-batches (SDMB) in [9,10,11,12] and using (19), (22), (23) had $l=6$ neurons in its hidden layer, mini-batches with a size of $y=32$, a tuning factor of $\alpha =0.0004$, and a number of epochs of $e=40$.

It should be noted that the neural network trained by the Hessian (H) in [17,18,19,20] and using (15), (18), (14), (9), (10) had $l=6$ neurons in its hidden layer, a tuning factor of $\alpha =0.0004$, and a number of epochs of $e=40$.

It should be noted that the neural network trained by the Hessian with mini-batches (HMB) in this investigation and using (15), (18), (14), (9), (10), (22), (23) had $l=6$ neurons in its hidden layer, mini-batches with a size of $y=32$, a tuning factor of $\alpha =0.0004$, and a number of epochs of $e=40$.

Figure 2 shows the cost maps during the training of the neural network with steepest descent (SD), steepest descent with mini-batches (SDMB), Hessian (H), and Hessian with mini-batches (HMB). As we can see, the Hessian with mini-batches provides better tuning when it comes to training the neural network and tends to converge more directly (with the help of the information provided from the second derivative) than with the use of the steepest descent. The issue with the normal downward steepest descent is that often a minimum cannot be quickly found. The use of mini-batches helps to quickly reach a minimum.

Figure 3 shows a zoom of the cost maps after $40$ epochs during neural network training using SD, SDMB, H, and HMB. The Hessian provides better tuning in comparison with steepest descent, and the Hessian with mini-batches provides better tuning in comparison with steepest descent with mini-batches.

The tuning of the neural network with SD, SDMB, H, and HMB during training is shown in Figure 4. During $40$ epochs, the neural network trained with steepest descent failed to tune, and its tuning was very slow when compared to the neural network trained with Hessian and mini-batches, which provided better tuning. The Hessian provided better tuning than steepest descent.

The neural network tuning using SD, SDMB, H, and HMB during testing is shown in Figure 5. Similar to the training results, the neural network trained using steepest descent did not have the ability to predict. The neural network prediction using the Hessian with mini-batches was better than that using the other methods, as can be seen in Figure 6, which is a zoom of the neural network prediction with SD, SDMB, H, and HMB during testing.

Table 1. Comparison results in terms of $R^2$ and cost ($E$).

	${\mathit{R}}^2$ (Training)	${\mathit{R}}^2$ (Testing)	$\mathit{E}$ (Training)
SD	0.306	0.107	0.032
SDMB	0.875	0.891	0.0031
H	0.298	0.257	0.0086
HMB	0.882	0.897	0.0014

compares the results of SD, SDMB, H, and HMB during training and testing with $40$ epochs in terms of the determination coefficient ($R^2$) and cost ($E$). It should be noted that the neural network had $6$ neurons in its hidden layer and each mini-batch had a size of $y=32$.

$R^2$ has values between $0$ and $1$, where values close to $1$ correspond to algorithms with better tuning. Since HMB obtained the biggest value of $R^2$ during training and testing and obtained the smallest value of $E$ during training, HMB provides the best tuning in comparison with H, SDMB, and SD.

Table 2. Comparison results in terms of mean absolute error ($MAE$) and mean absolute percent error ($MAPE$).

	$\mathit{M}\mathit{A}\mathit{E}$ (Testing)	$\mathit{M}\mathit{A}\mathit{P}\mathit{E}$ (Testing)
SD	2396.78 MWh	16.54%
SDMB	699.39 MWh	4.85%
H	1888.61 MWh	14.50%
HMB	681.42 MWh	4.77%

compares the results of SD, SDMB, H, and HMB during training and testing for $40$ epochs in terms of the mean absolute error ($MAE$) and the mean absolute percent error ($MAPE$).

Smaller values of $MAE$ and $MAPE$ correspond to algorithms with better tuning. Since HMB obtained the smallest values of the $MAE$ and $MAPE$ during testing, HMB provides the best tuning in comparison with H, SDMB, and SD.

As we decrease the mini-batch size, we speed up the training of the algorithm, but we also increase the computation cost. This means a trade-off between computation cost and training speed.

5. Conclusions

Our goal in this article was to design the Hessian with mini-batches to get better tuning than steepest descent for a neural network. The Hessian with mini-batches was compared with steepest descent, steepest descent with mini-batches, and the Hessian for electrical demand prediction; since we reached the nearest approximation between the neural network output and the target and reached the smallest value of the cost map using the proposed algorithm, we got the best tuning with our proposed algorithm. In future work, we will find the convergence of the Hessian with mini-batches, we will propose other algorithms different to the Hessian to compare our results, and we will apply our algorithm for the prediction of other processes.