Shallow Fully Connected Neural Network Training by Forcing Linearization into Valid Region and Balancing Training Rates

Abstract

A new supervisory training rule for a shallow fully connected neural network (SFCNN) is proposed in this present study. The proposed training rule is developed based on local linearization and analytical optimal solutions for linearized SFCNN. The cause of nonlinearity in neural network training is analyzed, and it is removed by local linearization. The optimal solution for the linearized SFCNN, which minimizes the cost function for the training, is analytically derived. Additionally, the training efficiency and model accuracy of the trained SFCNN are improved by keeping estimates within a valid range of the linearization. The superiority of the proposed approach is demonstrated by applying the proposed training rule to the modeling of a typical nonlinear pH process, Boston housing prices dataset, and automobile mileage per gallon dataset. The proposed training rule shows the smallest modeling error and the smallest iteration number required for convergence compared with several previous approaches from the literature for the case study.

1. Introduction

Artificial neural networks (ANNs) have been widely used in many research areas [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]. One of the most popular ANNs for modeling dynamic systems is the shallow fully connected neural network (SFCNN). In theory, it is capable of describing any complex nonlinear dynamics accurately using a sufficient number of hidden nodes [15]. Despite its capability, its application is limited by a huge computational load and convergence problems during supervisory training. Many researchers have developed a variety of supervisory training rules using gradient descent, Levenberg–Marquardt, quasi-Newton, and conjugate direction methods [16,17,18,19,20,21].

Gradient descent is one of the most popular algorithms for performing optimizations and is currently the most common method of training neural networks due to its effectiveness in dealing with enormous data. However, it inherently requires a massive number of learning iterations and has disadvantages in that it is vulnerable to getting stuck in saddle points under non-convex optimization problems [19,22]. Some algorithms were proposed to deal with this problem using momentum and adaptive learning rate, including the Nesterov accelerated gradient (NAG) [20], root mean square propagation (RMSProp), and adaptive moment estimation (ADAM) [21]. Although these approaches are widely used in neural network training and have proven their performance, their slow convergence remains a challenge for real-time applications [23]. An optimization method based on Hessian of training loss (e.g., Newton’s method and the Levenberg–Marquardt method) is free from slow convergence from massive iteration and getting stuck in a saddle point. Nevertheless, difficulty in computing exact Hessian eigenvalues of a large size model [24] places a method outside the mainstream of neural network training. In order to get over this hurdle, research teams proposed a diverse study focusing on computing a limited number of eigenvalues [25,26,27] and estimating information (i.e., curvature, sharpness, etc.) in the absence of sufficient information about an eigenvalue spectrum [28,29,30,31,32,33,34]. In our previous study, a training method to reduce the searching space as much as the number of the output weights in the high-dimensional nonlinear optimization problem has been proposed to alleviate the computational load [35]. Although this approach shows remarkable improvement compared with full-dimensional optimization approaches, much room remains to be improved as it cannot analytically estimate the optimal input weights.

In this paper, an improved supervisory training rule for an SFCNN using local linearization and the analytic optimal solution of a linearized SFCNN is proposed. First, the cause of nonlinearity in neural network training is analyzed, and it is locally linearized. Next, the optimal solution is derived that minimizes the cost function for training. A new cost function is proposed to keep the estimate within a valid domain of linearization. A new training rule is finally derived based on the derived optimal solution. The performance of the proposed approach is verified by applying the proposed training rule to the modeling of a typical nonlinear pH process, Boston housing prices dataset, and automobile mileage per gallon dataset. The proposed method is compared with four widely used methods: the gradient descent with momentum, stochastic gradient descent, Levenberg–Marquardt methods, and adaptive moment estimation. From the simulation results, it can be confirmed that the iteration number and the modeling error are the smallest when compared to the existing approaches.

The remaining contents of this manuscript are represented as follows: In Section 2, the proposed training method is introduced and derived based on local linearization and analytical optimal solution. Section 3 presents the performance of the proposed method compared with previous methods, including the gradient descent with momentum, the stochastic gradient descent, the adaptive momentum algorithm, and the Levenberg–Marquardt method. Finally, the conclusion and further extension of the proposed method are discussed in Section 4.

2. Proposed Supervisory Training Rule for SFCNN

Let us consider the SFCNN structure represented in Figure 1. The neural network is mathematically formulated as follows:

$${\widehat{z}}_j(q)={\displaystyle \sum _{i=1}^m{\nu }_{j,i}{u}_i(q)}, j=1, 2, 3, \cdots , l$$

(1)

$${\hat{h}}_j(q)=\frac{1}{1+\exp (-\alpha {\widehat{z}}_j(q))}, j=1, 2, 3, \cdots , l$$

(2)

$${\widehat{y}}_k(q)={\displaystyle \sum _{j=1}^l{w}_{k,j}{\hat{h}}_j(q)}, k=1, 2, 3, \cdots , n$$

(3)

where ${\nu }_{j,i}$ and $w_{k,j}$ are the input weight and the output weight, respectively; ${\hat{h}}_j(q)$ and ${\widehat{y}}_k(q)$ denote the output of the j-th hidden node and the k-th output node, respectively; u_i(q) is the input of the i-th input node; $q$ donates the q-th sampling of the process input and output.

We will now analyze the cause of the nonlinearity of the neural network. Note that ${\widehat{z}}_j(q)$ of Equation (1) is linear with respect to the input weight. Meanwhile, ${\hat{h}}_j(q)$ and ${\widehat{y}}_k(q)$ of Equations (2) and (3) are nonlinear with respect to the input weight. It now becomes apparent that the cause of the nonlinearity originates from the activation function of Equation (2). Therefore, it can be concluded that the optimal weights for the training can be analytically estimated if the activation function and the output layer are linearized.

2.1. Derivation of the Proposed Training Rule

In the present study, the following linearization for the activation function is adopted for the training. It is linearized at the estimates of the previous iteration in the supervisory training.

$${\hat{h}}_j(q)\approx {\hat{h}}_j^{lin}(q)={\frac{d{\hat{h}}_j}{d{\widehat{z}}_j}|}_{{\widehat{z}}_j(q)={\widehat{z}}_j^{iter-1}(q)}\left({\widehat{z}}_j(q)-{\widehat{z}}_j^{iter-1}(q)\right)+{\hat{h}}_j^{iter-1}(q)$$

(4)

where a superscript iter is an iteration number in the supervisory training and the superscript lin means a linearized function. The output layer is linearized by Equation (5):

$${\widehat{y}}_k(q)\approx {\widehat{y}}_k^{lin}(q)={\displaystyle \sum _{j=1}^l\left(\left(w_{k,j}-w_{k,j}^{iter-1}\right){\hat{h}}_j^{iter-1}(q)+w_{k,j}^{iter-1}\left({\hat{h}}_j(q)-{\hat{h}}_j^{iter-1}(q)\right)+w_{k,j}^{iter-1}{\hat{h}}_j^{iter-1}(q)\right)}$$

(5)

In addition, the following cost function of the multi-objective quadratic programming problem is used for supervisory training in this research. The proposed method will estimate the input and output weights at each iteration number of $iter$ by minimizing the cost function based on the linearized SFCNN from the previous iteration number of $iter-1$.

$$\begin{array}{ll}\underset{v_{j,i},w_{j,i}}{\min }Q& =0.5{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n{\left(y_k(q)-{\widehat{y}}_k^{lin}(q)\right)}^2}}+0.5R_h{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{j=1}^l{\left({\hat{h}}_j(q)-{\hat{h}}_j^{lin}(q)\right)}^2}}\\ & +0.5R_y{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n{\left({\widehat{y}}_k(q)-{\widehat{y}}_k^{lin}(q)\right)}^2}}+0.5R_v{\displaystyle \sum _{j=1}^l{\displaystyle \sum _{i=1}^m{\left(v_{j,i}^{iter-1}-v_{j,i}\right)}^2}}\\ & +0.5R_w{\displaystyle \sum _{k=1}^n{\displaystyle \sum _{j=1}^l{\left(w_{k,j}^{iter-1}-w_{k,j}\right)}^2}}\end{array}$$

(6)

Here, $y_k(q)$ is the measurement of the process output, and ${\widehat{y}}_k^{lin}(q)$ is the model output of the linearized SFCNN; ${\hat{h}}_j(q)$ and ${\widehat{y}}_k(q)$ denote the output of the hidden node and that of the output node of the SFCNN; ${\hat{h}}_j^{lin}(q)$, ${\widehat{y}}_k^{lin}(q)$ denote the output of the hidden node and that of the output node of the linearized SFCNN. The constants, $R_h$, $R_y$, $R_v$, and $R_w$ to balance the five terms of the right-hand side of the cost function are updated at each iteration, as will be discussed later.

It should be noted that a number of important terms for more efficient training are included in the cost function as follows: The first term on the right-hand side is the square modeling error between the measurement of the process output($y_k(q)$) and the SFCNN model output(${\widehat{y}}_k^{lin}(q)$). Most previous training methods train the SFCNN by minimizing the modeling error term.

The second and third terms are to minimize the distance between the signals of the SFCNN $({\hat{h}}_j(q)$ and ${\widehat{y}}_k(q))$ and those of the linearized SFCNN $({\hat{h}}_j^{lin}(q)$ and ${\widehat{y}}_k^{lin}(q))$ to keep the training within the valid range of the linearization. If the cost function does not include the second and the third term, the weights of the SFCNN obtained by minimizing the cost function would be placed outside of the valid range of the linearization, causing problems, such as the divergence and saturation phenomenon of the activation functions during the training.

The fourth and fifth terms are to minimize the distances between the optimal weights of $iter$ and the weights of $iter-1$ to keep the training from changing the input and output weights too much. These terms play an important role in preventing singularity problems during the training using the Hessian matrix. In most cases, the number of the weights of the SFCNN is huge (equivalent to overparameterization) and easily causes a singularity problem. Therefore, it is a must then to include the fourth and fifth terms in the cost function. The previous training rule using the Levenberg–Marquardt optimization method, known as the fastest method, adds a small diagonal matrix to the Hessian matrix to prevent a singularity problem during the training. The addition of the small diagonal matrix is equivalent to adding the fourth and fifth penalty terms of Equation (6) in the cost function, keeping the estimates from changing too much [36,37]. However, the Levenberg–Marquardt method has no direct instruments to guarantee the validity of the linearization because it does not consider the terms, such as the second and the third penalty terms, during the training. Thus, it cannot overcome the potential problem that training goes too far because of an unacceptable accuracy of linearization. This is one of the significant differences between the proposed method and previous approaches.

Note, however, that the optimal solution of Equation (6) can be estimated analytically because the activation function and the output layer are linearized as Equations (4) and (5). In order to derive the analytic solution, consider the following equations derived from Equations (1) and (4)–(6).

$$\begin{array}{l}\frac{\partial Q}{\partial v_{a,b}}=-{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\left(y_k(q)-{\widehat{y}}_k^{lin}(q)\right)}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}}-R_v{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{j=1}^l\left({\hat{h}}_j(q)-{\hat{h}}_j^{lin}(q)\right)}}\frac{\partial {\hat{h}}_j^{lin}(q)}{\partial v_{a,b}}\\ \hspace{1em}\hspace{1em}-R_w{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\left({\widehat{y}}_k(q)-{\widehat{y}}_k^{lin}(q)\right)}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}}+A\\ A=-R_v{\displaystyle \sum _{j=1}^l{\displaystyle \sum _{i=1}^m(v_{j,i}^{iter-1}-v_{j,i}) \mathrm{if} }}j=a \mathrm{and} i=b, A=0 \mathrm{if} j\ne a \mathrm{or} i\ne b\end{array}$$

(7)

$$\begin{array}{l}\frac{\partial Q}{\partial w_{a,b}}=-{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\left(y_k(q)-{\widehat{y}}_k^{lin}(q)\right)}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{a,b}}-R_w{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\left({\widehat{y}}_k(q)-{\widehat{y}}_k^{lin}(q)\right)}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{a,b}}+B\\ B=-R_w{\displaystyle \sum _{k=1}^n{\displaystyle \sum _{j=1}^l\left(w_{k,j}^{iter-1}-w_{k,j}\right) \mathrm{if} k=1 \mathrm{and} j-b, B=0 \mathrm{if} k\ne a \mathrm{or} }}j\ne b\end{array}$$

(8)

$$\begin{array}{ll}\frac{{\partial }^{2}Q}{\partial v_{a,b}\partial v_{c,d}}=& -{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\left(y_k(q)-{\widehat{y}}_k^{lin}(q)\right)}}\frac{{\partial }^2{\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}\partial v_{c,d}}+{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{c,d}}}}\\ & -R_v{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{j=1}^l\left({\hat{h}}_j(q)-{\hat{h}}_j^{lin}(q)\right)}}\frac{\partial {\hat{h}}_j^{lin}(q)}{\partial v_{a,b}\partial v_{c,d}}+R_v{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{j=1}^l\frac{\partial {\hat{h}}_j^{lin}(q)}{\partial v_{a,b}}\frac{\partial {\hat{h}}_j^{lin}(q)}{\partial v_{c,d}}}}\\ & -R_w{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\left({\widehat{y}}_k(q)-{\widehat{y}}_k^{lin}(q)\right)}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}\partial v_{c,d}}+R_w{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{c,d}}}}+C\\ C=R_v \mathrm{if} & k=a=c \mathrm{and} j=b=d, D=0 \mathrm{if} k\ne c, k\ne c, j\ne b, \mathrm{or} j\ne d\end{array}$$

(9)

$$\begin{array}{l}\frac{{\partial }^{2}Q}{\partial w_{a,b}\partial w_{c,d}}=-{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\left(y_k(q)-{\widehat{y}}_k^{lin}(q)\right)}}\frac{{\partial }^2{\widehat{y}}_k^{lin}(q)}{\partial w_{a,b}\partial w_{c,d}}+{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{a,b}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{c,d}}}}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-R_w{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\left({\widehat{y}}_k(q)-{\widehat{y}}_k^{lin}(q)\right)}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{a,b}\partial w_{c,d}}+R_w{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{a,b}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{c,d}}}}+D\\ D=R_w \mathrm{if} k=a=c \mathrm{and} j=b=d, \mathrm{D}=0 \mathrm{if} k\ne a, k\ne c, j\ne b, \mathrm{or} j\ne d\end{array}$$

(10)

$$\begin{array}{l}\frac{{\partial }^{2}Q}{\partial v_{a,b}\partial w_{c,d}}=-{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\left(y_k(q)-{\widehat{y}}_k^{lin}(q)\right)}}\frac{{\partial }^2{\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}\partial w_{c,d}}+{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{c,d}}}}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-R_w{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\left({\widehat{y}}_k(q)-{\widehat{y}}_k^{lin}(q)\right)}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}\partial w_{c,d}}+R_w{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{k=1}^n\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}}\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{c,d}}}}\end{array}$$

(11)

$$\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial v_{a,b}}={\displaystyle \sum _{j=1}^l{w}_{k,j}^{iter-1}\frac{\partial {\hat{h}}_j^{lin}(q)}{\partial v_{a,b}}}$$

(12)

$$\frac{\partial {\hat{h}}_j^{lin}(q)}{\partial v_{a,b}}={\frac{d{\hat{h}}_j}{d{\widehat{z}}_j}|}_{{\widehat{z}}_j(q)={\widehat{z}}_j^{iter-1}(q)}\frac{\partial {\widehat{z}}_j^{lin}(q)}{\partial v_{a,b}}$$

(13)

$$\frac{\partial {\widehat{z}}_j^{lin}(q)}{\partial v_{a,b}}=u_b(q) \mathrm{if} j=a, \frac{\partial {\widehat{z}}_j^{lin}(q)}{\partial v_{a,b}}=0 \mathrm{if} \mathrm{j}\ne a$$

(14)

$$\frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{a,b}}={\hat{h}}_b^{iter-1}(q) \mathrm{if} k=a, \frac{\partial {\widehat{y}}_k^{lin}(q)}{\partial w_{a,b}}=0 \mathrm{if} k\ne a$$

(15)

From Equations (14) and (15), it is clear that the second derivatives of ${\widehat{z}}_j^{lin}(q)$ and ${\widehat{y}}_k^{lin}(q)$ are zeros, which means that all the derivatives of Q more than twice are zeroes when we consider Equations (7)–(13). Therefore, the cost function Q becomes a quadratic form in terms of the input and output weights, resulting in the following quadratic equation:

$$\nabla Q={{\nabla }^{2}Q|}_{iter-1}(\theta -{\theta }^{iter-1})+{\nabla Q|}_{iter-1}$$

(16)

where $\theta$ is a vector composed of the input and the output weights. The $\nabla Q$ and ${\nabla }^{2}Q$ are the gradient and Hessian matrix of Q, respectively. Finally, the analytic optimal solution for the cost function of Equation (6) can be derived from Equation (16) by setting $\nabla Q=0$. Then, the following training rule for the cost function is obtained from the optimal solution.

$${\theta }^{iter}={\theta }^{iter-1}-{\left({{\nabla }^{2}Q|}_{iter-1}\right)}^{-1}{\nabla Q|}_{iter-1}$$

(17)

All the numerical problems in adopting the cost function of Equation (6) except determining the constants of $R_h$, $R_y$, $R_v$, and $R_w$, are now solved.

2.2. Determining Hyperparameters

The constant, $R_h$, $R_y$, $R_v$, and $R_w$, are update each iteration by:

$$R_{v,iter}=R_{iter}\frac{{\displaystyle \sum _{i=0}^{i=iter}\left({\Vert \partial Q/\partial v\Vert }_{F,i}^2{\lambda }^{iter-i}\right)}}{{\displaystyle \sum _{i=0}^{i=iter}\left({\Vert \partial Q/\partial w\Vert }_{F,i}^2{\lambda }^{iter-i}\right)}+{\displaystyle \sum _{i=0}^{i=iter}\left({\Vert \partial Q/\partial v\Vert }_{F,i}^2{\lambda }^{iter-i}\right)}}$$

(18)

$$R_{w,iter}=R_{iter}\frac{{\displaystyle \sum _{i=0}^{i=iter}\left({\Vert \partial Q/\partial w\Vert }_{F,i}^2{\lambda }^{iter-i}\right)}}{{\displaystyle \sum _{i=0}^{i=iter}\left({\Vert \partial Q/\partial w\Vert }_{F,i}^2{\lambda }^{iter-i}\right)}+{\displaystyle \sum _{i=0}^{i=iter}\left({\Vert \partial Q/\partial v\Vert }_{F,i}^2{\lambda }^{iter-i}\right)}}$$

(19)

$$R_{h,iter}=R_{v,iter}$$

(20)

$$R_{y,iter}=R_{w,iter}$$

(21)

where ${\Vert •\Vert }_F$ and ${\lambda }^{iter-i}<1$ donate the Frobenius norm and a forgetting factor, respectively. The subscript iter is the iteration number. $R_{iter}$ controls the overall speed of the convergence. If $R_{iter}$ is larger, the convergence rate decreases and the robustness increases, and vice versa. In this research, $R_{iter}$ is updated as follows: if the modeling error of the iter iteration decreases compared to that of the iter−1 iteration, it decreases $R_{iter}$ by $R_{iter}=R_{iter-1}/\alpha , \alpha >1$, and updates ${\theta }^{iter}$ by Equation (17) to increase the convergence rate. Otherwise, it increases $R_{iter}$ by $R_{iter}=\beta R_{iter-1}, \beta \hspace{0.17em}>\hspace{0.17em}1$ to increase the robustness and does not update ${\theta }^{iter}$ as ${\theta }^{iter}={\theta }^{iter-1}$. This is similar to the Levenberg–Marquardt method. However, the update of $R_{v,iter}$ and $R_{w,iter}$ is totally different. The update rule of Equations (18) and (19) play an important role in equalizing the two gradients of the cost function with respect to the input weights and the output weights. If the gradient for the output weights of ${\Vert \partial Q/\partial w\Vert }_F^2$ is bigger than the gradient for the input weights of ${\Vert \partial Q/\partial v\Vert }_F^2$, the proposed method adjusts the weights toward equalizing the two gradients by penalizing the movements of the output weights more than those of the input weights according to Equations (18)–(21). This feature can improve the efficiency of the training significantly because unbalanced training between the input weights and the output weights slows down the convergence rate and causes saturation of the activation function, resulting in unacceptable local minima. This is the other uniqueness of the proposed method. The previous training rules, such as gradient descent and Levenberg–Marquardt, cannot incorporate this feature

3. Results and Discussions

In order to verify the performance of the proposed training method, the study performed three case studies, including a highly nonlinear pH process, the Boston house price and automobile mileage per gasoline datasets, which are widely used in neural network training problems. In the present study, the performances of four different training methods, including the gradient descent with momentum (GDM), the Levenberg–Marquardt (LM), the stochastic gradient descent (SGD), and the adaptive momentum (ADAM) algorithms, are compared with the proposed method. All the neural network pieces of training are conducted via MATLAB. In the present study, the learning rate scheduler with an initial learning rate of 0.001, provided in the MATLAB embedded function, is used for the GDM, SGD, and ADAM methods. For the LM and the proposed methods, the initial damping parameter ($R_0$) of 1.0 and the update factors, $\alpha$ and $\beta$ of 1.2 and 10.0 are used, respectively.

3.1. A pH Process

A pH process known as one of the highly nonlinear chemical processes is simulated to demonstrate the advantages of the proposed method compared with previous ones. Figure 2 shows the pH process where the weak acid influent of phosphoric acid (H₃PO₄) is treated by the strong base of sodium hydroxide (NaOH) in a continuously stirred tank reactor. Here, the feed flow rate (F) and the reactor working volume (V) are 2 L/min and 10 L, respectively. The total concentrations of the phosphoric acid in the influent stream and the sodium hydroxide in the titrating stream are 0.05 and 0.2 mol/L, respectively. The initial total ion concentrations of the phosphoric acid and the sodium hydroxide in the reactor are 0.05 and 0.05 mol/L, respectively. For the detailed material balance equations and an equilibrium, equations are referred to previously published works [38].

The process is perturbed by the titrating stream (u) of uniformly distributed random noises between 0 and 1 L/min, and the sampling time is 1 min. This study assumes that the obtained pH data is corrupted by measurement noises of ±0.05. The data used in the present study is represented in Figure 3. Moreover, u and pH data before the 525th samples are used for training, and data after the 525th samples are used for validation. The input nodes of the neural network consist of u(t − 1), u(t − 2), pH(t − 1)/12, pH(t − 2)/12, and one bias. The output of the neural network is pH(t)/12. The number of hidden nodes is 3 or 7. The training of a pH process is performed for 20 cases with randomly generated input data to verify the performance of the proposed method more reliably.

Figure 3 and Figure 4 show one of the training results. As illustrated in Figure 3c, the proposed method has a higher modeling accuracy for the pH process under training and validation results. The comparison result between the proposed method and previous methods is represented in Figure 4. As confirmed in Figure 4, the mean square error (MSE) values of the previous approach (except the LM method) after 200 iterations are much bigger than those of the proposed method after just 10 iterations. The study multiply performs neural network training for the pH process in order to verify a reliable performance. The results of training error, the number of iterations to converge, and the computation time obtained from the repeated simulation (20 times with different datasets generated from simulations with randomly generated input values) are summarized in

Table 1. Training performance for the pH process after 100,000 iterations—training errors (mean squared error; MSE).

Method	Hidden Layer with 3 Nodes		Hidden Layer with 7 Nodes
	Mean	Standard Deviation	Mean	Standard Deviation
GDM	7.319 × 10−3	4.417 × 10−4	4.046 × 10−3	1.529 × 10−4
SGD	1.697 × 10−3	7.664 × 10−5	1.681 × 10−3	7.235 × 10−5
ADAM	1.668 × 10−3	7.142 × 10−5	1.668 × 10−3	7.142 × 10−5
LM	1.201 × 10−3	3.603 × 10−4	3.068 × 10−4	6.079 × 10−5
Proposed	1.140 × 10−3	3.284 × 10−4	2.825 × 10−4	4.940 × 10−5

and

Table 2. Training performance for the pH process after 100,000 iterations—number of iterations and computation time of hidden layer with 7 nodes.

Method	Number of Iterations to Converge		Computation Time to Converge (s)
	Mean	Standard Deviation	Mean	Standard Deviation
GDM	94,494	843.1	130.4	16.4
SGD	6318.3	2815.3	29.0	12.5
ADAM	637.57	180.25	3.82	1.23
LM	145.55	63.79	8.68	3.74
Proposed	107.40	44.99	7.94	3.30

. As enumerated in Table 1, the proposed method shows an outstanding modeling performance when comparing the training errors after 100,000 iterations. In order to make a reasonable comparison of the convergence rate between the diverse methods, this study compares the number of iterations and computational time for the convergence of reaching within ±1.0% of the MSE value in Table 1. The required number of iterations and computation time (except ADAM) is smaller than the previous methods, as represented in Table 2. Despite the fact that ADAM can compute many iterations in a very short time, it has a very poor modeling performance compared to the proposed method. Consequently, it can be concluded that the proposed method provides the fastest convergence rate and the smallest modeling error compared to all the previous approaches.

3.2. Representative Datasets for Neural Network Training: Boston Housing Price and Automobile Mileage Per Gallon

To ensure the general performance of the proposed method, the study performs SFCNN training with the Boston housing prices [39] dataset and automobile mileage per gallon (MPG) [40] dataset, which are commonly used datasets for testing neural network training. This study uses 75% and 25% of the dataset as the training set and validation set, respectively. Data for training and validation are randomly selected. The used neural network takes 20 nodes into account. The training results of both the Boston housing prices and automobile MPG are depicted in Figure 5.

As represented in Figure 5 of both cases, the proposed training method shows the fastest and most stable convergence. The proposed method can achieve a better result than the final results of GDM, SDG, and ADAM just within 10 iterations. In order to analyze the reliability of the training performance, the training is performed with 20 cases obtained from a randomly chosen training dataset. The average and standard deviation values of the training error and the number of iterations required for convergence are summarized in

Table 3. Boston housing prices and automobile MPG training performances using early stopping (validation patient of 15)—training errors (MSE).

Method	Boston Housing Prices		Automobile MPG
	Mean	Standard Deviation	Mean	Standard Deviation
GDM	Not converge	Not converge	Not converge	Not converge
SGD	1.127 × 10−3	1.066 × 10−3	1.405 × 10−3	1.215 × 10−3
ADAM	5.858 × 10−4	4.368 × 10−4	6.417 × 10−4	7.005 × 10−5
LM	1.001 × 10−4	2.653 × 10−5	2.477 × 10−4	4.010 × 10−5
Proposed	7.393 × 10−5	1.649 × 10−5	2.407 × 10−4	2.990 × 10−5

and

Table 4. Boston housing prices and automobile MPG training performances using early stopping (validation patient of 15)—number of iterations.

Method	Boston Housing Prices		Automobile MPG
	Mean	Standard Deviation	Mean	Standard Deviation
GDM	Not converge	Not converge	Not converge	Not converge
SGD	4330	5123	2833	4437
ADAM	4097	2579	3156	5506
LM	29.9	2.4	30.7	3.3
Proposed	26.6	2.0	29.1	2.3

. The case study uses early stopping, where training stops at the point where the validation performance starts to degrade continuously over 15 iterations. It can be confirmed that the proposed approach shows overwhelming training performances, especially compared with GDM, SGD, and ADAM.

4. Conclusions

A novel supervisory training rule for shallow, fully connected neural networks (SFCNN) using local linearization and an analytic optimal solution for a linearized SFCNN is proposed. The proposed cost function includes a number of important terms to keep the training within a valid region of the linearization and to prevent the training from changing the input and output weights too much, solving problems, such as divergence, singularity, and the saturation phenomenon of the activation functions. The performance of the proposed approach is verified by applying it to a highly nonlinear pH process, the Boston housing price dataset, and the automobile mileage per gallon dataset. It shows a superior modeling performance with the smallest modeling error and the smallest iteration number required for convergence compared with the previous approaches for the case study.

This research took a shallow neural network with just one hidden layer into account for developing the training methods. Currently, the research area of machine learning is expanding to various fields, such as computer vision and natural language processing. In accordance with this purpose, a deep neural network with a large number of hidden layers is generally used. Thus, as a future work, the proposed method will be extended to DNN.