Research on an Accuracy Optimization Algorithm of Kriging Model Based on a Multipoint Filling Criterion

Abstract

The optimization method based on the surrogate model has been widely used in the simulation and calculation process of complex engineering models. However, in this process, the low accuracy and computational efficiency of the surrogate model has always been an urgent problem that needs to be solved. Aimed at this problem, combined with the two characteristics of global search and local detection, a filling criterion with multiple points is firstly proposed named maximum of expected improvement & minimizing the predicted objective function & maximum of root mean squared error (EI&MP&RMSE) in this paper. Furthermore, the optimization procedure of the surrogate model based on EI&MP&RMSE is concluded. Meanwhile, the classical one-dimensional and two-dimensional functions are applied to verify the accuracy of the proposed method. The difference in the accuracy and mean square error of the surrogate model under different infill points criteria are analyzed. As expected, it shows that this method can effectively improve the accuracy of the surrogate model and reduce the number of iterations. It has extensive practicability and serviceability for the optimization of complex engineering structures.

1. Introduction

In modern society, models of engineering cases are complex, and it costs a lot of computing resources and time to solve them. Aimed at such problems, it is meaningless to use traditional design tools and optimization algorithms. Therefore, for the sake of improving the computational efficiency, the optimization algorithm based on the surrogate model has been introduced and applied, such as the polynomial response surface method (RSM) [1], radial basis function method (RBF) [2] and Kriging method [3]. The function form of the Kriging method is relatively complex among them, which is mainly suitable for nonlinear problems. The Kriging model was firstly proposed by geologists in South Africa. At that time, it was mainly used in geological circles to estimate the distribution of mineral reserves. It is an unbiased estimation model with minimum estimation variance based on random process. Jerome Sacks [4] applied the Kriging model to deterministic computer experimental design and analysis for the first time and gave a practical Kriging model. Subsequently, the Kriging model has been extensively used in environmental science, hydrogeology, automobile, aerospace and other engineering fields [5,6,7,8]. Additionally, scholars studied the approximation and optimization methods of the Kriging model. Jones [9,10] constructed the expected improvement function (EI) by using the function estimation and the root mean square error of estimation and proposed an effective global optimization method (EGO) by maximizing the expected improvement criterion. Sasena et al. [11] introduced several filling sampling criteria and described in detail how they affected the global or local search behavior. Furthermore, the generalized EI was advanced. Aimed at how to balance the relationship between global search and local detection of the model, Sobester et al. [12] proposed the selection criterion based on the expected improvement criterion, which improved the accuracy of the search range. Huang et al. [13] put forward the augmented expected improvement, which effectively addressed stochastic black-box systems. W Ponweiser et al. [14] introduced a new method named the Dasia clustered multiple generalized expectation improvement scheme (CMGEI), this method improved the global and local search behavior. Chaudhuri et al. [15,16] proposed an adaptive target setting method, which added multiple filling sampling points in each cycle. Parra et al. [17] used Pareto’s optimal boundary to obtain the filling sampling criteria of multiple update sampling points and proposed an improved multipoint EGO optimization strategy. Yi et al. [18] proposed an improved EGO method, which used multi-objective filling sampling criteria to effectively propose its own optimization ability. AIJ Forrester et al. [19] compared the Minimum Predicted objective function minimum criterion (MP) and the root mean square error maximum criterion (RMSE). The results showed that MP criterion quickly found the local minimum, and RMSE criterion performed better in improving generalization than simply using space filled samples with more times. Li et al. [20] proposed a multipoint plus point criterion named EI&MP, which could optimize the relationship between global and local optimization. Hamza et al. [21] proposed a simulation-based optimization design framework, which allowed multiple samples to be generated at one time and improves the accuracy of the surrogate model by adding sample point data.

Nowadays, with the rapid development of the super parallel computing system, we can solve high-dimensional optimization problems with a large number of local optima more efficiently [22]. Different from the research studies above, this paper firstly proposes a multiple addition criterion, which is named EI&MP&RMSE. Based on the Kriging optimization method, the EI infill point criterion, the RMSE infill point criterion and the MP infill point criterion are introduced simultaneously. This method makes up for the shortcomings of the surrogate model, reduces the number of iterations and improves the global accuracy and efficiency of the established surrogate model. Section 2 introduces the basic principle of the Kriging model and three addition criteria. On this basis, the principle of multiple addition points and the optimization process are put forward. In Section 3, the effectiveness of the proposed method is verified by two classical mathematic cases from two aspects of model accuracy and mean square error. The results show that the method can improve the accuracy of the surrogate model and reduce the number of iterations.

2. Kriging Model of Point Infill Criterion

2.1. Kriging Model

The Kriging method is suitable for constructing approximate models for highly nonlinear response values. Additionally, it is based on the position and correlation degree of known samples to calculate linear unbiased variables with the smallest estimation error in the new region. It is composed of global approximate model and local model deviation:

$$y(x)=f(x)+z(x)={\displaystyle \sum _{i=1}^p{\beta }_i{f}_i(x)+z(x)}$$

(1)

where f(x) is the polynomial, f_i(x) is the basis function, β_i is the regression coefficient, p is the number of regression vectors and z(x) is deduced by a normally distributed Gaussian random process with mean value 0, variance σ² and non-zero covariance.

At different positions in the design space, the covariance between the random quantities corresponding to z(x) can be expressed as [23]:

$$Cov[z(x^i),z(x^j)]={\sigma }^2[R(x^i,x^j)]$$

(2)

$$[R(x^i,x^j)]={\displaystyle \prod _{k=1}^n{R}_k(x_k^i-x_k^j,{\theta }_k)}$$

(3)

where R (xⁱ − x^j, θ_k) is the space correlation coefficient of x and θ_k. Additionally, it represents the correlation between random variables at different spatial distances. When the distance is zero, R = 1. When the distance is infinite, R = 0. θ_k is an undetermined coefficient. n is the number of design sample points. The regression coefficient and variance can be obtained by maximum likelihood estimation. It is assumed that the response value generated at any position conforms to the normal distribution N (f^Tβ, ${\sigma }_z^2$), the corresponding values $y_s$ of sample points can be shown as:

$$y_s=F\beta +Z$$

(4)

$$y_s={[y^{(1)},y^{(2)},\cdots ,y^{(n)}]}^T$$

(5)

$$F={[f^T(x^{(1)}), f^T(x^{(2)}),\cdots , f^T(x^{(n)})]}^T$$

(6)

$$f(x)={[f_1(x),f_2(x),\cdots f_p(x),]}^T$$

(7)

$$Z={[z(x^{\left(1\right)}),z(x^{\left(2\right)}),\cdots z(x^{\left(n\right)}),]}^T$$

(8)

$$\beta ={[{\beta }_1,{\beta }_2,\cdots {\beta }_p,]}^T$$

(9)

where F is the matrix of n × p, which is related to f(x). Then, there is the probability density at each sample point and the likelihood function of the sample set:

$$pdf(y_s;V)=\frac{1}{{\sigma }_z\sqrt{2\pi }}\exp (u)$$

(10)

$$u=-\frac{{(y^{(i)}-f{\left(x\right)}^T\beta )}^2}{2{\sigma }_z^2}$$

(11)

$$L(V)=\frac{1}{\sqrt{{\left({\sigma }_z^2\right)}^n{(2\pi )}^n(\det R)}}\exp \left(-\frac{{(y_s-F\beta )}^T{R}^{-1}(y_s-F\beta )}{2{\sigma }_z^2}\right)$$

(12)

where ${\sigma }_{\mathrm{z}}^2$ is an unknown variance. Both sides of the likelihood function in Equation (12) are taken logarithms, and the likelihood function obtains the maximum value, namely the following conditions are met:

$$\frac{\partial \left(\ln L(V)\right)}{\partial \beta }=\frac{2F^T{R}^{-1}\left(y_s-F\beta \right)}{2{\sigma }_z^2}=0$$

(13)

$$\frac{\partial \left(\ln L(V)\right)}{\partial {\sigma }_z^2}=-\frac{n}{2{\sigma }_z^2}+\frac{{\left(y_s-F\beta \right)}^T{R}^{-1}{\left(y_s-F\beta \right)}^{\prime }}{2{\sigma }_z^4}=0$$

(14)

The regression coefficient and variance can be solved from Equations (13) and (14) [24]:

$$\widehat{\beta }={(F^T{R}^{-1}F)}^{-1}{F}^T{R}^{-1}{y}_s$$

(15)

$${\widehat{\sigma }}_{{}^z}^2={\frac{\left(y_s-F\widehat{\beta }\right)}{n}}^T{R}^{-1}\left(y_s-F\widehat{\beta }\right)$$

(16)

By introducing Equations (15) and (16) into Equation (12), the expression of concentrated logarithmic likelihood function can be indicated:

$$CLLF=-\frac{n}{2}\ln ({\widehat{\sigma }}_z^2)-\frac{1}{2}\ln (\left|R\right|)-\frac{n}{2}\ln (2\pi )$$

(17)

$${\theta }_k=\max \left(-\frac{1}{2}[n\ln ({\widehat{\sigma }}_z^2)+\ln \left|R\right|]\right)$$

(18)

where the undetermined coefficient θ_k can be settled by the maximum likelihood estimation method.

After resolving the relevant parameters of the known sample points, the response value and variance of the unknown points can be obtained by using the Kriging model [25]

$$\widehat{y}(x)=f_x^T\widehat{\beta }+r_x^T{R}^{-1}(y_s-F\widehat{\beta })$$

(19)

$$MSE[\widehat{y}(x)]={\widehat{\sigma }}_z^2\left(1-r_x^T{R}^{-1}{r}_x+{\left(F^T{R}^{-1}{r}_x-f_x\right)}^T{\left(F^T{R}^{-1}F\right)}^{-1}\left(F^T{R}^{-1}{r}_x-f_x\right)\right)$$

(20)

$$r_x={[R(x^{(1)},x),R(x^{(2)},x),\cdots ,R(x^{(n)},x)]}^T$$

(21)

2.2. Infilling Points Criterion

The point adding criterion is the selection of new sample points in the process of objective optimization, which directly determines the accuracy and efficiency of the surrogate model. According to the properties of global and local exploration, the point adding criteria can be divided into pure exploitation, pure exploration and balanced exploration.

2.2.1. Maximum of Expected Improvement (EI)

The EI criterion can not only provide the function prediction value at any position, but also estimate the error. The response value predicted by the surrogate model should satisfy the normal distribution:

$$Y(x)\in N[\widehat{y}(x),s^2]$$

(22)

With regard to the minimization problem, $y_{\min }$ is the optimal value in the current experimental space, and the improvement of the optimal value at x can be represented as:

$$I(x)=\max \{y_{\min }-Y(x)\}$$

(23)

Owing to I(x) follows the normal distribution, mathematical expectation of I(x) can be obtained as follows [26]:

$$E[I(x)]=\left\{\begin{array}{ll}(y_{\min }-\widehat{y}(x))\varphi \left(\frac{y_{\min }-\widehat{y}(x)}{s}\right)+s\phi \left(\frac{y_{\min }-\widehat{y}(x)}{s}\right)& \hfill s>0\\ 0\hfill & \hfill s\le 0\end{array}\right.$$

(24)

where $\varphi$ function composites standard normal cumulative distribution, $\phi$ is probability density of the standard normal distribution function. Additionally, s is the standard deviation of the generated agent model.

EI [9] is an effective global optimization method, it still has some challenges and needs further improvement. Firstly, if EI changes greatly, it provides difficulty for the later optimization. In addition, if the distribution of initial sample points is inhomogeneous, it will reduce the convergence speed and accuracy.

2.2.2. Minimizing the Predicted Objective Function (MP)

On the premise of high accuracy of the surrogate model, MP directly finds the minimum value of the objective function and takes this sample point for the sample supplement point. The expression of objective function is shown in Equation (14).

MP is efficient and convenient, but this method only adopts the predicted value and ignores root mean square error. If the accuracy of the surrogate model is not enough, the optimal solution cannot be solved.

2.2.3. Maximum of Root Mean Squared Error (RMSE)

The RMSE filling criterion can effectively improve the optimization ability of the model using spatial filling samples with more points. While elaborating the Kriging model in Section 2.1, it has been mentioned that the model can predict RMSE at this point, it is shown in Equation (20). Therefore, introducing RMSE criterion, namely inserting sample points at the position with the largest root mean square error can greatly ameliorate the calculation accuracy of the surrogate model. The optimization mathematical model expression of RMSE is as follows:

$$\max :\mathrm{RMSE}(\mathrm{x})=\sqrt{\mathrm{MSE}(\widehat{\mathrm{y}}(\mathrm{x}))}$$

(25)

Although RMSE can improve the global accuracy of the surrogate model, it is not conducive to optimization. Selecting some sample points can improve the accuracy of the model. With the increase of samples, the accuracy of the model change is unapparent.

2.3. Multiple Points Infill Criterion (EI&MP&RMSE)

Different principles of adding points have relative merits. The single point adding criterion has lower computational efficiency and surrogate model accuracy. In order to tackle this problem, an optimization method based on the principle of multiple points named EI&MP&RMSE is proposed in this paper.

The particular procedure is shown in Figure 1. This method contains the following sections:

(1)

The samples are sampled in the sample space by the experimental design method (optimal hypercube design, opt LHD). The real response value of the sample point is obtained by calculating the sample point. The calculated samples and their response values are stored in the sample point database.

(2)

The sample points and response values in the sample point database are used as the initial data to construct the Kriging surrogate model of the objective function.

(3)

Based on the obtained initial surrogate model, the mean square deviation of the initial surrogate model is calculated to judge the convergence accuracy of the surrogate model. If the conditions are met, the surrogate model will be output. If not, multiple points infill criterion will be added.

(4)

According to the EI criterion, the RMSE criterion and the MP criterion, the response value of the initial surrogate model corresponding to each addition criterion can be obtained. Through the derivation of the response value, the new sample points corresponding to the alternative model of each objective function are preliminarily obtained. At the same time, in the process of updating this sample point, Gaussian function is used to delete redundant data and screen the new sample points. Deleting data mainly focuses on the duplicate data between new sample points and between new sample points and original sample data. Through the processing of these two aspects, the final new sample points can be obtained.

(5)

According to the final new sampling point with the initial sampling point, the sample point response value is calculated and updated. Then, return to step 2. When the convergence condition of the surrogate model can be represented, stop the repeat.

This method can not only ensure the global accuracy, but also achieve the effect of local optimization. RMSE can improve the global accuracy. In addition, it can provide prediction response value and error for the other addition criteria. EI seeks the global optimal solution in terms of global optimization. MP finds the extreme point near the optimal solution. Three infill criteria make up for the deficiency. On the one hand, this method reduces the number of iterations. On the other hand, it improves the accuracy of the surrogate model.

3. Numerical Tests for the EI&MP&RMSE Method

The proposed method has been described in Section 2. Next, its accuracy and efficiency need to be verified. Some classical one−dimensional and two−dimensional mathematical functions will be applied to verify the accuracy of the proposed method. The specific function information and characteristics are shown in

Table 1. Information and characteristics of the benchmarks.

Dim	Mathematical Formula	Range (x)
1	$f\left(x\right)={\exp }^{-2x}+\sin \left(1.5x\right)+\cos (4x)\times {\left(2x-2\right)}^2+x^2+0.9x$	x ∈ [0, 6]
2	$f(x)={\left(x_2-\frac{5.1}{4{\pi }^2}{x}_1^2+\frac{5}{\pi }{x}_1-6\right)}^2+10\left(1-\frac{1}{8\pi }\cos (x_1)\right)+10$	$x_1\in [-5,5]$
		$x_2\in [0,10]$

. At the same time, for the selection of the initial sample points of the verification function, this paper adopts the experimental design method of the optimal Latin hypercube design (Opt LHD). This method can ensure the uniformity of initial sample points.

3.1. 1−D Function Test

Firstly, for the one−dimensional function in

Table 2. 1−D function related validation data.

Infill Criterion	Initial Sample Points	Experimental Design Method	Number of Iterations
MP	4	Opt LHD	6
EI	4	Opt LHD	6
RMSE	4	Opt LHD	6
EI&MP&RMSE	4	Opt LHD	4

, the effectiveness of the proposed method is verified by using the EI criterion, the MP criterion, the RMSE criterion and the EI&MP&RMSE criterion.

As can be seen from Figure 2, the MP criterion is chiefly the local point adding. From the local point of view of generating the surrogate model, the fitting degree and accuracy between the surrogate model and the original function are very high. The new sample points distribution of EI criterion is relatively uniform. It is instrumental for global optimization, but the accuracy of the surrogate model is slightly lower. To improve the accuracy of the surrogate model, the number of iterations needs to be increased. Although the RMSE criterion is the global addition, it is not conducive to finding the optimal solution. In addition, it can be seen from Table 2 that the EI&MP&RMSE criterion needs less iterations under the same initial sample points. Additionally, the fitting degree of the surrogate model is much higher than that of the single point addition criterion. Therefore, the EI&MP&RMSE criterion combines global and local optimization, which not only improves the accuracy of the surrogate model, but also reduces number of iterations.

3.2. 2−D Function Test

Next, the classical Branin function will be applied to test the accuracy of this method. The expression is shown in Table 1. The initial sample points and relevant information of EI&MP&RMSE criterion are shown in

Table 3. 2−D function (Branin) related validation data.

Infill Criterion	Initial Sample Points	Experimental Design Method	Number of Iterations
MP	8	Opt LHD	6
EI	8	Opt LHD	6
RMSE	8	Opt LHD	6
EI&MP&RMSE	8	Opt LHD	4

.

Based on the EI&MP&RMSE criterion, the shape and characteristics of the Branin function surrogate model are shown in Figure 3. Test results on the predicted MSE of other criterions for the Branin function are presented in Figure 4.

Figure 3a,b show the original shape of the Branin function and the shape of the surrogate model, respectively. Figure 3d describes the location distribution information of initial sample points and new sample points. Due to the repetition of new sample points in the iterative calculation process, some new points are ignored. Moreover, results on the predicted MSE of the Branin function for the 2−D problem are shown in Figure 3c and Figure 4a–c. The larger the MSE value, the lower the accuracy of the prediction surrogate model, which can reflect the characteristics of the point adding criterion.

It can be concluded from these plots that the mean square error of the single criterion is much larger than that of the EI&MP&RMSE criterion, which reveals that the EI&MP&RMSE criterion has higher precision accuracy of the surrogate model. As can be seen from Table 3, the EI&MP&RMSE criterion has fewer iterations than other single criteria, which shows that when the total number of sample points is certain, the number of iterations is reduced and the optimization efficiency is doubled. In a word, the EI&MP&RMSE criterion has higher accuracy and fewer iterations of the surrogate model.

4. Conclusions

Firstly, the Kriging model and each addition criterion are introduced in this paper. On this basis, because there are some problems in the process of using the surrogate model to deal with complex engineering cases, such as low accuracy and computational efficiency, this paper proposes a multiple addition criterion named EI&MP&RMSE criterion. Simultaneously, the specific procedure is depicted. The multiple point infilling criterion is verified by one−dimensional and two−dimensional mathematical functions and compared with other single point adding criteria. It is verified that the EI&MP&RMSE criterion has improved the surrogate model accuracy. When the total number of sample points is certain, iterations are reduced and the optimization efficiency is higher than before. This method has realistic significance for the optimization of complex engineering cases.