A Novel Gaussian Process Surrogate Model with Expected Prediction Error for Optimization under Constraints

Abstract

Optimization, particularly constrained optimization problems (COPs), is fundamental in engineering, influencing various sectors with its critical role in enhancing design efficiency, reducing experimental costs, and shortening testing cycles. This study explores the challenges inherent in COPs, with a focus on developing efficient solution methodologies under stringent constraints. Surrogate models, especially Gaussian Process Regression (GPR), are pivotal in our approach, enabling the approximation of complex systems with reduced computational demand. We evaluate the efficacy of the Efficient Global Optimization (EGO) algorithm, which synergizes GPR with the Expected Improvement (EI) function, and further extend this framework to Constrained Expected Improvement (CEI) and our novel methodology Constrained Expected Prediction Error (CEPE). We demonstrate the effectiveness of these methodologies by numerical benchmark simulations and the real-world application of optimizing a Three-Bar Truss Design. In essence, the innovative CEPE approach promises a potent balance between solution accuracy and computational prowess, offering significant potential in the broader engineering field.

1. Introduction

In the fields of science and engineering, optimization problems, especially those with constraints, play a crucial role. Constrained optimization problems (COPs) are vital in a wide range of applications, including process control [1,2], reactor design [3], and production scheduling [4]. Among these, specific challenges such as tension/compression coil spring design [5] highlight the intricate nature of these problems. The aim of addressing COPs is to enhance design efficiency, minimize experimental costs, and reduce testing cycles. These problems hold intrinsic value across various engineering disciplines, including chemical, mechanical, and electrical engineering [6,7,8], as they seek optimal solutions within the boundaries set by constraints, be they equalities or inequalities [9]. To solve these complex issues, a range of methodologies have been developed, from gradient-based methods [10], known for their efficiency in differentiable problems, to population-based strategies like Genetic Algorithms [11], which are capable of addressing non-differentiable and multi-modal challenges, albeit often at a higher computational cost [12]. COPs are typically represented through mathematical formulations, underscoring their quantitative nature and the systematic approach required for their resolution. We define the formulation

$$\begin{array}{cc}min\hfill & y=f(\mathbf{x}),\hfill \\ st:\hfill & \mathbf{l}\le \mathbf{g}(\mathbf{x})=(g_1(\mathbf{x}),g_2(\mathbf{x}),\dots ,g_m(\mathbf{x}))\le \mathbf{u},\hfill \\ where\hfill & \mathbf{l}=(l_1,l_2,\dots ,l_m),\mathbf{u}=(u_1,u_2,\dots ,u_m),\hfill \\ \hfill & \mathbf{x}=(x_1,x_2,\dots ,x_n)\in \mathbf{X},\hfill \\ \hfill & \mathbf{X}=\left\{\mathbf{x}\right|{\mathbf{x}}_{\mathbf{l}}\le \mathbf{x}\le {\mathbf{x}}_{\mathbf{u}}\},\hfill \\ \hfill & {\mathbf{x}}_{\mathbf{l}}=(x_{l_1},x_{l_2},\dots ,x_{l_n}),{\mathbf{x}}_{\mathbf{u}}=(x_{u_1},x_{u_2},\dots ,x_{u_n}),\hfill \end{array}$$

(1)

where $\mathbf{x}$ is a solution vector within the solution space $\mathbf{X}$, $\mathbf{g}(\mathbf{x})$ represents the constraints, and $\mathbf{l}$ and $\mathbf{u}$ are the lower and upper bounds of the constraints, respectively. There are two types of scenarios for the solution to this formulation: feasible and infeasible. A solution is considered feasible if the solution vector $\mathbf{x}\in \mathbf{X}$ satisfies all the constraints $\mathbf{g}(\mathbf{x})$. Conversely, a solution is deemed infeasible if $\mathbf{x}\in \mathbf{X}$ but does not satisfy all the constraints $\mathbf{g}(\mathbf{x})$. At times, the feasibility ratio (the percentage of feasible solutions out of the total number of feasible and infeasible solutions) can be very low, which is equivalent to having very stringent constraints. In such cases, it is imperative to find a solution that satisfies the constraints as efficiently as possible.

For solving complex optimization problems, especially those with constraints, the use of surrogate models has emerged as a highly effective strategy. Surrogate models, pivotal in the realm of optimization, provide simplified yet powerful approximations of complex systems, enabling efficient exploration and optimization of design spaces where direct evaluations are prohibitively expensive. These models, also referred to as metamodels, span a diverse array of methodologies, each tailored to capture the intricacies of different problem domains [13]. To encapsulate these varying surrogate model methodologies and their application in constrained optimization problems,

Table 1. Summary of surrogate model methodologies in constrained optimization problems.

Methodology	Reference	Approach	Advantages	Limitations
Polynomial Regression	Box and Wilson, 1992 [14]	Utilizes polynomial equations to model relationships between variables	Straightforward approach for problems with smooth relationships	May not capture complex, nonlinear behaviour effectively
Radial Basis Function (RBF) Networks	Buhmann, 2000 [15]	Employs radial basis functions to approximate unknown functions	Effective for scattered multidimensional data	Computationally intensive for large datasets
Kriging	Oliver and Webster, 1990 [16]	A best-unbiased predictor based on geostatistical methods	Provides an estimate of prediction error; well-suited for surrogate modelling	Requires significant computational resources for large problems
Gaussian Process Regression (GPR)	Gramacy, 2020 [17]	Models unknown functions as samples from a Gaussian process	Robust in capturing trends and uncertainty quantification	May be challenging to scale to very high-dimensional problems

provides a comprehensive summary of the key literature, highlighting techniques, tested datasets, and their respective advantages and disadvantages.

Despite these advancements, a comprehensive review of the literature underscores certain limitations in the current surrogate modelling approaches, particularly when applied to constrained optimization problems [18]. A notable challenge is the difficulty in accurately capturing and adapting to the complex constraint boundaries that often define feasible regions in optimization landscapes [19]. Many existing surrogate models excel in unconstrained scenarios or situations with simple constraints but may fall short when faced with complex, nonlinear, or high-dimensional constraints [20]. Additionally, there is a paucity of methodologies that effectively incorporate constraint handling mechanisms within the surrogate model itself, often relying on external penalty functions or constraint relaxation techniques that may not always yield optimal solutions. This gap in the literature highlights the need for more sophisticated surrogate models that can inherently deal with complex constraints while maintaining the balance between exploration and exploitation in the search space. These insights into the limitations of current surrogate models in handling constrained optimization problems have motivated the development of our proposed methodologies. By addressing these identified gaps, we aim to contribute to the advancement of surrogate-assisted optimization strategies, providing more robust and efficient solutions for complex constrained optimization challenges.

Among the diverse array of surrogate modelling techniques, Gaussian Process Regression (GPR) stands out for its efficiency and effectiveness [17]. GPR is renowned for its robustness in capturing the underlying trends of the data with a quantifiable measure of uncertainty, making it particularly suitable for optimization problems where uncertainty plays a critical role [21]. The historical development of surrogate models in optimization was notably advanced in 1998 when Jones et al. [22] introduced the Efficient Global Optimization (EGO) algorithm. This algorithm integrates GPR with the Expected Improvement (EI) function, a pivotal concept in Bayesian optimization. The EI function is designed to systematically identify the global optimum of computationally expensive black-box functions, which are often subject to inherent uncertainties [23]. This approach prioritizes sampling in regions where the anticipated improvement in the function’s value is maximized, thereby enhancing the model’s accuracy in critical areas. This methodology exemplifies an active learning strategy, dynamically refining the model with each new data point to efficiently navigate the complex landscape of the optimization problem. However, when dealing with optimization problems that include stringent constraints, the standard EI approach may require modifications to accommodate these limitations. In response to this challenge, recent developments have introduced variants such as the Constrained Expected Improvement (CEI), which adapt the EI principle to handle constraints effectively [24,25].

In addressing the challenges inherent in constrained optimization, this study introduces significant innovations that extend the current state-of-the-art methodologies. Firstly, we propose a novel approach to compute the Expected Prediction Error (EPE) [26] at an untested point by leveraging the cross-validation error from a nearby tested point. This method enhances the prediction accuracy of our surrogate model, particularly in the sparse regions of the design space where traditional interpolation methods might falter. Secondly, we introduce the Constrained Expected Prediction Error (CEPE) criterion, a pioneering metric designed to navigate the intricate landscape of constrained optimization problems efficiently. By integrating CEPE within a Differential Evolution (DE) [27] framework augmented with Gaussian Process (GP) surrogates, our approach not only capitalizes on the strengths of evolutionary algorithms in exploring complex solution spaces but also harnesses the predictive prowess of GP models to make informed decisions during the optimization process. The synergy between these innovations presents a robust framework that promises improved optimization performance, especially in scenarios characterized by stringent constraints and expensive function evaluations.

The remainder of this paper is structured as follows. In Section 2, we introduce the theoretical knowledge about Gaussian Process, Expected Improvement and Expected Prediction Error. In Section 3, we discuss methodologies for Constrained Expected Improvement, introduce our novel method Constrained Expected Prediction Error, and provide the GP surrogate-assisted Differential Evolution (DE) algorithm. In Section 4, we evaluate the efficacy of CEI and CEPE using benchmark problems, and in Section 5, we illustrate their application in a real-world problem. Finally, we conclude this paper with discussions and remarks in Section 6.

2. Background and Concepts

In this section, we provide a brief theoretical overview of Gaussian Process Regression (GPR), Expected Improvement (EI), and Expected Prediction Error (EPE).

2.1. Gaussian Process Regression

We aim to use the Gaussian Process Regression (GPR) framework to model an unknown function $f(\mathbf{x})$, by assuming that $f(\mathbf{x})$ is a sample from a Gaussian process denoted as $y(\mathbf{x})$. Within this context, for any input point $\mathbf{x}$, the function value $f(\mathbf{x})$ is viewed as a sample from a Gaussian random variable $y(\mathbf{x})$ and $y(\mathbf{x})$ is distributed as $\mathcal{N}(\mu ,{\sigma }^2)$, where $\mu$ and ${\sigma }^2$ are constants independent of $\mathbf{x}$. The covariance of $y(\mathbf{x})$ with another random variable $y({\mathbf{x}}^{\prime })$, where ${\mathbf{x}}^{\prime }$ represents another point in the input space, is given by

$$\mathrm{cov}[y(\mathbf{x}),y({\mathbf{x}}^{\prime })]={\sigma }^2\mathrm{Corr}(y(\mathbf{x}),y({\mathbf{x}}^{\prime })).$$

(2)

The correlation function $\mathrm{Corr}(y(\mathbf{x}),y({\mathbf{x}}^{\prime }))$ is defined as

$$\mathrm{Corr}(y(\mathbf{x}),y({\mathbf{x}}^{\prime }))=exp\left(-\sum _{k=1}^p{\theta }_k{|x_k-x_k^{\prime }|}^2\right),$$

(3)

where p is the dimension of the input space and ${\theta }_k$ are the hyperparameters.

Considering a set of N input points ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N$, let ${\mathbf{f}}^N={(f({\mathbf{x}}_1),f({\mathbf{x}}_2),\dots ,f({\mathbf{x}}_N))}^T$ and ${\mathbf{y}}^N={(y({\mathbf{x}}_1),y({\mathbf{x}}_2),\dots ,y({\mathbf{x}}_N))}^T$. The parameters $\mu ,{\sigma }^2,{\theta }_1,\dots ,{\theta }_p$ can be estimated by maximizing the joint Gaussian probability density function of ${\mathbf{y}}^N$ at ${\mathbf{f}}^N$

$$p({\mathbf{f}}^N|\mu ,{\sigma }^2,\mathbf{R})=\frac{1}{{(2\pi {\sigma }^2)}^{N/2}\sqrt{det(\mathbf{R})}}exp\left[-\frac{{({\mathbf{f}}^N-\mu \mathbf{1})}^T{\mathbf{R}}^{-1}({\mathbf{f}}^N-\mu \mathbf{1})}{2{\sigma }^2}\right],$$

(4)

where $\mathbf{R}$ is an $N\times N$ matrix with elements $R_{i,j}=\mathrm{Corr}(y({\mathbf{x}}_{\mathbf{i}}),y({\mathbf{x}}_{\mathbf{j}}))$, and $\mathbf{1}$ is an N-dimensional column vector of ones.

The log-likelihood function is then

$$\mathcal{L}(\mu ,{\sigma }^2,{\theta }_1,\dots ,{\theta }_p|{\mathbf{f}}^N)=-\frac{N}{2}log(2\pi {\sigma }^2)-\frac{1}{2}log(det(\mathbf{R}))-\frac{{({\mathbf{f}}^N-\mu \mathbf{1})}^T{\mathbf{R}}^{-1}({\mathbf{f}}^N-\mu \mathbf{1})}{2{\sigma }^2}.$$

(5)

Now, let us differentiate the log-likelihood with respect to $\mu$:

$$\frac{\partial \mathcal{L}}{\partial \mu }=\frac{1}{{\sigma }^2}({\mathbf{1}}^T{\mathbf{R}}^{-1}({\mathbf{f}}^N-\mu \mathbf{1})).$$

(6)

Setting this derivative to zero and solving for $\mu$ gives the maximum likelihood estimate

$$\widehat{\mu }=\frac{{\mathbf{1}}^T{\mathbf{R}}^{-1}{\mathbf{f}}^N}{{\mathbf{1}}^T{\mathbf{R}}^{-1}\mathbf{1}}.$$

(7)

Next, let us differentiate the log-likelihood with respect to ${\sigma }^2$

$$\frac{\partial \mathcal{L}}{\partial {\sigma }^2}=-\frac{N}{2{\sigma }^2}+\frac{{({\mathbf{f}}^N-\mathbf{1}\widehat{\mu })}^T{\mathbf{R}}^{-1}({\mathbf{f}}^N-\mathbf{1}\widehat{\mu })}{2{\sigma }^4}.$$

(8)

Setting this derivative to zero and solving for ${\sigma }^2$ gives the maximum likelihood estimate

$${\widehat{\sigma }}^2=\frac{{({\mathbf{f}}^N-\mathbf{1}\widehat{\mu })}^T{\mathbf{R}}^{-1}({\mathbf{f}}^N-\mathbf{1}\widehat{\mu })}{N}.$$

(9)

There is no analytical form for the estimates of ${\theta }_1,\dots ,{\theta }_p$, so the maximization of the log-likelihood function with respect to ${\theta }_1,\dots ,{\theta }_p$ is usually completed by conjugate descent.

With the maximum likelihood estimates of the parameters, we can now derive expressions for predicting $f(\mathbf{x})$ at a new point $\mathbf{x}$. Let $\mathbf{r}$ be the vector of correlations between the new point and the N data points, where the i-th element of $\mathbf{r}$ is $Corr(y(\mathbf{x}),y({\mathbf{x}}_i))$. The best linear unbiased predictor (BLUP) of $f(\mathbf{x})$ can be written as:

$$\widehat{f}(\mathbf{x})=\widehat{\mu }+{\mathbf{r}}^T{\mathbf{R}}^{-1}({\mathbf{f}}^N-\mathbf{1}\widehat{\mu }),$$

(10)

where $\widehat{\mu }$ is the estimate of the mean in (7) and $\mathbf{R}$ is the correlation matrix calculated using the estimates of ${\theta }_i$.

Similarly, the prediction variance can be derived as:

$$s^2(\mathbf{x})={\widehat{\sigma }}^2\left(1-{\mathbf{r}}^T{\mathbf{R}}^{-1}\mathbf{r}+\frac{{(1-{\mathbf{1}}^T{\mathbf{R}}^{-1}\mathbf{r})}^2}{{\mathbf{1}}^T{\mathbf{R}}^{-1}\mathbf{1}}\right).$$

(11)

This forms the basis for Gaussian Process Regression, where we model the unknown function $f(\mathbf{x})$ as a Gaussian process and make predictions at new points by combining information from the observed data.

2.2. Expected Improvement

In this section, we introduce a widely-adopted infill sampling method known as Expected Improvement (EI), developed by Jones et al. [22] for the optimization of expensive black-box functions. Expected Improvement is particularly useful once a surrogate model, typically Gaussian Process Regression, is constructed.

As previously defined in Equation (1), let $f(\mathbf{x})$ denote the objective function with a surrogate model $y(\mathbf{x})$ that follows a Gaussian process. Consider a set of test inputs $\{x_1,x_2,\dots ,x_N\}$ with corresponding observed function values ${\mathbf{f}}^N={(f(x_1),f(x_2),\dots ,f(x_N))}^T$. We define $f_{\min }$ as the best observed value of the function among the test inputs, i.e., $f_{\min }=min({\mathbf{f}}^N)$. The improvement of $f(\mathbf{x})$ at a new input point $\mathbf{x}$ is defined as

$$I(\mathbf{x})=max\{f_{\min }-f(\mathbf{x}),0\}.$$

(12)

Expected Improvement is then defined as the expected value of this improvement, given the observed data ${\mathbf{f}}^N$

$$E\left[I(\mathbf{x})\right|{\mathbf{f}}^N]=E[max\{f_{\min }-f(\mathbf{x}),0\}|{\mathbf{f}}^N].$$

(13)

Through some non-trivial mathematical manipulations, we can express the Expected Improvement in terms of the predictive mean and standard deviation of the surrogate model as follows:

$$E\left[I(\mathbf{x})\right|{\mathbf{f}}^N]=(f_{\min }-\widehat{f}(\mathbf{x}))\mathsf{\Phi }\left(\frac{f_{\min }-\widehat{f}(\mathbf{x})}{s(\mathbf{x})}\right)+s(\mathbf{x})\varphi \left(\frac{f_{\min }-\widehat{f}(\mathbf{x})}{s(\mathbf{x})}\right),$$

(14)

where $\widehat{f}(\mathbf{x})$ and $s(\mathbf{x})$ are the predictive mean and standard deviation of the surrogate model at $\mathbf{x}$, computed according to Equations (10) and (11), and $\varphi (·)$ and $\mathsf{\Phi }(·)$ denote the probability density function (PDF) and cumulative distribution function (CDF) of the standard normal distribution, respectively.

2.3. Expected Prediction Error

In this section, we introduce the Expected Prediction Error (EPE), an active learning strategy employed to evaluate the accuracy of a predictive model. EPE quantifies the discrepancy between the predicted and actual values. Specifically, it is the expected value of the squared prediction error over a sample of data. EPE is sometimes referred to as the mean squared error (MSE).

Within the context of the GPR methodology, the posterior distribution of the response at a given location $\mathbf{x}$ is denoted as $\widehat{y}(\mathbf{x})$, which follows a Gaussian distribution expressed as $\widehat{y}(\mathbf{x})\sim \mathcal{N}({\mu }_{\mathbf{x}},{\sigma }_{\mathbf{x}}^2)$. In this setting, the mean ${\mu }_{\mathbf{x}}$ symbolizes the predictive estimate given by $\widehat{f}(\mathbf{x})$, while the variance ${\sigma }_{\mathbf{x}}^2$ reflects the uncertainty associated with the prediction, denoted as $s_{\mathbf{x}}^2$. Given this framework, we can define the prediction error, often referred to as the loss function, $\mathcal{L}$, for a point $\mathbf{x}$ as

$$\mathcal{L}(\mathbf{x})={(y(\mathbf{x})-\widehat{y}(\mathbf{x}))}^2.$$

(15)

Here, $y(\mathbf{x})$ embodies the true underlying response, perturbed by inherent observational noise.

Then, the overall generalization error of the GPR-based surrogate model is given by

$$e={\int }_{\mathcal{D}}E\left[\mathcal{L}(\mathbf{x})\right]d\mathbf{x},$$

(16)

where $E[\mathcal{L}(\mathbf{x})]$ denotes the expected value of the prediction error, and $\mathcal{D}$ is a subset of ${\mathbb{R}}^p$. This expectation can be decomposed into

$$\begin{array}{cc}\hfill E[\mathcal{L}(\mathbf{x})]=& {(E\left[y(\mathbf{x})\right]-E\left[\widehat{y}(\mathbf{x})\right])}^2+E{\left[(\widehat{y}(\mathbf{x})-E\left[\widehat{y}(\mathbf{x})\right])\right]}^2+E\left[{(y(\mathbf{x})-E\left[y(\mathbf{x})\right])}^2\right],\hfill \end{array}$$

(17)

where the first term represents the squared bias, capturing the average difference between the predicted and observed responses; the second term is the prediction variance of the surrogate model; and the third term is the variance of the noise $Var(ϵ)$, which is often negligible.

The EPE quantifies the discrepancy between the GPR model’s predictions and the true function values. In this context, $f(\mathbf{x})$ denotes the true function we aim to approximate, and $\widehat{f}(\mathbf{x})$ represents the GPR model’s prediction at $\mathbf{x}$. Thus, EPE at a point $\mathbf{x}$ is given by

$$EPE(\mathbf{x})=\underset{{\mathrm{bias}}^2}{\underbrace{{(f(\mathbf{x})-\widehat{f}(\mathbf{x}))}^2}}+\underset{\mathrm{variance}}{\underbrace{s^2(\mathbf{x})}}.$$

(18)

However, the true response $f(\mathbf{x})$ is unknown in the bias term. Following the approach by Liu et al. [26], we employ leave-one-out cross-validation for estimation. Initially, we estimate the cross-validation errors at all training sample locations:

$$\begin{array}{c}\hfill e_{\mathrm{CV}}^2(x_i)={(f({\mathbf{x}}_i)-{\widehat{f}}^{-i}({\mathbf{x}}_i))}^2,i=1,2,\dots ,N,\end{array}$$

(19)

where ${\mathbf{x}}_i$ represents the i-th training sample, and ${\widehat{f}}^{-i}$ denotes the GP model trained using all training samples except $({\mathbf{x}}_i,f({\mathbf{x}}_i))$. Next, for any point $\mathbf{x}$, we locate the nearest training sample to $\mathbf{x}$, denoted as ${\mathbf{x}}_i$, and assign its associated cross-validation error to $e_{\mathrm{CV}}^2(\mathbf{x})$:

$$\begin{array}{c}\hfill e_{\mathrm{CV}}^2(\mathbf{x})=e_{\mathrm{CV}}^2({\mathbf{x}}_i),\mathrm{where}i=arg\underset{i}{min}|\mathbf{x}-{\mathbf{x}}_i|,i=1,2,\dots ,N.\end{array}$$

(20)

Finally, incorporating the cross-validation error, we obtain a refined expression for the EPE:

$$\begin{array}{c}\hfill EPE(\mathbf{x})=e_{\mathrm{CV}}^2(\mathbf{x})+s^2(\mathbf{x}).\end{array}$$

(21)

The Expected Prediction Error (EPE) plays a crucial role in assessing the accuracy and reliability of our surrogate model, particularly in constrained optimization problems. By calculating the EPE, we can identify regions in the design space where the model’s predictions are less reliable, informing us where additional sampling might be beneficial. Moreover, as EPE comprises both bias and variance, it aids in striking a balance between the two, ensuring that the model is neither too simplistic nor too complex. This information is vital for efficient exploration of the search space and making informed decisions in the optimization process.

3. Methodology

In this section, we first present the CEI method proposed by Jiao et al. [25], then we introduce a novel method CEPE for solving constrained optimization problems (COP). To solve a COP, the objective is to minimize the objective function subject to several constraints. Before delving into the details of our proposed method, we will define some notations in the following subsection.

In this work, we treat $\mathbf{x}$ as an untested point, while ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N$ are considered tested points. Let $y(\mathbf{x})$ denote the objective function, and $G_i(\mathbf{x})$ represent the constraint functions for $i=1,2,\dots ,m$. For a given point $\mathbf{x}$, we define a Gaussian random vector $\mathbf{G}(\mathbf{x})$ as

$$\begin{array}{c}\hfill \mathbf{G}(\mathbf{x})=(G_1(\mathbf{x}),G_2(\mathbf{x}),\dots ,G_m(\mathbf{x})).\end{array}$$

(22)

We introduce $\mathbf{G}(\mathbf{x})$ as a Gaussian random vector to represent the constraints’ evaluations at the point $\mathbf{x}$. This probabilistic modelling approach allows us to account for and manage the uncertainties associated with constraint evaluations, especially when exact determinations of these constraints are not feasible or when they are subject to variability due to measurement errors, model inaccuracies, or other sources of uncertainty.

For a set of N tested points ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N$, we define the following Gaussian random vectors:

$$\begin{array}{cc}\hfill {\mathbf{y}}^N& ={(y({\mathbf{x}}_1),y({\mathbf{x}}_2),\dots ,y({\mathbf{x}}_N))}^T,\hfill \\ \hfill {\mathbf{G}}_1^N& ={(G_1({\mathbf{x}}_1),G_1({\mathbf{x}}_2),\dots ,G_1({\mathbf{x}}_N))}^T,\hfill \\ \hfill {\mathbf{G}}_2^N& ={(G_2({\mathbf{x}}_1),G_2({\mathbf{x}}_2),\dots ,G_2({\mathbf{x}}_N))}^T,\hfill \\ \hfill & ⋮\hfill \\ \hfill {\mathbf{G}}_m^N& ={(G_m({\mathbf{x}}_1),G_m({\mathbf{x}}_2),\dots ,G_m({\mathbf{x}}_N))}^T,\hfill \\ \hfill {\mathbf{G}}^{mN}& ={(G_1({\mathbf{x}}_1),G_2({\mathbf{x}}_1),\dots ,G_m({\mathbf{x}}_1),\dots ,G_1({\mathbf{x}}_N),G_2({\mathbf{x}}_N),\dots ,G_m({\mathbf{x}}_N))}^T,\hfill \end{array}$$

where ${\mathbf{y}}^N$, ${\mathbf{G}}_1^N$, ${\mathbf{G}}_2^N$, …, ${\mathbf{G}}_m^N$, and ${\mathbf{G}}^{mN}$ represent the function values at the tested points. In contrast, $\mathbf{y}(\mathbf{x})$ and $\mathbf{G}(\mathbf{x})$ are the quantities to be predicted. The evaluated function values are

$$\begin{array}{cc}\hfill {\mathbf{f}}^N& ={(f({\mathbf{x}}_1),f({\mathbf{x}}_2),\dots ,f({\mathbf{x}}_N))}^T,\hfill \\ \hfill {\mathbf{g}}_1^N& ={(g_1({\mathbf{x}}_1),g_1({\mathbf{x}}_2),\dots ,g_1({\mathbf{x}}_N))}^T,\hfill \\ \hfill {\mathbf{g}}_2^N& ={(g_2({\mathbf{x}}_1),g_2({\mathbf{x}}_2),\dots ,g_2({\mathbf{x}}_N))}^T,\hfill \\ \hfill & ⋮\hfill \\ \hfill {\mathbf{g}}_m^N& ={(g_m({\mathbf{x}}_1),g_m({\mathbf{x}}_2),\dots ,g_m({\mathbf{x}}_N))}^T,\hfill \\ \hfill {\mathbf{g}}^{mN}& ={(g_1({\mathbf{x}}_1),g_2({\mathbf{x}}_1),\dots ,g_m({\mathbf{x}}_1),\dots ,g_1({\mathbf{x}}_N),g_2({\mathbf{x}}_N),\dots ,g_m({\mathbf{x}}_N))}^T,\hfill \end{array}$$

which are considered as samples of ${\mathbf{y}}^N$, ${\mathbf{G}}_1^N$, ${\mathbf{G}}_2^N$, …, ${\mathbf{G}}_m^N$, and ${\mathbf{G}}^{mN}$. Essentially, by using the known function values, we can predict $y(\mathbf{x})$ and $G_i(\mathbf{x})$ for $i=1,2,\dots ,m$ at the tested points ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N$ without additional function evaluations.

3.1. Constrained Expected Improvement

In the exploration of constrained optimization problems, we often encounter a combination of both feasible and infeasible solutions, as categorized by Jiao et al. [25]. The feasibility of a solution $\mathbf{x}\in \mathcal{X}$ is determined by whether it satisfies all constraints $g(\mathbf{x})$. An essential aspect of optimization lies in efficiently transitioning from an infeasible to a feasible situation to reduce computational costs. In our methodology, we initially identify infeasible solutions by evaluating them against all defined constraints $G_i(\mathbf{x})$.

In the following, we introduce the concept of Constrained Expected Improvement to aid this process. For a solution $\mathbf{x}$, we measure the extent of constraint violation by defining

$$\begin{array}{c}\hfill G_i^{+}(\mathbf{x})=max\{0,l_i-G_i(\mathbf{x}),G_i(\mathbf{x})-u_i\},\end{array}$$

(23)

where $l_i$ and $u_i$ denote the lower and upper constraint boundaries, respectively. We represent the constraint violation of a solution as the maximum violation across all constraints,

$$\begin{array}{c}\hfill G^{+}(\mathbf{x})=\underset{i=1,2,\dots ,m}{max}\left\{G_i^{+}(\mathbf{x})\right\}.\end{array}$$

(24)

This value is zero for feasible solutions and positive for infeasible ones. A solution $\mathbf{x}$ is deemed infeasible if it violates any of these constraints, i.e., if $G_i^{+}(\mathbf{x})>0$ for at least one constraint. This preliminary step is crucial for distinguishing between solutions that require improvement to meet feasibility criteria and those that are already feasible.

Let $g_{min}^{+N}$ denote the smallest observed constraint violation among all evaluated points up to the current iteration, effectively representing the ‘best’ constraint violation scenario. This value is crucial for assessing the relative improvement of constraint satisfaction for new candidate solutions compared to the existing ones. To quantify the improvement in constraint violation at an untested point $\mathbf{x}$ relative to the current best solution, we define the constrained improvement for infeasible solutions as

$$\begin{array}{c}\hfill I_{c,N}(\mathbf{x})=\left\{\begin{array}{cc}{g}_{min}^{+N}-G^{+}(\mathbf{x}),\hfill & \mathrm{if}{G}^{+}(\mathbf{x})\le g_{min}^{+N},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(25)

Here, $I_{c,N}(\mathbf{x})$ quantifies how much the constraint violation at an untested point $\mathbf{x}$ improves relative to the current best solution.

We can now derive the solution $\mathbf{x}$ as follows:

$$\begin{array}{cc}\hfill E\left[I_{c,N}(\mathbf{x})\right|g^{mN}]& ={\int }_0^{g_{min}^{+N}}(g_{min}^{+N}-z)\times p_{G^{+}(\mathbf{x})|g^{mN}}(z)dz\hfill \\ \hfill & ={\int }_0^{g_{min}^{+N}}(g_{min}^{+N}-z)\times d{P}_{G^{+}(\mathbf{x})|g^{mN}}(z)\hfill \\ \hfill & ={\int }_0^{g_{min}^{+N}}{P}_{G^{+}(\mathbf{x})|g^{mN}}(z)dz-g_{min}^{+N}\times P_{G^{+}(\mathbf{x})|g^{mN}}(0),\hfill \end{array}$$

(26)

where $p_{G^{+}(\mathbf{x})|g^{mN}}(·)$ represents the probability density function of the random variable conditional on $G^{mN}=g^{mN}$, and $P_{G^{+}(\mathbf{x})|g^{mN}}(·)$ denotes the cumulative distribution function under the same condition.

For $z\le 0$,

$$\begin{array}{c}\hfill P(G^{+}(\mathbf{x})\le z|{\mathbf{g}}^{mN})=0;\end{array}$$

(27)

and for $z>0$, the computation is as follows:

$$\begin{array}{cc}\hfill & P_{G^{+}(\mathbf{x})|{\mathbf{g}}^{mN}}(z)\hfill \\ \hfill =& P\{G^{+}(\mathbf{x})\le z|{\mathbf{g}}^{mN}\}\hfill \\ \hfill =& P\{[l_1-G_1(\mathbf{x})\le z|{\mathbf{g}}^{mN}]\cap [G_1(\mathbf{x})-u_1\le z|{\mathbf{g}}^{mN}]\cap ···\hfill \\ \hfill & \cap [l_m-G_m(\mathbf{x})\le z|{\mathbf{g}}^{mN}]\cap [G_m(\mathbf{x})-u_m\le z|{\mathbf{g}}^{mN}]\}\hfill \\ \hfill =& P\{[l_1-z\le G_1(\mathbf{x})\le u_1+z|{\mathbf{g}}^{mN}]\cap ···\cap [l_m-z\le G_m(\mathbf{x})\le u_m+z|{\mathbf{g}}^{mN}]\hfill \\ \hfill =& {\int }_{l_1-z}^{u_1+z}d{g}_1···{\int }_{l_m-z}^{u_m+z}d{g}_m\times p_{G(\mathbf{x})|{\mathbf{g}}^{mN}}(g_1,\dots ,g_m),\hfill \end{array}$$

(28)

where $p_{G(\mathbf{x})|{\mathbf{g}}^{mN}}(g_1,\dots ,g_m)$ is the joint Gaussian probability function of $G(\mathbf{x})$ conditioned on ${\mathbf{g}}^{mN}$. Since the objective and all constraints are mutually independent, $p_{G(\mathbf{x})|{\mathbf{g}}^{mN}}(g_1,\dots ,g_m)$ can be computed by

$$\begin{array}{cc}\hfill & p_{G(\mathbf{x})|{\mathbf{g}}^{mN}}(g_1,\dots ,g_m)\hfill \\ \hfill =& \prod _{i=1}^m{p}_{G_i(\mathbf{x})|{\mathbf{g}}_i^N}(g_i)\hfill \\ \hfill =& \prod _{i=1}^m\frac{1}{s_{g_i}(\mathbf{x})}\varphi (\frac{g_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}),\hfill \end{array}$$

(29)

where ${\widehat{g}}_i(\mathbf{x})$ and $s_{g_i}(\mathbf{x})$ ($i=1,\dots ,m$) are the expectations and variances of the constraints respectively.

By substituting the result $p_{G(\mathbf{x})|{\mathbf{g}}^{mN}}(g_1,\dots ,g_m)$ obtained from Equation (29) into Equation (28), we obtain the cumulative distribution function for $P_{G^{+}(\mathbf{x})|{\mathbf{g}}^{mN}}(·)$ under the condition ${\mathbf{G}}^{mN}={\mathbf{g}}^{mN}$ as follows

$$\begin{array}{cc}\hfill & P_{G^{+}(\mathbf{x})|{\mathbf{g}}^{mN}}(z)\hfill \\ \hfill =& {\int }_{l_1-z}^{u_1+z}d{g}_1···{\int }_{l_m-z}^{u_m+z}d{g}_m\times \prod _{i=1}^m\frac{1}{s_{g_i}(\mathbf{x})}\varphi (\frac{g_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})\hfill \\ \hfill =& \prod _{i=1}^m{\int }_{l_i-z}^{u_i+z}\frac{1}{s_{g_i}(\mathbf{x})}\varphi (\frac{g_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})d{g}_i\hfill \\ \hfill =& \prod _{i=1}^m[\mathsf{\Phi }(\frac{u_i+z-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})-\mathsf{\Phi }(\frac{l_i-z-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})].\hfill \end{array}$$

(30)

Finally, the constrained expected improvement for a solution $\mathbf{x}$ is given by

$$\begin{array}{c}\hfill E\left[I_{c,N}(\mathbf{x})\right|{\mathbf{g}}^{mN}]={\int }_0^{g_{min}^{+N}}{P}_{G^{+}(\mathbf{x})|{\mathbf{g}}^{mN}}(z)dz-g_{min}^{+N}\times P_{G^{+}(\mathbf{x})|{\mathbf{g}}^{mN}}(0),\end{array}$$

(31)

with $P_{G^{+}(\mathbf{x})|{\mathbf{g}}^{mN}}(z)={\prod }_{i=1}^m[\mathsf{\Phi }(\frac{u_i+z-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})-\mathsf{\Phi }(\frac{l_i-z-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})]$ calculated as mentioned above.

In situations where a feasible point has already been identified, it is desired to maximize the Expected Improvement (EI) of the objective, ensuring that the feasibility conditions are met. This is essentially a quest for a feasible solution that provides the best possible objective value. This strategy is denoted as the Constrained Expected Improvement (CEI), represented by $E\left[I_{c,N}(\mathbf{x})\right|{\mathbf{f}}^N,{\mathbf{g}}^{mN}]$. We define the improvement in the objective function subject to the satisfaction of constraints as $I_{c,N}(\mathbf{x})$. Assuming that $y(\mathbf{x})$ and $G_i(\mathbf{x})$ for $i=1,2,\dots ,m$ are mutually independent, the constrained expected improvement based on ${\mathbf{y}}^N={\mathbf{f}}^N$ and ${\mathbf{G}}^{mN}={\mathbf{g}}^{mN}$ is given by

$$\begin{array}{c}\hfill E\left[I_{c,N}(\mathbf{x})\right|{\mathbf{f}}^N,{\mathbf{g}}^{mN}]=E\left[I(\mathbf{x})\right|{\mathbf{f}}^N]\times \prod _{i=1}^{m}P\left\{\{l_i\le G_i(\mathbf{x})\le u_i\}\right|{\mathbf{g}}_i^N\},\end{array}$$

(32)

where ${\prod }_{i=1}^{m}P\left\{l_i\le G_i(\mathbf{x})\le u_i|{\mathbf{g}}_i^N\right\}$ represents the probability of feasibility (PF), and $E\left[I(\mathbf{x})\right|{\mathbf{f}}^N]$ denotes the expected improvement at the point $\mathbf{x}$.

In the CEI method, the constrained improvement can be defined as

$$\begin{array}{c}\hfill I_{c,N}(\mathbf{x})=\left\{\begin{array}{cc}{f}_{min}-y(\mathbf{x}),\hfill & \mathrm{if}y(\mathbf{x})\le f_{min}\mathrm{and}{l}_i\le G_i(\mathbf{x})\le u_i,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(33)

Here, $f_{min}$ is the best value of $y(\mathbf{x})$ over all the test values ${\mathbf{f}}^N$, and

$$\begin{array}{cc}\hfill & \prod _{i=1}^{m}P\left\{\{l_i\le G_i(\mathbf{x})\le u_i\}\right|{\mathbf{g}}_i^N\}\hfill \\ \hfill =& P\left\{\{l_1\le G_1(\mathbf{x})\le u_1\}\right|{\mathbf{g}}_1^N\}\times P\left\{\{l_2\le G_2(\mathbf{x})\le u_2\}\right|{\mathbf{g}}_2^N\}\times ···\times P\left\{\{l_m\le G_m(\mathbf{x})\le u_m\}\right|{\mathbf{g}}_m^N\}\hfill \\ \hfill =& {\int }_{l_1}^{u_1}{p}_{G_1(\mathbf{x})|{\mathbf{g}}_1^N}(g_1)d{g}_1{\int }_{l_2}^{u_2}{p}_{G_2(\mathbf{x})|{\mathbf{g}}_2^N}(g_2)d{g}_2···{\int }_{l_m}^{u_m}{p}_{G_m(\mathbf{x})|{\mathbf{g}}_m^N}(g_m)d{g}_m\hfill \\ \hfill =& \prod _{i=1}^m[\mathsf{\Phi }(\frac{u_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})-\mathsf{\Phi }(\frac{l_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})].\hfill \end{array}$$

(34)

Using this, we can now define the Constrained Expected Improvement as

$$\begin{array}{cc}\hfill E\left[I_{c,N}(\mathbf{x})\right|{\mathbf{f}}^N,{\mathbf{g}}^{mN}]=& E\left[I(\mathbf{x})\right|{\mathbf{f}}^N]\times \prod _{i=1}^m[\mathsf{\Phi }(\frac{u_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})-\mathsf{\Phi }(\frac{l_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})].\hfill \end{array}$$

(35)

Here, $E\left[I(\mathbf{x})\right|{\mathbf{f}}^N]$ is the same with Equation (14), and the CEI of a solution $\mathbf{x}$ is given by

$$\begin{array}{cc}\hfill E\left[I_{c,N}(\mathbf{x})\right|{\mathbf{f}}^N,{\mathbf{g}}^{mN}]=& \left[(f_{min}-\widehat{f}(\mathbf{x}))\mathsf{\Phi }\left(\frac{f_{min}-\widehat{f}(\mathbf{x})}{s(\mathbf{x})}\right)+s(\mathbf{x})\varphi \left(\frac{f_{min}-\widehat{f}(\mathbf{x})}{s(\mathbf{x})}\right)\right]\hfill \\ & \times \prod _{i=1}^m\left[\mathsf{\Phi }\left(\frac{u_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}\right)-\mathsf{\Phi }\left(\frac{l_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}\right)\right].\hfill \end{array}$$

(36)

As previously defined in Equation (1), the special case for COP formula has a lower constraint bound of $\mathbf{l}=-\infty$ and an upper constraint bound of $\mathbf{u}=0$. Equations (31) and (36) for CEI can be summarized as follows:

For an infeasible situation:

$$\begin{array}{c}\hfill E\left[I_{c,N}(\mathbf{x})\right|{\mathbf{g}}^{mN}]={\int }_0^{g_{min}^{+N}}\prod _{i=1}^m\mathsf{\Phi }\left(\frac{z-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}\right)dz-g_{min}^{+N}\times \prod _{i=1}^m\mathsf{\Phi }\left(\frac{-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}\right).\end{array}$$

(37)

For a feasible situation:

$$\begin{array}{cc}\hfill E\left[I_{c,N}(\mathbf{x})\right|{\mathbf{f}}^N,{\mathbf{g}}^{mN}]=& \left[(f_{min}-\widehat{f}(\mathbf{x}))\mathsf{\Phi }\left(\frac{f_{min}-\widehat{f}(\mathbf{x})}{s(\mathbf{x})}\right)+s(\mathbf{x})\varphi \left(\frac{f_{min}-\widehat{f}(\mathbf{x})}{s(\mathbf{x})}\right)\right]\hfill \\ & \times \prod _{i=1}^m\mathsf{\Phi }\left(\frac{-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}\right).\hfill \end{array}$$

(38)

3.2. Constrained Expected Prediction Error

Building upon the preceding subsection, which discussed the workings of Constrained Expected Improvement (CEI) under both infeasible and feasible situations, we now move our focus towards the Constrained Expected Prediction Error (CEPE). It is our novel method for constrained optimization problems.

In terms of infeasible situations, our approach with CEPE aligns with the methods used in the preceding CEI subsection. In terms of feasible situations of the CEPE method, we introduce the concept of prediction error, denoted as $P{E}_{c,N}(\mathbf{x})$, which is defined as follows:

$$\begin{array}{c}\hfill P{E}_{c,N}(\mathbf{x})=\left\{\begin{array}{cc}{(f(\mathbf{x})-f({\mathbf{x}}_{*}))}^2,\hfill & \mathrm{if}{l}_i\le G_i(\mathbf{x})\le u_i,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(39)

Here, $f({\mathbf{x}}_{*})$ is the observed output sample at the sample point ${\mathbf{x}}_{*}$, which is the nearest evaluated point to the candidate point $\mathbf{x}$. This approach assumes that $f(\mathbf{x})$ exhibits moderate continuity in the vicinity of ${\mathbf{x}}_{*}$, an assumption that is generally valid for smooth functions commonly encountered in engineering and optimization applications.

$$\begin{array}{c}\hfill E\left[P{E}_{c,N}(\mathbf{x})\right|{\mathbf{f}}^N]=(\widehat{f}({\mathbf{x}}_{*})-\widehat{f}{(\mathbf{x}))}^2+s^2(\mathbf{x}).\end{array}$$

(40)

In this formulation, $\widehat{f}({\mathbf{x}}_{*})$ is the surrogate model’s prediction at the point ${\mathbf{x}}_{*}$, and $\widehat{f}(\mathbf{x})$ is the model’s prediction at the candidate point $\mathbf{x}$. The term $s^2(\mathbf{x})$ represents the prediction variance at $\mathbf{x}$, reflecting the model’s uncertainty.

Then we can obtain the formulation of the CEPE of a solution $\mathbf{x}$ below,

$$\begin{array}{cc}\hfill E\left[P{E}_{c,N}(\mathbf{x})\right|{\mathbf{f}}^N,{\mathbf{g}}^{mN}]=& E\left[P{E}_{c,N}(\mathbf{x})\right|{\mathbf{f}}^N]\times \prod _{i=1}^m[\mathsf{\Phi }(\frac{u_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})-\mathsf{\Phi }(\frac{l_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})})]\hfill \\ \hfill =& [{(\widehat{f}({\mathbf{x}}_{*})-\widehat{f}(\mathbf{x}))}^2+s^2(\mathbf{x})]\times \prod _{i=1}^m\left[\mathsf{\Phi }\left(\frac{u_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}\right)-\mathsf{\Phi }\left(\frac{l_i-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}\right)\right].\hfill \end{array}$$

(41)

This formulation takes into account the constraints by incorporating the product of the cumulative distribution functions for the constraints. It effectively combines the expected prediction error with the feasibility of the solution, guiding the optimization process towards regions of the design space where the model uncertainty is high and potential improvements in the objective function are likely, while still adhering to the constraints.

In summary, the CEPE method is underpinned by the assumption of moderate function continuity near ${\mathbf{x}}_{*}$, allowing the use of surrogate model predictions at nearby points to estimate the prediction error at untested points. This approach enables the efficient exploration of the design space, especially in the context of expensive function evaluations and stringent constraints.

Similarly, as previously defined in Equation (1), the special case for COP formula has a lower constraint bound of $\mathbf{l}=-\infty$ and an upper constraint bound of $\mathbf{u}=0$. Equations (31) and (41) for CEPE can be summarized as follows:

For an infeasible situation:

$$\begin{array}{c}\hfill E\left[P{E}_{c,N}(\mathbf{x})\right|{\mathbf{g}}^{mN}]={\int }_0^{g_{min}^{+N}}\prod _{i=1}^m\mathsf{\Phi }\left(\frac{z-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}\right)dz-g_{min}^{+N}\times \prod _{i=1}^m\mathsf{\Phi }\left(\frac{-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}\right).\end{array}$$

(42)

For a feasible situation:

$$\begin{array}{c}\hfill E\left[P{E}_{c,N}(\mathbf{x})\right|{\mathbf{f}}^N,{\mathbf{g}}^{mN}]=\left[{(\widehat{f}({\mathbf{x}}_{*})-\widehat{f}(\mathbf{x}))}^2+s^2(\mathbf{x})]\right]\prod _{i=1}^m\mathsf{\Phi }\left(\frac{-{\widehat{g}}_i(\mathbf{x})}{s_{g_i}(\mathbf{x})}\right).\end{array}$$

(43)

3.3. The GP Surrogate-Assisted DE Algorithm

In this section, we outline and describe the steps of the GP surrogate-assisted Differential Evolution (DE) algorithm. The algorithm applies to problems with any number of dimensions n and any number of constraints m. The steps are as follows:

• Step 1: Design of Experiments (DOE)
  Initialize the algorithm by assigning samples using the Latin Hypercube Design (LHD). The steps are as follows:
  • Divide each dimension into $popsize$ intervals of equal probability.
  • Randomly assign one value to each interval within each dimension.
  • For each dimension, permute the vector of assigned values randomly.
  • Combine the permuted vectors of assigned values across dimensions into a $popsize\times n$ matrix.
• Step 2: Identify the Best Point and Assess Feasibility
  Extract the objective function values and constraint violation values from the initial samples. Identify the feasible solutions (those with no constraint violations) and select the one with the optimal objective function value as the best feasible solution. If no feasible solutions exist, select the solution with the least constraint violation as the best infeasible solution. Update the data structure with information about the best solution discovered thus far, and return this information along with a feasibility flag.
• Step 3: Clustering—Organize Tested Points into Clusters
  Given N tested points $x_1,x_2,\dots ,x_N$, if $N\le L_1$ (where $L_1$ is the maximum number of points a local model can contain), use all the points to build a single local model. If $N>L_1$, apply fuzzy clustering to divide the points into $c_{size}$ clusters, where

  $$\begin{array}{c}\hfill c_{size}=1+⌈\frac{N-L_1}{L_2}⌉,\end{array}$$

  and $L_1>L_2$ (where $L_2$ is the number of points for adding one more local model). The clustering minimizes the following function J:

  $$\begin{array}{c}\hfill J=\sum _{i=1}^N\sum _{j=1}^{c_{size}}{u}_{ij}^{\alpha }{\parallel x_i-v_j\parallel }^2,\end{array}$$

  where $\alpha$ is a constant greater than 1, $v_j$ is the centre of the cluster j, $u_{ij}$ represents the membership degree of $x_i$ in cluster j, and $\parallel ·\parallel$ denotes the Euclidean norm. $v_j$ can be calculated as

  $$\begin{array}{c}\hfill v_j=\frac{{\sum }_{i=1}^N{(u_{ij}^t)}^{\alpha }{x}_i}{{\sum }_{i=1}^N{(u_{ij}^t)}^{\alpha }}.\end{array}$$
  Initialize $u_{ij}^0$ for $i=1,2,\dots ,N$ and $j=1,2,\dots ,c_{size}$, and set $t=0$. Compute $u_{ij}^{t+1}$ using

  $$\begin{array}{c}\hfill u_{ij}^{t+1}=\frac{1}{{\sum }_{k=1}^{c_{size}}{\left(\frac{\parallel x_i-v_j\parallel }{\parallel x_i-v_k\parallel }\right)}^{\frac{2}{\alpha -1}}}.\end{array}$$
  If ${max}_{1\le i\le N,1\le j\le c_{size}}|u_{ij}^{t+1}-u_{ij}^t|<ϵ$, terminate and output $v_j$ and $u_{ij}=u_{ij}^{t+1}$; otherwise, increment t and recalculate $v_j$ and $u_{ij}^{t+1}$ until the stopping criterion $ϵ$ is satisfied.
• Step 4: Modelling—Build Local GP Surrogate Models
  For each cluster, construct local Gaussian Process (GP) surrogate models for the objective function and the constraint functions separately.
• Step 5: Differential Evolution–Generate and Evaluate Candidate Points
  Use Differential Evolution (DE) [27] to generate $NP$ candidate points and evaluate them using the surrogate models. DE is a population-based optimization algorithm that aims to find the global minimum of a given objective function within a high-dimensional search space. The algorithm employs mutation, crossover, and selection strategies to evolve the population towards the optimal solution. During the mutation step, new candidate solutions are generated by combining existing solutions with a weighted difference vector, where the scaling factor is denoted by F. In the crossover step, these candidates are combined with the original population to form a trial population, with a defined crossover probability $CR$.
  Incorporate the Constrained Expected Improvement (CEI) and Constrained Expected Prediction Error (CEPE) criteria, derived from the Gaussian Process (GP) surrogate models, into the evaluation process of candidate points. By computing the CEI and CEPE values for each candidate point, the algorithm leverages these metrics to guide the selection process within DE. Candidate points are prioritized based on higher CEI values, indicating a greater expected improvement, or lower CEPE values, signifying a lower prediction error. This strategic prioritization ensures the exploration of regions in the search space with the highest potential for improvement while adhering to the constraints. This iterative process continues until a termination criterion is met, such as reaching the maximum number of generations $Maxt$ or achieving a predetermined level of convergence. The flowchart of the DE algorithm, highlighting the integration of CEI and CEPE, is depicted in Figure 1.
• Step 6: Evaluation—Use Original Objective and Constraint Functions
  Evaluate the $NP$ candidate points using the original objective and constraint functions. This step can also be considered as the final selection step of Differential Evolution.
• Step 7: Iteration—Update Data and Repeat Steps 3 to 6
  Incorporate new data from the observations into the model, and repeat Steps 3 to 6 until the termination condition is reached. The termination condition is typically based on the number of function evaluations, denoted as $FEs$.

4. Simulation

In this section, we illustrate the efficacy of the proposed CEI (Constrained Expected Improvement) and CEPE (Constrained Expected Prediction Error) methods for addressing constrained optimization problems. We begin by introducing traditional benchmark problems, and then undertake a comparison and evaluation of the two methods by analyzing statistics for each problem.

4.1. Benchmark Problem Description

We employ the classic set of problems presented in CEC 2006 by Liang et al. [28], which was proposed as part of the IEEE Congress on Evolutionary Computation in 2006, and all problems we solve including the modified parts can be found by Chaiyotha and Krityakierne [24]. This suite encompasses 25 real-parameter optimization problems, representing a diverse array of characteristics. GP (Gaussian Process) is advantageous in low-dimensional settings (i.e., when the number of dimensions $n\le 10$) [29]. Consequently, we select problems with suitable dimensions n and numbers of constraints m, as higher dimensions would entail significantly extended computation times with our GP surrogate-assisted DE algorithm. Nine problems are chosen, including two that we modify. The characteristics of these problems can be found in

Table 2. Summary of benchmark characteristics.

$\mathbf{Prob}$	$\mathbf{Dim}$	Type of f	$\mathit{f}({\mathit{x}}^{*})$	$\mathit{\rho }$
G02mod	2	Nonlinear	-	-
G03mod	2	Polynomial	-	-
G04	5	Quadratic	$-30665.539$	52.123%
G06	2	Cubic	−6961.8139	0.0066%
G08	2	Nonlinear	−0.095825	0.8560%
G09	7	Polynomial	680.630	0.5121%
G11	2	Quadratic	0.7499	0.0000%
G12	3	Quadratic	−1.0	4.7713%
G24	2	Linear	−5.508	79.6556%

, where $Prob$ denotes the name of the problems, $Dim$ represents the number of decision variables or dimensions, $f({\mathbf{x}}^{*})$ signifies the best-known results, and $\rho$ is the approximate ratio of feasible solutions within the search space (inclusive of both infeasible and feasible solutions).

4.2. Experimental Settings

As discussed in Section 3.3 regarding the GP surrogate-assisted DE algorithm, we will specify certain parameters for solving these numerical simulations, follows closely the methodologies and values reported in existing literature, particularly the work by Jiao et al. [25]. This reference provided a foundational basis for our initial parameter settings, ensuring consistency and comparability with established practices in the field. The problems have n dimensions and m constraints. In the Latin Hypercube Design (LHD), we generate $(11n-1)$ initial samples, which serve as our population size ($popsize$). The parameters utilized in the fuzzy clustering are $L_1=80$, $L_2=20$, $\alpha =2$, and $ϵ=0.5$. In the Differential Evolution (DE) process, we set the population size ($NP$) to 30, the number of generations to 500, the crossover probability ($CR$) to 0.9, and the scaling factor (F) to 0.5. In the CEPE method, the number of candidates is set to $50\times n$. The number of function evaluations ($FEs$) is also set to $50\times n$. Each benchmark problem is repeated 100 times using both methodologies (CEI and CEPE) with identical parameters to ensure fairness in the comparison.

The experiments were conducted on a personal computer with an Intel(R) Core(TM) i5-8265U CPU @ 1.60 GHz (turbo up to 1.80 GHz). The Intel Corporation is based in Santa Clara, California, USA. The computational analysis and simulations were performed using MATLAB version 2020b. This setup provided a consistent and controlled environment for all computational experiments, ensuring the reliability and reproducibility of our results.

4.3. Results

Figure 2 showcases box plots for nine different benchmark problems. Each box plot displays the distribution of values obtained from 100 independent experiments using two methods: CEI and CEPE. Additionally,

Table 3. Function values (mean, standard deviation, best, and worst) achieved by CEI and CEPE methods after 100 independent repetitions. Values in bold indicate the superior results across comparisons.

		CEI	CEPE
G02	mean	−0.3360827	−0.3510043
	sd	0.0398702	0.02986116
	best	−0.364923	−0.364946
	worst	−0.261689	−0.262792
G03	mean	−0.999943	−0.9999153
	sd	0.000125786	0.000146517
	best	−1.000086	−1.000097
	worst	−0.999447	−0.999467
G04	mean	−30663.44	−30663.82
	sd	2.397651	2.18125
	best	−30665.54	−30665.54
	worst	−30656.25	−30656.61
G06	mean	−6673.657	−6761.299
	sd	280.4894	172.4326
	best	−6942.869	−6949.9
	worst	−5990.58	−6282.121
G08	mean	−0.0958143	−0.09581347
	sd	1.36 × 10−5	1.51 × 10−5
	best	−0.095825	−0.095825
	worst	−0.095764	−0.095762
G09	mean	988.2063	995.5317
	sd	128.8924	110.2954
	best	744.6612	740.9385
	worst	1282.99	1204.271
G11	mean	0.7501684	0.7501476
	sd	0.000233982	0.000241632
	best	0.749901	0.749902
	worst	0.750818	0.751034
G12	mean	−0.9999771	−0.9999772
	sd	1.89 × 10−5	1.85 × 10−5
	best	−1	−1
	worst	−0.999935	−0.999937
G24	mean	−5.505601	−5.505723
	sd	0.002281037	0.00178044
	best	−5.50792	−5.507943
	worst	−5.498976	−5.499741

lists the mean, standard deviation (sd), best, and worst values of the objective function obtained by these methods after 100 iterations. The mean and standard deviation values indicate the quality of the best feasible solutions across the 100 independent experiments. The best and worst entries signify the lowest and highest values of the best feasible solution obtained from the 100 independent experiments for each method, respectively.

Examining Table 3, we observe that in six of the benchmark problems (G02mod, G04, G06, G11, G12, and G24), the CEPE method yields lower mean values compared to the CEI method, indicating better performance. Conversely, the CEI method performs better in the other three problems. Moreover, the standard deviation values mostly align with the mean values, with seven benchmark problems (G02mod, G03mod, G04, G06, G08, G12, and G24) exhibiting lower standard deviations for the method that has the lower mean. Regarding the best values, CEPE outperforms or matches CEI in eight out of nine problems, with the exception being G11, where CEI has a slight edge. For the worst values, CEPE is superior in seven out of nine problems, with CEI prevailing only in G08 and G11.

To statistically compare the performance differences between CEI and CEPE, a non-parametric Wilcoxon rank-sum test was employed. This test is particularly suitable when the data is not normally distributed, and it assumes equal variances between groups. The null hypothesis posits that the two methods yield the same distribution of values, whereas the alternative hypothesis contends that the distributions differ. A test statistic representing the sum of the ranks of the values is computed. A significant test statistic indicates a notable difference between the two methods, and a p-value below 0.05 is deemed statistically significant. The notation $X<Y$ indicates that method X is significantly better than method Y at the 5% significance level, whereas $X\approx Y$ signifies that there is no significant difference between the two methods. The results are compiled in

Table 4. Wilcoxon signed-ranks test results at the 5% significance level.

Problem	p-Value	Results
G02mod	0.001996	CEPE < CEI
G03mod	0.1895	CEPE ≈ CEI
G04	0.2478	CEPE ≈ CEI
G06	0.1301	CEPE ≈ CEI
G08	0.7026	CEPE ≈ CEI
G09	0.5071	CEPE ≈ CEI
G11	0.3431	CEPE ≈ CEI
G12	0.9248	CEPE ≈ CEI
G14	0.4606	CEPE ≈ CEI

. The table reveals that for most benchmark problems, CEI and CEPE exhibit similar performance, with CEPE being significantly better than CEI only for problem G02.

5. Application

In this section, we will apply the GP surrogate-assisted DE algorithm to the Three-Bar Truss Design problem. The Three-Bar Truss Design problem [30,31] is a classical optimization problem that is widely studied within the engineering and mathematical communities. The objective is to design a truss structure, composed of three bars, capable of withstanding a specified load while minimizing weight. This problem is particularly significant in the design of lightweight and efficient structures in fields such as aerospace and civil engineering. Moreover, it serves as a quintessential example of how optimization algorithms can address complex societal and civilizational challenges. The problem can be formulated as follows:

$$\begin{array}{cc}\hfill \mathrm{minimize}f(\mathbf{x})=(2\sqrt{2}{x}_1+x_2)& \times l,\hfill \\ \hfill \mathrm{subject}\mathrm{to}:\frac{\sqrt{2}{x}_1+x_2}{\sqrt{2}{x}_1^2+2x_1{x}_2}P-\sigma & \le 0,\hfill \\ \hfill \frac{x_2}{\sqrt{2}{x}_1^2+2x_1{x}_2}P-\sigma & \le 0,\hfill \\ \hfill \frac{1}{x_1+\sqrt{2}{x}_2}P-\sigma & \le 0,\hfill \end{array}$$

(44)

where $0\le x_1\le 1$ and $0\le x_2\le 1$; $l=100$ cm, $P=2$ KN/${\mathrm{cm}}^2$, and $\sigma =2$ KN/cm${\mathrm{cm}}^2$.

For the experiment, we employ the same parameter settings as in the numerical simulations. The problem has two dimensions and three constraints. Using Latin hypercube design (LHD), we generate 21 initial samples for our population size. The parameters in the fuzzy cluster are set to $L_1=80$, $L_2=20$, $\alpha =2$, and $ϵ=0.5$. In Differential Evolution (DE), the population size is assumed to be $NP=30$, the number of generations is set to 500, the crossover probability is $CR=0.9$, and the scaling factor is $F=0.5$. In the CEPE method, we set the candidate number to 100. The number of function evaluations (FEs) is 100. The experiment is repeated 100 times using two different methodologies (CEI and CEPE), and the parameters are kept the same to ensure a fair comparison.

Figure 3 depicts box plots that display the distribution of the objective function values obtained using the CEI and CEPE methods across 100 independent experiments.

Table 5. Function values (mean, standard deviation, best and worst) achieved by CEI and CEPE methods after 100 independent repetitions. Values in bold indicate the superior results across comparisons.

		CEI	CEPE
Three-Bar Truss Design	mean	264.2152	264.0456
	sd	0.3605	0.2208
	best	263.8962	263.8972
	worst	265.3764	265.0683

lists the mean, standard deviation (sd), best, and worst of the objective function values obtained by both methods over 100 iterations. The best-known result for this problem is approximately $263.8958$. From the table, it is evident that CEPE performs better in terms of mean, standard deviation, and worst objective function values, while CEI marginally outperforms CEPE in the best objective function value.

Additionally, we employ the Wilcoxon rank-sum test to statistically compare the performance of the CEI and CEPE methods. With a p-value of 0.0538%, the test indicates that the CEPE method is significantly superior to the CEI method.

6. Discussion and Remarks

In this paper, we introduced the formulation of constrained optimization problems, the methodologies of Constrained Expected Improvement (CEI) and Constrained Expected Prediction Error (CEPE) in conjunction with the Gaussian Process (GP) surrogate-assisted Differential Evolution (DE) algorithm, and presented numerical benchmark simulations and applications. Additionally, we would like to emphasize the potential application of these methodologies in the field of engineering, where optimization plays a crucial role.

The implementation of CEI and CEPE has showcased their capacity for the efficient utilization of computational resources, making them highly advantageous in engineering contexts where simulation models can be computationally demanding. Their adeptness in managing multiple constraints, robustness against uncertainties, and flexibility to adapt to various problem domains highlight the methodologies’ applicability in real-world engineering problems. These attributes facilitate the optimization of processes, enhancement of product yields, reduction of waste, and improvement in safety and environmental performance.

Despite these advantages, the methodologies do present limitations, such as potential inflexibility in highly specialized engineering challenges and sensitivity to parameter configuration, which could impact their effectiveness. Moreover, the computational intensity required by CEI and CEPE, especially for large-scale problems, warrants consideration. These limitations suggest a cautious approach to their application, emphasizing the need for tailoring or integrating them with other techniques to meet specific engineering requirements.

In summary, the findings from this study affirm the promise of CEI and CEPE in advancing surrogate-assisted optimization strategies for engineering applications. While addressing the identified gaps in constrained optimization, this research also paves the way for future investigations into more sophisticated surrogate models and optimization techniques. Further exploration in this domain could yield even more robust and efficient solutions for complex optimization challenges, contributing to the continuous advancement of engineering optimization practices.