Determination of the Optimal Neural Network Transfer Function for Response Surface Methodology and Robust Design

Abstract

Response surface methodology (RSM) has been widely recognized as an essential estimation tool in many robust design studies investigating the second-order polynomial functional relationship between the responses of interest and their associated input variables. However, there is scope for improvement in the flexibility of estimation models and the accuracy of their results. Although many NN-based estimations and optimization approaches have been reported in the literature, a closed functional form is not readily available. To address this limitation, a maximum-likelihood estimation approach for an NN-based response function estimation (NRFE) is used to obtain the functional forms of the process mean and standard deviation. While the estimation results of most existing NN-based approaches depend primarily on their transfer functions, this approach often requires a screening procedure for various transfer functions. In this study, the proposed NRFE identifies a new screening procedure to obtain the best transfer function in an NN structure using a desirability function family while determining its associated weight parameters. A statistical simulation was performed to evaluate the efficiency of the proposed NRFE method. In this particular simulation, the proposed NRFE method provided significantly better results than conventional RSM. Finally, a numerical example is used for validating the proposed method.

1. Introduction

Among the many quality engineering methodologies, robust design (RD) based on statistical design and analysis methods and optimization methods have contributed significantly to the improvement of product/process quality for more than 20 years. The main objective of RD is to identify the optimal factor settings that can minimize both variability and bias (namely, the deviation between the process mean and desired target value) in a process/product. To demonstrate this RD principle, Taguchi introduced a two-step model based on new design and analysis approaches, particularly orthogonal array and signal-to-noise ratios [1]. However, orthogonal arrays, statistical analysis, and signal-to-noise ratios are highly controversial [2,3,4,5]. Consequently, Vining and Myers [6] improved Taguchi’s model by proposing dual-response (DR) estimation and optimization approaches, in which the separate functions of the mean and variance responses are estimated using response surface methodology (RSM) based on the least-squares method (LSM). Their proposed optimization approach can calculate robust factor settings by minimizing process variability while maintaining the process mean at the target value. This results in the primary procedure of RD being identified in three sequential steps: design of experiment (DoE), response function estimation, and optimization. However, Lin and Tu [7] pointed out that process bias and variability are not considered simultaneously in the DR model and proposed a mean square error (MSE) model as an alternative approach providing more model flexibility when compared with that provided by the DR approach. These DR and MSE models were further extended by Del Castillo and Montgomery [8], Copeland and Nelson [9], Cho et al. [10], Ames et al. [11], Borror [12], Kim and Lin [13], Koksoy and Doganaksoy [14], Ding et al. [15], Shin and Cho [16,17,18], Robinson et al. [19], Fogliatto [20], Goethals and Cho [21], Truong and Shin [22,23], Nha et al. [24], Baba et al. [25], and Yanıkoğlu et al. [26].

In the response function estimation step of the RD procedure, RSM is used extensively to obtain the estimated function associated with an output response and its associated input factors based on the DoE results. RSM is applied extensively in real-world industrial situations, mainly where several input factors potentially affect a product or process’s performance measures or quality characteristics [27]. The relationship between several input factors and one or more associated output responses can be estimated using RSM. Conventionally, LSM can estimate the unknown coefficients of a regression model with many specific error assumptions, such as the errors in the output responses should be followed by a normal distribution (i.e., zero mean and constant variance) identically and independently. However, these assumptions are often violated in many practical problems. Several alternative approaches, such as transformation, weighted least squares (WLS), maximum-likelihood estimation (MLE), Bayesian, and inverse problems, can be used to address this problem. For example, the WLS method has been applied to RSM to estimate the model parameters for unbalanced data [28,29], and Lee and Park [30] and Cho et al. [31] have integrated an MLE method and an expectation–maximization algorithm for unknown parameter estimations using incomplete data. In addition, Goegebeur et al. [32] and Chen and Ye [33,34] proposed a Bayesian approach to estimate the coefficients of response functions where the variances follow a log-normal distribution. Truong and Shin [22,23] proposed a new estimation method that used an inverse problem approach based on Bayesian perspectives. Most of the existing estimation methods in the RSM and RD literature attempt to generate functional forms in input factors for output responses (namely, the process mean, process variance, and quality characteristic). However, relaxing the error assumptions and increasing the estimation precision would result in further improvements.

Artificial neural networks (ANNs) or neural networks (NNs) with nonlinear mapping structures using the human brain function have been widely used as powerful tools in data classification, forecasting, clustering, function approximation, and optimization in recent decades. Irie and Miyake [35], Hornik et al. [36], Cybenko [37], and Funahashi [38] recently addressed the question of approximation using feed-forward NNs and proved that NNs are universal functional approximators with the desired accuracy. ANNs belong to a class of self-adaptive and data-driven techniques, and therefore the undefined relationships between input factors and outputs of a process/product can be ascertained effectively. ANNs can provide a linear or nonlinear relationship between inputs and several outputs without any assumptions based on generalizing capacities of the activation function. Consequently, the functional relationships between input factors and their related output responses in an RD procedure can be estimated without any specific assumptions effectively. Most of the literature on NN function approximation has concentrated on their structure, hidden layers, hidden neurons, and training algorithms. On the contrary, less of them focus on the transfer or activation function, which can strongly affect the complexity and performance of NNs. In addition, most of the transfer functions reported in the literature can provide only a particular configuration to transfer the information from inputs to the associated outputs of a neuron.

The desirability function (DF) approach is one of the most commonly utilized techniques for multiple response optimization problems, in which each quality characteristic is converted into an individual DF that takes a value ranging from 0 to 1. The DF concept was introduced by Harrington [39] for the simultaneous optimization of multiple response problems in the industry field. This approach can be used to determine those optimal factors settings of input variables that result in the most desirable output values, while none of these response values are beyond the specified boundaries. Derringer and Suich [40] implemented a further extension of Harrington’s DF. Depending on whether the response will minimize, maximize, or specify the desired target value, there are three associated DFs: STB (smaller is better), NTB (nominal is better), and LTB (larger is better). Based on the weight parameters, the DF family may provide a flexible transfer function configuration.

Rowlands et al. [41] integrated an NN approach into RD and used the NN to perform the DoE. Su and Hsieh [42], Cook et al. [43], Chow et al. [44], Chang [45], and Chang and Chen [46] combined an NN approach as an estimation method with a genetic algorithm to determine the optimal process parameter settings or the optimal costs in both static and dynamic characteristics in Taguchi’s formula without considering the process mean and variance functions. Recently, Arungpadang and Kim [47] presented a feed-forward NN structure based on RSM in the RD concept.

In the robust design (RD) concept, the process mean and standard deviation of an output response in experiment results can be estimated as a function by many different statistical approaches (i.e., LSM, MLE, and so on). These processes mean and standard deviation functions can be optimized simultaneously based on their desired targets using existing optimization methods. The significant objective of this paper is to propose an alternative estimation method to DR functions (i.e., mean and standard deviation functions) by integrating NN approaches for RSM and RD modeling and then to compare this method to the conventional LSM-based estimation method. First, a feed-forward NN is integrated into the RD procedure as a new DR estimation approach to estimate the process mean and standard deviation response functions. In order to develop the new NN-based estimation method, many different types of transfer functions that can affect the quality of estimation results should be considered.

For this reason, DFs can be utilized as a transfer function family because DFs can represent all three different types (i.e., L-type, S-type, and N-type) of quality characteristics. Among these transfer functions based on DFs, the best transfer function for experimental data using the identified learning and validation approaches is then investigated in this proposed approach. This proposed best transfer function determination procedure for a given NN structure based on DF is derived for the proposed DR estimation method. DFs with their associated weight parameters can provide a variety of transfer function families. In addition, the process mean is often kept at the target value, and the process standard deviation is minimized to the least possible in the RD methodology. Therefore, the “nominal is best” and “smaller is better” DFs are proposed as new NN transfer functions. Next, the best transfer functions for the process mean response function and standard deviation response function are determined based on the optimal weight parameters of the proposed transfer functions using MLE. The associated confidence intervals for the optimal weight parameters are also determined. The results of a case study are presented to support the view that better solutions are obtained from the proposed feed-forward NN-based DR estimation method compared with those from the conventional LSM method and the feed-forward NN estimation approach by using the conventional log-sigmoid transfer function. An overview of the proposed NN-based DR function estimation method is presented in Figure 1.

2. RD Estimation Method Based on RSM

RSM, developed by Box and Wilson [48], consists of mathematical and statistical techniques based on the fittings of empirical models obtained from an experimental design. RSM can estimate the functional empirical relationship between the input factors and their related output responses. The basic theory, estimation methods, and analytical techniques of RSM can be found in the study by Myers and Montgomery [27]. Myers [49] and Khuri and Mukhopadhyay [50] provide insights into the various developmental stages and future directions of RSM. In general, the output response $y$ can be identified as a function of the input factor x as follows:

$$y=\mathbf{x}\mathsf{\beta }+\epsilon ,$$

(1)

where x, β, and ε denote the vector of control factors, column vector of model parameters, and random error. The estimated second-order models for the process mean and standard deviation functions are represented as follows:

$${\widehat{\mu }}_{LSM}\left(x\right)={\widehat{\beta }}_0+{{\displaystyle \sum }}_{i=1}^p{\widehat{\beta }}_i{x}_i+{{\displaystyle \sum }}_{i=1}^p{\widehat{\beta }}_i{x}_i^2+{{\displaystyle \sum }}_{\begin{array}{c}i=1\\ i<j\end{array}}^p{{\displaystyle \sum }}_{j=1}^p{\widehat{\beta }}_{ij}{x}_i{x}_j,$$

(2)

$${\widehat{\sigma }}_{LSM}\left(x\right)={\widehat{\delta }}_0+{{\displaystyle \sum }}_{i=1}^p{\widehat{\delta }}_i{x}_i+{{\displaystyle \sum }}_{i=1}^p{\widehat{\delta }}_i{x}_i^2+{{\displaystyle \sum }}_{\begin{array}{c}i=1\\ i<j\end{array}}^p{{\displaystyle \sum }}_{j=1}^p{\widehat{\delta }}_{ij}{x}_i{x}_j,$$

(3)

where $\widehat{\beta }$ and $\widehat{\delta }$ are the estimators of the unknown parameters in the process mean and standard deviation functions, respectively. These coefficients are estimated using the conventional LSM, as follows:

$$\widehat{\beta }={\left(X^{\mathrm{T}}X\right)}^{-1}{X}^{\mathrm{T}}\overline{y_{obs}}\mathrm{and}\widehat{\delta }={\left(X^{\mathrm{T}}X\right)}^{-1}{X}^{\mathrm{T}}{s}_{obs},$$

(4)

where $\overline{y_{obs}}$, $s_{obs}$, and $X$ are the observation average, the standard deviation of replicated responses, and design matrix, respectively.

3. Proposed Feed-Forward NN-Based DR Estimation Method

3.1. Feed-Forward NN Structure

NNs may include a broad class of flexible nonlinear regression, discriminant and data reduction models, and nonlinear dynamical systems [51]. An NN consists of simple, highly interconnected computational units called artificial neurons. Zainuddin and Pauline [52] demonstrated that feed-forward NNs from the input to the output are the most popular NNs for use in function approximation. In addition, Cybenko [37], Hornik et al. [36], Funahashi [38], and Hartman et al. [53] demonstrated that a feed-forward NN including one hidden layer could approximate a continuous, arbitrary nonlinear, and multidimensional function. Both processes mean and standard deviation can be two output response characteristics of interest in the RD methodology. The proposed feed-forward NN-based method for estimating the functional relationship associated with input factors and output responses in the RD is depicted in Figure 2.

A vector $x$ comprising $k$ control factors $x_1$,…, $x_i$, …, $x_k$, named as the input layer, is the input to the hidden neuron in the hidden layer. The summation of $k$ inputs of neurons with identified weights (i.e., coefficients) and bias $a$ is transformed using the transfer function $f$ to generate the associated output of the hidden neuron. The outputs of the hidden neuron are used as inputs to the neuron in the output layer. Assume that there are h hidden nodes $1, \dots , j, \dots ,h$, $w_{ij}$, $v_j$,$a_j$, and $b$ denotes the weight connecting input factor $x_i$ to hidden node $j,$ the weight connecting hidden node $j$ to the final output, the bias at the hidden node $j$, and the bias at the output, respectively. Then, the general form of the response function obtained using the proposed NN-based estimation method is as follows:

$${\widehat{y}}_{NN}=g\left\{{{\displaystyle \sum }}_{j=1}^h{v}_j{y}_j^{hid}+b_{out}\right\}=g\left\{{{\displaystyle \sum }}_{j=1}^h{v}_j\left[f\left({{\displaystyle \sum }}_{i=1}^k{w}_{ij}{x}_i+a_j\right)\right]+b_{out}\right\}.$$

(5)

3.2. Back-Propagation Learning Algorithm

Among the learning algorithms reported in the literature for performing NN estimation, the back-propagation algorithm is commonly used for training a network owing to its simplicity and applicability [54]. This algorithm comprises two sequential steps: first, the weights of the NN are initialized randomly, and then, the output of the NN is computed and compared with the actual output. The error at the output layer, that is, the network error, is calculated by comparing the actual output ${\widehat{y}}_{out}$ to the desired value $y_{out}$ to iteratively adjust the weights of the hidden and output layers in accordance with the following function:

$$E=\frac{1}{2}\sum {\left(y_{out}-{\widehat{y}}_{out}\right)}^2.$$

(6)

Epoch is a hyperparameter that defines the number of times the entire training vectors are used to update the weights. During the training process, the error is optimally minimized. The iterative step of the gradient descent algorithm changes weights $w_j$ based on the following relationship:

$$w_j\to w_j+\Delta w_j,\mathrm{where}\Delta w_j=-\eta \frac{\partial E\left(w\right)}{\partial w_j}.$$

(7)

The parameter $\eta (>0)$ is called the learning rate. There are numerous variations in the basic algorithm based on other optimization techniques (namely, conjugate gradient and Newton methods).

The Marquardt–Levenberg algorithm (trainlm) approximates Newton’s method, which is designed to approach the second-order training speed without applying the Hessian matrix.

The resilient back-propagation training algorithm (trainrp) is a local adaptive learning technique, which performs supervised batch learning in a feed-forward NN. Trainrp removes the negative impact of the dimension of the partial derivative. Therefore, only the sign of the derivative implies the direction of a weight update process.

4. Proposed Transfer Functions Based on DFs

4.1. Integration of DFs into Transfer Functions

DF $d\left(y\right)$ scales the possible response (y) value to the range of [0, 1], where $d\left(y\right)=0$ and $d\left(y\right)=1$ represent completely undesirable and desirable values, respectively. Among the three DFs proposed by Derringer and Suich [40] for three situations of a particular response, the “nominal is best” and “smaller is better” DFs are utilized as new transfer functions in the hidden NN layers in order to estimate the process mean and standard deviation functions. Assume that there are $q$ runs, and hence, the summation at each hidden neuron represents vector $y_j$ consisting of $\left\{y_1,\dots ,y_p, \dots ,y_q\right\}$ values, where the input of each hidden neuron can then be calculated as $y_p\left(x\right)={{\displaystyle \sum }}_{i=1}^k{x}_i{w}_{ij}+a_j$ at each run order. Therefore, the associated DF output can be represented as $d_j=\left\{d_1,\dots ,d_p, \dots ,d_q\right\}$. When a response belongs to the “nominal is best” case, the individual DF is defined as

$$d_p\left(y_p\right)=\left\{\begin{array}{c}0 if y_p\left(x\right)<L_p\\ {\left(\frac{y_p\left(x\right)-L_p}{T_p-L_p}\right)}^s if L_p\le y_p\left(x\right)\le T_p\\ {\left(\frac{y_p\left(x\right)-U_p}{T_p-U_p}\right)}^t if T_p\le y_p\left(x\right)\le U_p\\ 0 if y_p\left(x\right)>U_p\end{array}\right.$$

(8)

When a response needs to be minimized, the individual DF is

$$d_p\left(y_p\right)=\left\{\begin{array}{c}1 if y_p\left(x\right)<L_p\\ {\left(\frac{y_p\left(x\right)-U_p}{L_p-U_p}\right)}^r if L_p\le y_p\left(x\right)\le U_p\\ 0 if y_p\left(x\right)>U_p\end{array}\right.$$

(9)

where $L_p$, $T_p$, and $U_p$ denote the lower specification, target, and upper specification values, respectively. Superscripts $r$ and $s$ in Equation (8) and superscript $t$ in Equation (9) represent weight parameters of the “nominal is best” and the “smaller is better” DFs, respectively. When the values of $s, t,$ and $r$ are set to 1, the DFs become linear. For $s<1$, $t<1$, and $r<1$ the functions are convex, and for $s>1$, $t>1$, and $r>1$ the functions become concave. The integration of the “nominal is best” and “smaller is better” DFs as new transfer functions in the hidden layers to estimate the process mean and standard deviation is depicted in Figure 3.

From Equation (5), the proposed estimated process mean and standard deviation functions (namely, ${\widehat{\mu }}_{pro}\left(x\right)$ and ${\widehat{\sigma }}_{pro}\left(x\right)$) are defined, respectively, as

$${\widehat{\mu }}_{pro}\left(x\right)={{\displaystyle \sum }}_{j=1}^{h_{mea{n}_{pro}}}\left\{v_j^{{\mathrm{mean}}_{\mathrm{pro}}}\left[d_{mean}^{*}\left({{\displaystyle \sum }}_{i=1}^k{w}_{ij}^{{\mathrm{mean}}_{\mathrm{pro}}}{x}_i+a_j^{{\mathrm{mean}}_{\mathrm{pro}}}\right)\right]\right\}+b_{mean}^{pro},$$

(10)

$${\widehat{\sigma }}_{pro}\left(x\right)={{\displaystyle \sum }}_{j=1}^{h\text{\_}std\text{\_}pro}\left\{v_j^{\mathrm{std}\text{\_}\mathrm{pro}}\left[d_{std}^{*}\left({{\displaystyle \sum }}_{i=1}^k{w}_{ij}^{\mathrm{std}\text{\_}\mathrm{pro}}{x}_i+a_j^{\mathrm{std}\text{\_}\mathrm{pro}}\right)\right]\right\}+b_{std}^{pro},$$

(11)

where $h\text{\_}mean\text{\_}pro$, $h\text{\_}std\text{\_}pro$, $a_j^{\mathrm{mean}\text{\_}\mathrm{pro}}$, $a_j^{\mathrm{std}\text{\_}\mathrm{pro}}$, $b_{mean}^{pro}$, $b_{std}^{pro}$, $v_j^{\mathrm{mean}\text{\_}\mathrm{pro}}$, $v_j^{\mathrm{std}\text{\_}\mathrm{pro}}$, $w_{ij}^{\mathrm{mean}\text{\_}\mathrm{pro}}$, $w_{ij}^{\mathrm{std}\text{\_}\mathrm{pro}}$, $d_{mean}^{*}$, and $d_{std}^{*}$ denote the number of hidden neurons, bias at the hidden node $j$, bias at the output neuron, weights connecting hidden node $j$ to the output responses, weights connecting input factors $x_i$ to hidden node $j$ of the proposed feed-forward NN, and the optimal DFs for the process mean and standard deviation functions, respectively. In addition, the log-sigmoid and linear functions are usually used as conventional transfer functions in hidden and output layers, respectively. The conventional log-sigmoid function $f$ and linear function $g$ with input layer $x$ are given as

$$f=\frac{1}{1+\exp \left(-x\right)}\mathrm{and}g=x.$$

(12)

Similarly, the estimated process mean and standard deviation functions are given, respectively, as follows:

$${\widehat{\mu }}_{NN}\left(x\right)={{\displaystyle \sum }}_{j=1}^{h\text{\_}mean}\left\{v_j^{\mathrm{mean}}\left[\frac{1}{1+\exp \left(-\left({{\displaystyle \sum }}_{i=1}^k{w}_{ij}^{\mathrm{mean}}{x}_i+a_j^{\mathrm{mean}}\right)\right)}\right]\right\}+b_{mean}^{NN},$$

(13)

$${\widehat{\sigma }}_{NN}\left(x\right)={{\displaystyle \sum }}_{j=1}^{h\text{\_}std}\left\{v_j^{\mathrm{std}}\left[\frac{1}{1+\exp \left(-\left({{\displaystyle \sum }}_{i=1}^k{w}_{ij}^{\mathrm{std}}{x}_i+a_j^{\mathrm{std}}\right)\right)}\right]\right\}+b_{std}^{NN},$$

(14)

where $h\text{\_}mean$, $h\text{\_}std$, $a_j^{\mathrm{mean}}$, $a_j^{\mathrm{std}}$, $b_{mean}^{NN}$, $b_{std}^{NN}$, $v_j^{\mathrm{mean}}$, $v_j^{\mathrm{std}}$, $w_{ij}^{\mathrm{mean}}$, and $w_{ij}^{\mathrm{std}}$ denote the number of hidden neurons, bias at hidden node $j$, bias at the output neuron, weight connecting hidden node $j$ to the final output, and the weight connecting input factor $x_i$ to hidden node $j$ of the feed-forward NNs for the process mean and standard deviation functions, respectively.

4.2. Estimation of Weight Parameters for NN Transfer Functions

The DF can provide a flexible NN transfer function family based on various values of the weight parameter. Among the likelihood-based estimation methods, the MLE method was proposed to estimate the optimal weight parameters for NN transfer functions. On the premise that DF values $d_j$ follow a normal distribution $\left(\mu ,{\sigma }^2\right)$, the distribution of summation $y_j$ can be inferred.

4.2.1. Estimation of Parameter $s$

Using Equation (8), the parametric family in this case can be given as $\left(\mu ,{\sigma }^2,s\right)$. Based on the principles of MLE, parameters $\left(\mu ,{\sigma }^2,s\right)$ can be identified when log-likelihood functions $\mathcal{\ell }\left(d_j,\mu ,{\sigma }^2,s\right)$ or $\mathcal{\ell }\left(d_j,s\right)$ are the highest. The log-likelihood function in relation to summation $\mathrm{y}$ is obtained by multiplying the normal probability density function by the Jacobian of the DF, $\mathit{J}\left(d_j,s\right)$:

$$\mathcal{\ell }\left(d_j,s\right)=\log \left[{{\displaystyle \prod }}_{p=1}^q\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{{\left(d_p-\mu \right)}^2}{2{\sigma }^2}}\times \mathit{J}\left(d_j,s\right)\right],$$

(15)

where

$$\mathit{J}\left(d_j,s\right)={{\displaystyle \prod }}_{p=1}^q\left|\frac{\partial d_p}{\partial y_p}\right|={\left(\frac{s}{T_p-L_p}\right)}^q{{\displaystyle \prod }}_{p=1}^q{\left(\frac{y_p-L_p}{T_p-L_p}\right)}^{s-1}.$$

(16)

Therefore, the log-likelihood function is

$$\mathcal{\ell }\left(d_j,s\right)=-\frac{q}{2}\log {\sigma }^2-\frac{q}{2}\log 2\pi -\frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\mu \right)}^2}{2{\sigma }^2}+q\log \left(\frac{s}{T_p-L_p}\right)+\left(s-1\right){\displaystyle \sum }_{p=1}^q\log \left(\frac{y_p-L_p}{T_p-L_p}\right)$$

(17)

For a fixed value $s$, obtaining the derivative of $\mathcal{\ell }\left(d_j,s\right)$ with respect to ${\sigma }^2$, setting this derivative function equal to zero, and solving this function by ${\sigma }^2$ yields

$${\sigma }^2=\frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\mu \right)}^2}{q}.$$

(18)

Further, obtaining the derivative of $\mathcal{\ell }\left(d_j,s\right)$ with respect to $\mu$, assuming it equal to zero, and solving for $\mu$ yields

$$\mu =\frac{1}{q}{{\displaystyle \sum }}_{p=1}^q{d}_p=\overline{d_p}.$$

(19)

Accordingly, the log-likelihood function can be represented as follows:

$$\mathcal{\ell }\left(d_j,s\right)=-\frac{q}{2}\log 2\pi -\frac{q}{2}\log \frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\overline{d_p}\right)}^2}{q}-\frac{q}{2}+q\log \left(\frac{s}{T_p-L_p}\right)+\left(s-1\right){\displaystyle \sum }_{p=1}^q\log \left(\frac{y_p-L_p}{T_p-L_p}\right)$$

(20)

Optimal value $\widehat{s}$ is specified using the statistical simulation method.

4.2.2. Estimation of Parameter $t$

Similarly, from Equation (8), the log-likelihood function for parameter $t$ is

$$\mathcal{\ell }\left(d_j,t\right)=-\frac{q}{2}\log {\sigma }^2-\frac{q}{2}\log 2\pi -\frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\mu \right)}^2}{2{\sigma }^2}+q\log \left(\frac{t}{T_p-U_p}\right)+\left(t-1\right){\displaystyle \sum }_{p=1}^q\log \left(\frac{y_p-U_p}{T_p-U_p}\right)$$

(21)

The profile log-likelihood function used to estimate parameter $t$ can be defined as follows:

$$\mathcal{\ell }\left(d_j,t\right)=-\frac{q}{2}\log 2\pi -\frac{q}{2}\log \frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\overline{d_p}\right)}^2}{q}-\frac{q}{2}+q\log \left(\frac{t}{T_p-U_p}\right)+\left(t-1\right){\displaystyle \sum }_{p=1}^q\log \left(\frac{y_p-U_p}{T_p-U_p}\right)$$

(22)

Optimal value $\widehat{t}$ is also identified using the statistical simulation method.

4.2.3. Estimation of Parameter $r$

Using Equation (9), the log-likelihood function for parameter $r$ is

$$\mathcal{\ell }\left(d_j,r\right)=-\frac{q}{2}\log {\sigma }^2-\frac{q}{2}\log 2\pi -\frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\mu \right)}^2}{2{\sigma }^2}+q\log \left(\frac{r}{L_p-U_p}\right)+\left(r-1\right){\displaystyle \sum }_{p=1}^q\log \left(\frac{y_p-U_p}{L_p-U_p}\right)$$

(23)

The profile log-likelihood function can then be defined as follows:

$$\mathcal{\ell }\left(d_j,r\right)=-\frac{q}{2}\log 2\pi -\frac{q}{2}\log \frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\overline{d_p}\right)}^2}{q}-\frac{q}{2}+q\log \left(\frac{r}{L_p-U_p}\right)+\left(r-1\right){\displaystyle \sum }_{p=1}^q\log \left(\frac{y_p-U_p}{L_p-U_p}\right)$$

(24)

Optimal value $\widehat{r}$ is also determined using the statistical simulation method.

4.3. Confidence Intervals for Weight Parameters

The asymptotic distribution of a maximum likelihood estimator is proposed to identify confidence intervals for weight parameters because the distribution of parameter $\widehat{\mathit{\theta }}$ estimated using MLE is asymptotically normal [55]. An approximate $\left(1-\mathsf{\alpha }\right)100\%$ confidence interval for a parameter $\mathsf{\theta }$ can be given as follows:

$${\widehat{\mathit{\theta }}}_i\pm z^{*}\widehat{\mathrm{SE}}\left({\widehat{\mathit{\theta }}}_i\right),$$

(25)

where $z^{*}$ refers to the $\left(1-\frac{\alpha }{2}\right){100}^{th}$ percentile of the standard normal distribution and $\widehat{\mathrm{SE}}\left({\widehat{\mathit{\theta }}}_i\right)$ represents the estimated value of the standard error of ${\widehat{\mathit{\theta }}}_i$. Based on the estimates of the parameters $\widehat{\mathit{\theta }}$, the standard errors of the parameters can be defined by the square root of the diagonal elements of the estimated covariance matrix. First, the distribution of $\widehat{\mathit{\theta }}$ is stated, followed by the derivation of the approximate asymptotic distribution of parameters $\widehat{s}$, $\widehat{t},$ or $\widehat{r}$.

The first derivative of log-likelihood function $\mathcal{\ell }(\mathit{\theta }|d_j)$ can be given as

$$V=\frac{\partial \mathcal{\ell }(\mathit{\theta }|d_j)}{\partial \mathit{\theta }}.$$

(26)

The observed Fisher’s total information matrix is given as

$$\widehat{\mathit{J}}\left(\mathit{\theta }\right)=-\frac{\partial V}{\partial \mathit{\theta }}=-\frac{{\partial }^2\mathcal{\ell }(\mathit{\theta }|d_j)}{\partial {\mathit{\theta }}^2}.$$

(27)

Therefore, Fisher’s total information matrix $\mathit{J}$ can then be defined as follows:

$$\mathit{J}\left(\mathit{\theta }\right)=E\left[\widehat{\mathit{J}}\left(\mathit{\theta }\right)\right].$$

(28)

The distribution of $\sqrt{q}\left(\widehat{\mathit{\theta }}-\mathit{\theta }\right)$ as $q\to \infty$ converges to a normal distribution. In particular,

$${\left[\frac{\mathit{J}\left(\mathit{\theta }\right)}{q}\right]}^{\frac{1}{2}}\sqrt{q}\left(\widehat{\mathit{\theta }}-\mathit{\theta }\right)\stackrel{dist}{\to }N\left(0,I\right)$$

(29)

where $\mathit{J}\left(\mathit{\theta }\right)$ is Fisher’s total information matrix [55]. $\mathit{J}\left(\mathit{\theta }\right)$ is typically not known, but by substituting a $q^{-\frac{1}{2}}$ consistent estimator, $\frac{\widehat{\mathit{J}}\left(\widehat{\mathit{\theta }}\right)}{q}$ for $\frac{\mathit{J}\left(\mathit{\theta }\right)}{q}$ in Equation (29), the same result can be obtained. Accordingly,

$$\widehat{\mathit{J}}{\left(\widehat{\mathit{\theta }}\right)}^{\frac{1}{2}}\left(\widehat{\mathit{\theta }}-\mathit{\theta }\right)\stackrel{dist}{\to }N\left(0,I\right)\mathrm{and}\widehat{\mathit{\theta }}~N\left(\mathit{\theta }, \widehat{\mathit{J}}{\left(\widehat{\mathit{\theta }}\right)}^{-1}\right).$$

(30)

4.3.1. Confidence Intervals for Parameter $\mathrm{s}$

The general parameter $\mathit{\theta }$ can be determined as $\mathit{\theta }={\left(\mu {\sigma }^2 s\right)}^T$. From Equation (27), $-\widehat{\mathit{J}}\left(\mathit{\theta }\right)$ can be given as

$$-\widehat{\mathit{J}}\left(\mathit{\theta }\right)=\left(\begin{array}{ccc}\frac{{\partial }^2\mathcal{\ell }(\mathit{\theta }|d_j)}{\partial {\mu }^2}& \frac{{\partial }^2\mathcal{\ell }(\mathit{\theta }|d_j)}{\partial \mu \partial {\sigma }^2}& \frac{{\partial }^2\mathcal{\ell }(\mathit{\theta }|d_j)}{\partial \mu \partial s}\\ \frac{{\partial }^2\mathcal{\ell }(\mathit{\theta }|d_j)}{\partial {\sigma }^2\partial \mu }& \frac{{\partial }^2\mathcal{\ell }(\mathit{\theta }|d_j)}{\partial {\left({\sigma }^2\right)}^2}& \frac{{\partial }^2\mathcal{\ell }(\mathit{\theta }|d_j)}{\partial {\sigma }^2\partial s}\\ \frac{{\partial }^2\mathcal{\ell }(\mathit{\theta }|d_j)}{\partial s\partial \mu }& \frac{{\partial }^2\mathcal{\ell }(\mathit{\theta }|d_j)}{\partial s\partial {\sigma }^2}& \frac{{\partial }^2\mathcal{\ell }(\mathit{\theta }|d_j)}{\partial s^2}\end{array}\right).$$

(31)

From the log-likelihood function in Equation (17), $-\widehat{\mathit{J}}\left(\mathit{\theta }\right)$ can be calculated as follows (see Appendix A.1):

$$-\widehat{\mathit{J}}\left(\mathit{\theta }\right)=\left(\begin{array}{ccc}-\frac{q}{{\sigma }^2}& -\frac{{{\displaystyle \sum }}_{p=1}^q\left(d_p-\mu \right)}{{\left({\sigma }^2\right)}^2}& 0\\ -\frac{{{\displaystyle \sum }}_{p=1}^q\left(d_p-\mu \right)}{{\left({\sigma }^2\right)}^2}& \frac{q}{2}\frac{1}{{\left({\sigma }^2\right)}^2}-\frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\mu \right)}^2}{{\left({\sigma }^2\right)}^3}& 0\\ 0& 0& -\frac{q}{s^2}\end{array}\right)=\left(\begin{array}{ccc}\widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{11}& \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{12}& \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{13}\\ \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{21}& \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{22}& \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{23}\\ \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{31}& \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{32}& \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{33}\end{array}\right)$$

(32)

The square root of the diagonal elements of $\widehat{\mathit{J}}{\left(\widehat{\mathit{\theta }}\right)}^{-1}$ represents an estimate of the standard error for each estimator. An estimator for the standard error of $\widehat{s}$ is of particular interest and can be found using the inverses of the partitioned matrices. First, the partition $-\widehat{\mathit{J}}\left(\mathit{\theta }\right)$ is as follows:

$$-\widehat{\mathit{J}}\left(\mathit{\theta }\right)=\left(\begin{array}{cc}\widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{11}& \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{12}\\ \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{21}& \widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{22}\end{array}\right).$$

(33)

Next, one estimator for $\mathrm{Var}\left(\widehat{s}\right)$ can be obtained as follows:

$$\begin{array}{c}\mathrm{Var}\left(\widehat{s}\right)={\left(\widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{11}\widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{22}-\widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{21}\widehat{\mathit{J}}{\left(\mathit{\theta }\right)}_{12}\right)}^{-1}\\ =-{\left[\frac{q}{{\sigma }^2}\left(\frac{q}{2}\frac{1}{{\left({\sigma }^2\right)}^2}-\frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\mu \right)}^2}{{\left({\sigma }^2\right)}^3}\right)+{\left(\frac{{{\displaystyle \sum }}_{p=1}^q\left(d_p-\mu \right)}{{\left({\sigma }^2\right)}^2}\right)}^2\right]}^{-1}.\end{array}$$

(34)

The associated standard error of $\widehat{s}$ can be calculated by

$$\widehat{\mathrm{SE}}\left(\widehat{s}\right)=\frac{1}{\sqrt{-\left[\frac{q}{{\sigma }^2}\left(\frac{q}{2}\frac{1}{{\left({\sigma }^2\right)}^2}-\frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\mu \right)}^2}{{\left({\sigma }^2\right)}^3}\right)+{\left(\frac{{{\displaystyle \sum }}_{p=1}^q\left(d_p-\mu \right)}{{\left({\sigma }^2\right)}^2}\right)}^2\right]}}.$$

(35)

Therefore, an approximation $\left(1-\mathsf{\alpha }\right)100\%$ confidence interval of $s$ can be obtained by using

$$\widehat{s}\pm z^{*}\widehat{\mathrm{SE}}\left(\widehat{s}\right).$$

(36)

4.3.2. Confidence Intervals for Parameter $t$

Similarly, a confidence interval of $t$ can be obtained by using

$$\widehat{t}\pm z^{*}\widehat{\mathrm{SE}}\left(\widehat{t}\right).$$

(37)

The standard error of $\widehat{t}$ can be defined as follows (see Appendix A.2):

$$\widehat{\mathrm{SE}}\left(\widehat{t}\right)=\frac{1}{\sqrt{-\left[\frac{q}{{\sigma }^2}\left(\frac{q}{2}\frac{1}{{\left({\sigma }^2\right)}^2}-\frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\mu \right)}^2}{{\left({\sigma }^2\right)}^3}\right)+{\left(\frac{{{\displaystyle \sum }}_{p=1}^q\left(d_p-\mu \right)}{{\left({\sigma }^2\right)}^2}\right)}^2\right]}}$$

(38)

4.3.3. Confidence Intervals for Parameter $r$

Similarly, an confidence interval of $r$ can be obtained by using

$$\widehat{r}\pm z^{*}\widehat{\mathrm{SE}}\left(\widehat{r}\right).$$

(39)

The standard error of $\widehat{r}$ can be defined as follows (see Appendix A.3):

$$\widehat{\mathrm{SE}}\left(\widehat{r}\right)=\frac{1}{\sqrt{-\left[\frac{q}{{\sigma }^2}\left(\frac{q}{2}\frac{1}{{\left({\sigma }^2\right)}^2}-\frac{{{\displaystyle \sum }}_{p=1}^q{\left(d_p-\mu \right)}^2}{{\left({\sigma }^2\right)}^3}\right)+{\left(\frac{{{\displaystyle \sum }}_{p=1}^q\left(d_p-\mu \right)}{{\left({\sigma }^2\right)}^2}\right)}^2\right]}}$$

(40)

5. Case Study

This case study was performed using the printing data example proposed by Vining and Myers [6] and Lin and Tu [7]. The impacts of three input factors, that is, speed ($x_1$), pressure ($x_2$), and distance ($x_3$), on the capabilities of a printing machine that applies colored inks to package levels (y) were investigated. Each input factor had three levels, resulting in a total of 27 runs. Three replications were performed for each data combination.

In comparative studies between methods, the expected quality loss (EQL) is usually applied as a critical optimization criterion in RD. The expectation of the loss function is defined as

$$\mathrm{EQL}=\mathit{\theta }\left[{\left(\widehat{\mu }\left(x\right)-\tau \right)}^2+{\widehat{\sigma }}^2\left(x\right)\right].$$

(41)

where $\mathit{\theta }$ represents a positive loss coefficient. $\widehat{\mu }\left(x\right)$, $\tau$, and $\widehat{\sigma }\left(x\right)$ are the estimated process mean function, desirable target value, and estimated standard deviation function, respectively. In this case study, the target value was $\tau =500$. Using MATLAB, the process-estimated mean and standard deviation functions obtained using the LSM based on RSM were as follows:

$${\widehat{\mu }}_{\mathrm{LSM}}\left(x\right)=327.6296+x^T{m}_{LSM}+x^T{M}_{LSM}x ,$$

(42)

$${\widehat{\sigma }}_{\mathrm{LSM}}\left(x\right)=34.8832+x^T{n}_{LSM}+x^T{N}_{LSM}x$$

(43)

where

$$\begin{gathered} x=\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right],m_{LSM}=\left[\begin{array}{c}177.000\\ 109.426\\ 131.463\end{array}\right],M_{LSM}=\left[\begin{array}{ccc}32.000& 66.028& 75.472\\ 66.028& -22.389& 43.583\\ 75.472& 43.583& -29.056\end{array}\right], \\ {n}_{LSM}=\left[\begin{array}{c}11.527\\ 15.323\\ 29.190\end{array}\right],\mathrm{and}{N}_{LSM}=\left[\begin{array}{ccc}4.204& 7.7120& 5.1093\\ 7.720& -1.316& 14.082\\ 5.109& 14.082& 16.778\end{array}\right]. \end{gathered}$$

For comparison, the criteria used in the NN (mainly, the number of hidden neurons, training algorithms, and the number of epochs of NNs) for the conventional log-sigmoid transfer function in the hidden layer were the same as those of the proposed NNs. The NN information for the conventional log-sigmoid transfer function and the proposed DR estimation method are presented in

Table 1. Feed-forward NNs information used to estimate DR functions.

Model		Transfer Functions	Training Function	Architecture	Number of Epochs
Conventional	Mean	Logsig-Purelin	Trainlm	3-20-1	5
	Standard deviation	Logsig-Purelin	Trainrp	3-20-1	83
Proposed	Mean	“nominal is best”-Purelin	Trainlm	3-20-1	5
	Standard deviation	“smaller is better”-Purelin	Trainrp	3-20-1	83

. The NN architecture consists of several input factors, hidden neurons, and output responses. The weight and bias matrix values of the NNs using the conventional log-sigmoid transfer function for the process mean and standard deviation functions are presented in

Table 2. Weight and bias values from the NNs approach by using the conventional log-sigmoid transfer function.

(a) Estimated Process Mean						(b) Estimated Process Standard Deviation
Weights				Biases		Weights				Biases
${\mathit{W}}_{\mathit{i}\mathit{j}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}}$			${\left({\mathit{V}}_{\mathit{j}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}}\right)}^{\mathit{T}}$	${\mathit{a}}_{\mathit{j}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}}$	${\mathit{b}}_{\mathit{m}\mathit{e}\mathit{a}\mathit{n}}^{\mathit{N}\mathit{N}}$	${\mathit{W}}_{\mathit{i}\mathit{j}}^{\mathit{s}\mathit{t}\mathit{d}}$			${\left({\mathit{V}}_{\mathit{j}}^{\mathit{s}\mathit{t}\mathit{d}}\right)}^{\mathit{T}}$	${\mathit{a}}_{\mathit{j}}^{\mathit{s}\mathit{t}\mathit{d}}$	${\mathit{b}}_{\mathit{s}\mathit{t}\mathit{d}}^{\mathit{N}\mathit{N}}$
7.048	3.571	−1.123	0.134	−7.259	−0.343	6.693	3.771	−0.855	0.0516	−7.531	−0.456
4.726	−5.71	−1.371	−0.212	−6.831		4.928	−5.169	−1.884	−0.038	−7.045
−4.981	4.672	3.442	−0.789	6.003		−4.907	4.677	3.435	−0.147	5.755
4.459	3.990	3.498	1.079	−6.208		4.251	4.486	3.809	0.467	−5.651
2.679	3.682	−6.266	−0.199	−4.201		2.864	3.086	−5.745	0.330	−4.851
−6.436	4.389	0.793	0.209	3.175		−6.849	3.647	0.288	0.467	3.440
−4.908	5.623	−1.339	0.124	2.863		−5.123	5.469	−1.166	−0.356	2.874
1.562	−4.993	5.438	0.190	−2.048		1.447	−4.815	5.474	−0.271	−2.451
6.692	2.115	3.035	−0.139	−0.859		6.152	2.694	2.570	−0.250	−1.183
5.521	−4.127	3.079	0.099	−0.243		6.115	−3.555	2.652	−0.034	0.050
−5.977	3.126	−3.407	0.325	−0.669		−6.131	2.967	−3.265	0.230	−0.850
5.256	−5.103	1.936	0.633	1.256		5.651	−4.723	1.535	−0.040	1.651
6.483	−3.230	2.033	−0.294	2.105		6.082	−3.433	2.667	0.697	1.926
−0.589	−5.911	−4.442	−0.462	−3.229		−0.441	−5.646	−4.983	−0.529	−2.783
3.818	−4.951	−4.368	−0.443	3.550		4.070	−5.306	−4.141	0.407	3.526
−6.018	4.983	−0.135	−0.385	−4.140		−6.091	4.643	−0.476	−0.545	−4.469
−1.158	2.826	6.948	−0.294	−5.321		−1.107	3.380	6.462	−0.155	−5.651
6.528	−3.113	−2.385	0.419	6.011		6.112	−2.436	−2.970	−0.514	6.074
4.157	6.574	1.314	0.425	6.397		4.160	5.845	1.643	0.072	7.251
4.931	−5.012	−2.952	−0.233	7.576		4.467	−4.909	−3.407	−0.070	7.674

. Based on the structures, training algorithms, transfer functions, and the number of epochs of the proposed feed-forward NNs used to estimate the functions of process mean and standard deviation, optimal weight parameters $\widehat{s}$, $\widehat{t}$, and $\widehat{r}$ were calculated using Equations (20), (22), and (24), respectively. As shown in

Table 3. Estimated weight parameters and associated confidence intervals.

Hidden Neurons	$\widehat{\mathit{s}}$	$\widehat{\mathit{s}}-$ ${\mathit{z}}^{*}\widehat{\mathit{S}\mathit{E}}\left(\widehat{\mathit{s}}\right)$	$\widehat{\mathit{s}}+$ ${\mathit{z}}^{*}\widehat{\mathit{S}\mathit{E}}\left(\widehat{\mathit{s}}\right)$	$\widehat{\mathit{t}}$	$\widehat{\mathit{t}}-$ ${\mathit{z}}^{*}\widehat{\mathit{S}\mathit{E}}\left(\widehat{\mathit{t}}\right)$	$\widehat{\mathit{t}}+$ ${\mathit{z}}^{*}\widehat{\mathit{S}\mathit{E}}\left(\widehat{\mathit{t}}\right)$	$\widehat{\mathit{r}}$	$\widehat{\mathit{r}}-$ ${\mathit{z}}^{*}\widehat{\mathit{S}\mathit{E}}\left(\widehat{\mathit{r}}\right)$	$\widehat{\mathit{r}}+$ ${\mathit{z}}^{*}\widehat{\mathit{S}\mathit{E}}\left(\widehat{\mathit{r}}\right)$
1st	1.150	1.024	1.276	0.100	−0.007	0.207	0.180	0.118	0.242
2nd	0.100	−0.006	0.206	0.180	0.095	0.265	0.160	0.128	0.192
3rd	0.120	0.008	0.232	0.100	0.019	0.181	0.190	0.139	0.241
4th	0.100	−0.022	0.222	0.150	0.056	0.244	0.910	0.777	1.043
5th	0.100	−0.038	0.238	0.110	0.017	0.203	0.170	0.070	0.270
6th	0.100	−0.056	0.256	0.650	0.469	0.831	0.310	0.156	0.464
7th	0.100	−0.076	0.276	0.100	−0.056	0.256	0.120	−0.236	0.476
8th	0.100	−0.108	0.308	0.140	0.005	0.275	0.100	−0.115	0.315
9th	0.100	−0.187	0.387	0.150	0.023	0.277	0.160	0.029	0.291
10th	0.820	0.667	0.973	0.370	0.245	0.495	0.110	0.012	0.208
11th	0.110	−0.063	0.283	1.050	0.852	1.248	0.100	0.029	0.171
12th	0.130	−0.073	0.333	0.100	−0.137	0.337	0.150	0.095	0.205
13th	0.100	−0.123	0.323	0.110	−0.173	0.393	2.370	2.249	2.491
14th	0.120	−0.139	0.379	0.120	−0.283	0.523	0.390	0.267	0.513
15th	0.310	0.019	0.601	0.190	−0.424	0.804	0.150	0.013	0.287
16th	0.480	0.313	0.647	0.100	−0.324	0.524	0.230	0.071	0.389
17th	0.110	−0.069	0.289	3.200	3.085	3.315	0.120	−0.079	0.319
18th	0.240	0.042	0.438	0.100	−0.017	0.217	0.100	−0.261	0.461
19th	0.180	−0.046	0.406	0.120	0.001	0.239	0.100	−0.167	0.367
20th	0.100	−0.154	0.354	0.100	−0.021	0.221	0.400	0.204	0.596

, twenty hidden neurons in the hidden layer are used to estimate mean function, and twenty hidden neurons in the hidden layer are used to estimate standard deviation function. For each hidden neuron, a desirability function is proposed as a transfer function. In addition, weight parameters (i.e., s, t, and r) for DFs identified in Equations (8) and (9) and associated confidence intervals are demonstrated in Table 3. The respective 95% confidence intervals for these weight parameters calculated by using Equations (35), (38), and (40) are presented in Table 3. The weight and bias matrix values of the feed-forward NNs using the proposed transfer functions for the process mean and standard deviation functions are presented in

Table 4. Weight and bias values by using the NN-based DR estimation method.

(a) Estimated Mean Response							(b) Estimated Standard Deviation Response
Weights					Biases		Weights				Biases
${\mathit{W}}_{\mathit{i}\mathit{j}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}\text{\_}\mathit{p}\mathit{r}\mathit{o}}$			${\left({\mathit{V}}_{\mathit{j}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}\text{\_}\mathit{p}\mathit{r}\mathit{o}}\right)}^{\mathit{T}}$	${\mathit{a}}_{\mathit{j}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}\text{\_}\mathit{p}\mathit{r}\mathit{o}}$		${\mathit{b}}_{\mathit{m}\mathit{e}\mathit{a}\mathit{n}}^{\mathit{p}\mathit{r}\mathit{o}}$	${\mathit{W}}_{\mathit{i}\mathit{j}}^{\mathit{s}\mathit{t}\mathit{d}\text{\_}\mathit{p}\mathit{r}\mathit{o}}$			${\left({\mathit{V}}_{\mathit{j}}^{\mathit{s}\mathit{t}\mathit{d}\text{\_}\mathit{p}\mathit{r}\mathit{o}}\right)}^{\mathit{T}}$	${\mathit{a}}_{\mathit{j}}^{\mathit{s}\mathit{t}\mathit{d}\text{\_}\mathit{p}\mathit{r}\mathit{o}}$	${\mathit{b}}_{\mathit{s}\mathit{t}\mathit{d}}^{\mathit{p}\mathit{r}\mathit{o}}$
7.048	3.571	−1.123	0.134	−7.259		−0.343	6.693	3.771	−0.855	0.0516	−7.531	−0.456
4.726	−5.71	−1.371	−0.212	−6.831			4.928	−5.169	−1.884	−0.038	−7.045
−4.981	4.672	3.442	−0.789	6.003			−4.907	4.677	3.435	−0.147	5.755
4.459	3.990	3.498	1.079	−6.208			4.251	4.486	3.809	0.467	−5.651
2.679	3.682	−6.266	−0.199	−4.201			2.864	3.086	−5.745	0.330	−4.851
−6.436	4.389	0.793	0.209	3.175			−6.849	3.647	0.288	0.467	3.440
−4.908	5.623	−1.339	0.124	2.863			−5.123	5.469	−1.166	−0.356	2.874
1.562	−4.993	5.438	0.190	−2.048			1.447	−4.815	5.474	−0.271	−2.451
6.692	2.115	3.035	−0.139	−0.859			6.152	2.694	2.570	−0.250	−1.183
5.521	−4.127	3.079	0.099	−0.243			6.115	−3.555	2.652	−0.034	0.050
−5.977	3.126	−3.407	0.325	−0.669			−6.131	2.967	−3.265	0.230	−0.850
5.256	−5.103	1.936	0.633	1.256			5.651	−4.723	1.535	−0.040	1.651
6.483	−3.230	2.033	−0.294	2.105			6.082	−3.433	2.667	0.697	1.926
−0.589	−5.911	−4.442	−0.462	−3.229			−0.441	−5.646	−4.983	−0.529	−2.783
3.818	−4.951	−4.368	−0.443	3.550			4.070	−5.306	−4.141	0.407	3.526
−6.018	4.983	−0.135	−0.385	−4.140			−6.091	4.643	−0.476	−0.545	−4.469
−1.158	2.826	6.948	−0.294	−5.321			−1.107	3.380	6.462	−0.155	−5.651
6.528	−3.113	−2.385	0.419	6.011			6.112	−2.436	−2.970	−0.514	6.074
4.157	6.574	1.314	0.425	6.397			4.160	5.845	1.643	0.072	7.251
4.931	−5.012	−2.952	−0.233	7.576			4.467	−4.909	−3.407	−0.070	7.674

.

The estimated process mean and standard deviation functions obtained by applying the conventional LSM-RSM, the NN-based estimation method using the conventional log-sigmoid transfer function, and the proposed DR approach with the estimated optimal weight parameters listed in Table 3, along with the associated coefficients of determination $R^2$ of each estimated response function, are illustrated in Figure 4, Figure 5 and Figure 6 as contour plots, respectively.

The optimal control factor settings, the optimal process parameters (i.e., process mean, bias, and variance), and their associated EQL values obtained by the three different approaches (i.e., the conventional LSM based on RSM, the NN-based estimation method using the conventional log-sigmoid transfer functions, and the proposed NN-based DR estimation method) are presented in

Table 5. Comparative results of various methods.

Estimation Model	Optimal Factor Settings			Process Mean	Process Bias	Process Variance	EQL
	${\mathit{x}}_1$	${\mathit{x}}_2$	${\mathit{x}}_3$
LSM	1.000	0.072	−0.250	494.672	5.328	1977.533	2005.917
Feed-forward NN	−0.067	0.170	1.000	499.321	0.679	397.507	397.967
Proposed method	0.133	−0.014	0.096	499.885	0.115	103.900	103.913

.

As demonstrated in Table 5, the process variance obtained from the NN-based estimation methods is remarkably smaller than that of the conventional statistical LSM approach. Therefore, NN-based estimation methods can provide considerably smaller EQL values (i.e., significant criteria to evaluate the level of quality) than those of the conventional LSM approach in this particular example, although the same RD optimization model is utilized. Of the two NN-based estimation methods, the proposed DR approach provides better optimal solutions when compared with that of the NN-based estimation method that uses the conventional log-sigmoid transfer function because the variance obtained from the proposed DR approach is almost zero in this particular example. Therefore, in terms of EQL, the best optimal solutions can be obtained using the NN-based DR estimation method. The criterion spaces (squared bias vs. variance) of the three estimated functions by using the conventional LSM approach based on RSM, the N-based estimation method using the log-sigmoid transfer function, and the proposed DR approach are illustrated in Figure 7. A green cross marks the optimal settings in each figure.

6. Conclusions and Further Studies

This study proposes a feed-forward NN-based DR estimation method to estimate the process mean and standard deviation functions for RSM and RD modeling. The proposed method can avoid basic error assumptions of the conventional LSM approach in this particular situation. Further, integrating the DFs into the NN structures in the proposed method results in a general and flexible transfer function family for transforming data based on the proposed DFs. The optimal weight parameters of the DF family can be determined using the MLE and the confidence intervals of the optimal weight parameters using the proposed asymptotic distribution of maximum-likelihood estimators. The best transfer function in the NN structure was achieved using the proposed method. The proposed NN-based DR estimation method provides better solutions for EQL than those of the conventional LSM approach and the NN-based estimation method that uses the conventional log-sigmoid transfer function in the particular case study example. In order to improve the reliability of the proposed methods, different types of experimental data (i.e., small and large data sets, different DoE results, and small and large sets of replications) should be considered.

In further studies, the assumption that the error follows a normal distribution and whether the DF values can follow any distribution will be investigated. Additionally, the optimal weight parameters of the DF-based transfer functions were estimated using a Bayesian approach or the Newton–Raphson algorithm. We then plan to determine the confidence intervals of the optimal weight parameters using a t-distribution. In addition, a more comprehensive comparative study between the proposed NN models and higher-order models of conventional RSM can be a significant further research issue. Based on this comparative study and a Bayesian approach, we may develop a new optimal estimation system by integrating the proposed NN models and conventional RSM models.