Machine Learning Alternatives to Response Surface Models

Abstract

In the Design of Experiments, we seek to relate response variables to explanatory factors. Response Surface methodology (RSM) approximates the relation between output variables and a polynomial transform of the explanatory variables using a linear model. Some researchers have tried to adjust other types of models, mainly nonlinear and nonparametric. We present a large panel of Machine Learning approaches that may be good alternatives to the classical RSM approximation. The state of the art of such approaches is given, including classification and regression trees, ensemble methods, support vector machines, neural networks and also direct multi-output approaches. We survey the subject and illustrate the use of ten such approaches using simulations and a real use case. In our simulations, the underlying model is linear in the explanatory factors for one response and nonlinear for the others. We focus on the advantages and disadvantages of the different approaches and show how their hyperparameters may be tuned. Our simulations show that even when the underlying relation between the response and the explanatory variables is linear, the RSM approach is outperformed by the direct neural network multivariate model, for any sample size (<50) and much more for very small samples (15 or 20). When the underlying relation is nonlinear, the RSM approach is outperformed by most of the machine learning approaches for small samples (n ≤ 30).

1. Introduction

Design of Experiments (DoE) is used in any manufacturing process where, on the one hand, input parameters called factors affect a process or a formula and can be controlled, and on the other hand, output parameters of this process, which therefore represent the results, are called responses. For example, input parameters can be a temperature or a pH, and output parameters can be a yield or an impurity rate. Experiments must be performed to determine a relationship between factors and responses. DoE allows us to select these experiments in such a way that the position of the experimental points in the space to be studied is optimal. In particular, spatial modeling can be used to better understand the process and to predict the variation of responses throughout the variation range of the factors. In these experiments, responses and factors may be quantitative, qualitative, or a mix of both. The number of experiments is often very small, because each may need several days or even weeks, and the cost can be considerable. Among the important tasks that are used in the DoE approach, response surface methodology (RSM) estimates the values of all responses for any combination of factors’ values. This is achieved by means of a regression of the responses on the factors. The most common approach considers each response separately and then uses polynomial regression, including or excluding interactions between factors. Polynomial regression suffers from several limitations, as follows: (i) the model is linear in the coefficients; (ii) to get a good approximation of the surfaces, higher degrees of polynomials may be necessary together with interactions, and this considerably increases the number of variables to include in the model and therefore the number of experiments to be performed; (iii) the model may only be estimated if the number of experiments is larger than the number of factors included in the model; and (iv) polynomial models are inappropriate in the case of nonlinear systems [1].

Several papers have suggested machine learning (ML) approaches as an alternative to polynomial regression. The approaches that are considered in the literature include Support Vector Machines (SVM) [1,2,3,4,5,6], Neural Networks (NN) [1,2,4,6,7,8,9], Random Forests (rf) [1,3,4,10], Boosting and its extension [1,3], Extra Tree regression (ET) [3,4] and Classification And Regression Trees (CART) [1]. In these papers, the authors compared some of these approaches only on real datasets, using specific settings with respect to the nature and dimensions of the data. As well as focusing on a specific dataset, the comparisons considered very few approaches at the same time and were based on very different metrics. In this work, we will mainly consider the contribution of ML with respect to the RSM; these approaches have been shown to be very efficient in various fields, such as ecology [11] and epidemiology [12].

In this paper, we give a state of the art of the ML approaches that have often been used as alternatives to polynomial regression in RSM, including direct multi-output approaches. We will briefly describe these approaches and compare them using simulation models in various situations with different sample sizes and over a real DoE case study. The simulations were actually limited to numerical variables; including categorical ones as input or output did not have any significant effect on our main results or conclusions. The advantage and the contribution of our work lies, on one hand, in the choices of the simulations; some responses are in favor of the RSM approach and others are not. On the other hand, to our knowledge, direct multivariate output approaches have never been used in this context.

This paper is organized as follows. Section 2 gives a summary of the polynomial RSM-based approach and a brief description of the most well-known ML approaches. Section 3 gives the state of the art for the applications of these approaches in various domains where DoE is practiced. In Section 4, we compare all of the ML approaches to polynomial regression on simulated multiple output datasets, varying sample sizes and optimizing the models’ hyperparameters. A real dataset is used for similar experiments. The last section includes our discussions and conclusion.

2. Response Surface Methodology

Among the many objectives of experimental design, Response Surface Methodology (RSM) [13] is used to determine the value of one or more outputs at any point in the experimental domain of interest [14] without carrying out an infinite number of experiments. This is achieved using a regression model of the form:

$$Y=f(X_1,X_2,\dots X_K)+ϵ$$

where f is the unknown regression function. To estimate f, we need to make some hypothesis about its shape. Polynomial models are most commonly used to address this issue. Using high-degree polynomials, f may be approximated correctly. This is a linear model in the coefficients. With K factors and degree d, the number of coefficients of the model is $\frac{(K+d)!}{K!d!}$. According to the type of study, polynomial models of degree 1 or 2 are used because the number of observations is, in general, very small (often less than 20).

Let $U_k$ be the original K controlled factors, which are also called the natural variables, whose space is called the experimental domain. These variables are generally on different scales, and they may bias the modeling results. Thus, they are often normalized and linearly transformed into codified variables $X_1,X_2,\dots ,X_K$, which are dimensionless quantities and with the same range. The most commonly used transformation is $X_k=\frac{(U_k-U_k^0)}{\Delta U_k}$, where $U_k^0=\frac{(max{U}_k+min{U}_k)}{2}$ is called the central value and $\Delta U_k=\frac{(max{U}_k-min{U}_k)}{2}$ is the step. The transformed variables ${\left(X_k\right)}_{k=1,\dots ,K}$ lie in the interval $[-1,1]$, and the $n\times K$ matrix $X={\left(X_k\right)}_{k=1,\dots ,K}$ is called the experimental design, where n is the sample size. A polynomial model of degree 2 may be written as:

$$Y={\beta }_0+\sum _{k=1}^K{\beta }_k{X}_k+\sum _{k=1}^K{\beta }_{kk}{X}_k^2+\sum _{1\le j<k\le K}{\beta }_{j,k}{X}_j{X}_k$$

where Y is the response and $\beta$ are the unknown coefficients. Once the design of the experiments is carried out and the data are available, the coefficients of the model are estimated by ordinary least squares, and their estimate is given by:

$$b={\left(X^{{}^{\prime }}X\right)}^{-1}{X}^{{}^{\prime }}Y$$

where $X=(1,X_1,X_2,\dots ,X_K,X_1^2,X_2^2,\dots ,X_K^2,X_1{X}_2,X_1{X}_3,\dots ,X_{K-1}{X}_K)$ is the model matrix, $X^{{}^{\prime }}$ is the transpose of X, Y is the response, $X^{{}^{\prime }}X$ is the information matrix, and ${\left(X^{{}^{\prime }}X\right)}^{-1}$ is the dispersion matrix.

Test points may be used to validate the model, or an analysis of the variance may be performed. After validation, the model is used to predict the variation of responses throughout the experimental domain represented by a response surface (Figure 1). These predictions can then be used to construct a Design Space, which is a region in which the response specifications will be achieved with a fixed probability [15]. Nevertheless, due to the small sample sizes available in experimental designs, polynomial models of degree 2 may give very poor estimation for surface responses.

Alternative models have been proposed for several years. ML models present more flexible methods of estimating a response surface function: they are nonparametric and nonlinear models, and may even be efficient when the number of experiments is small compared to the number of factors.

3. Machine Learning Approaches

In this section, we will give a brief description of the most commonly used approaches in ML, as follows: k-nearest neighbors (knn), CART, Ensemble models (Bagging, Boosting, and rf), SVM, and NN.

3.1. knn

This is one of the simplest ML approaches that may be used in regression and in classification [17]. It is also quite different from all of the other approaches, in that the model estimation and prediction are embedded. Once the number of neighbors k is fixed, for any new observation x, the method seeks its k nearest neighbors within the learning set. The prediction for x is the average of the neighbors’ outputs in regression, or the majority vote in classification. The decision rule may be refined using, for instance, weighted averages or weighted majority votes, where weights may be inversely proportional to the distances of the neighbors from x. Note that the distance used to identify neighbors is a hyperparameter for this approach, together with k.

3.2. CART

Classification and Regression Trees [18] are based on recursive partitioning of the input space using rectangles. These partitions are represented by a sequence of binary splitting rules of the form $X_m<s$, where $X_m$ is one of the input variables and s is a threshold over it. The split at each stage of the partitioning is optimized by an exhaustive search looking for the couple variable and threshold that minimizes the heterogeneity of the obtained subsamples. Heterogeneity is measured with respect to the output y. When the output is continuous, the deviance is used (the variance multiplied by the sample size). For discrete output, entropy or Gini criteria are used.

CART is is the most popular nonlinear and nonparametric method when one wants to understand how the model relates the outputs to the explanatory variables. It is one of the few methods which is graphically representable.

3.3. Ensemble Methods

The idea of ensemble methods, in their simplest approach, is to build K bootstrap samples of the dataset at hand, adjust a model chosen from within a class of functions (e.g., decision trees) for each bootstrap sample (denoted $f_k$), and then use the K models for predictions. In regression, the prediction by the ensemble is a weighted average of the K predictions given from each $f_k$. In classification, it is a weighted majority vote over the K predictions given by the models $f_k$.

Several variants of ensemble methods exist, differing either in the way in which the bootstrap samples are generated, in the estimation of each $f_k$, in the way in which the individual models $f_k$ are combined, or in the algorithm used to estimate the whole process.

Random Forests (rf; [19] are among the most well-known and used approaches. This approach combines trees built over bootstrap samples by simple averages in regression and majority vote in classification. The trees that are built in rf may be very big, because they are not optimized, and the choice of the splits at each node in the trees is made on a small subset of variables chosen at random. Because bootstrap samples are independent, random forests may be parallelized.

Boosting [20] is quite different because the trees built over bootstrap samples are obtained sequentially; weights $w_i$ are associated to each observation in the original dataset, and at each step of the algorithm, a tree $f_k$ is built and tested over the original dataset. Observations whose prediction at step k is wrong will have their weight increased. The modified weights are used at the next step $k+1$ for the next bootstrap sample. Each tree that is built in the ensemble has a weight $W_k$ related to its performance, and the final output is a weighted majority vote in classification and a weighted average in regression. The trees’ estimates and their weights may be obtained using a gradient approach, as is applied in the gradient boosting method (gbm) [21].

Extreme gradient boosting (xgboost [22]) is another gradient boosting algorithm, which employs some additional tricks to estimate the parameters (i.e., trees and their weights). The optimization is carried out globally at the level of each split in the different trees and, as in rf, splits are optimized over a randomly sampled subset of covariates. The loss function optimized in the algorithm uses regularization over the weights and, optionally, a shrinkage of each tree added to the ensemble.

3.4. Support Vector Machines

Support vector machines [23] seek a linear separation of observations, like linear regression, but minimize a loss function based on the margins. In binary classification, the optimal regression is the hyper-plane, which perfectly separates the two classes and stays the farthest distance possible from its nearest points from the two classes. This hyper-plane may be easily found if the classes are linearly separable. If they are not, then the dataset is projected using nonlinear transformations in a much higher space where linear separation may be guaranteed. The linear transformations that are used in this case are expressed using kernel functions, among which the most commonly used are polynomial and radial kernels. These kernels, together with the optimization algorithm used to find the optimal hyper-plane, depend in general on several hyperparameters that must be fixed or tuned by the user. We will denote svmPoly (resp. svmRadial) as the support vector regression using polynomial (resp. radial) kernel.

Support Vector Machines may show high performance when the data are linearly separable, and are very efficient for large datasets. They also have a very solid mathematical justification.

3.5. Neural Networks

NNs, and specifically multi-layer perceptron (mlp) [17], are designed to handle multiple outputs. An mlp is composed of successive layers, each having a fixed number of neurons. Each neuron of a layer receives information (its inputs) from all of the neurons of the previous layers and outputs a nonlinear transformation $\varphi$ of a linear combination of its inputs. In Figure 2, the left-hand panel shows a simple neuron having d inputs $x_1,x_2,\dots ,x_d$. This neuron will output the value:

$$\varphi (\sum _{j=0}^d{w}_j{x}_j)$$

where the $w_j$ are the weights $w_0=b$, $x_0=1$, and $\varphi$ is a nonlinear function. The right-hand panel in Figure 2 shows an mlp with one hidden layer containing three neurons. Various types of layers may also be used to design a NN. We shall mainly use the fully connected layers (also called dense layers).

Once the structure of the network is specified (number of layers and number of neurons per layer), all of the weights of the NN that are unknown are estimated. To achieve this, a loss function should be specified (mean squared error for regression), together with an optimization algorithm (typically gradient-based algorithms). In some situations, specific activation layers are added to the networks, mainly for the last layer (softmax for multi-class learning, sigmoid for multi-label learning).

Neural networks are among the most difficult to tune because of the large number of hyperparameters that may be involved in the structure of the network and in the choice of the optimization algorithms. For small datasets, they may be unstable due to random initializations of the weights within the training.

3.6. Multidimensional Output Approaches

In many situations, the output variable to be predicted may be multidimensional in dimension q; multi-target regression, multi-label learning, distribution learning, semantic segmentation (for images), or time series. Various approaches may be used to tackle this situation. Direct multi-output regression is the simplest approach, which consists of using one model per output component. In this case, one assumes that the output components are independent. Chained regression is another approach that accounts for dependence among output variables. The output variables should first be ranked, and one model is learnt for each output, as follows: for output variable j, the original explanatory variables X are used as input, together with the predictions of all of the $j-1$ outputs already modelled. In this approach, the result depends strongly on the order used for the responses.

Some supervised learning algorithms also have the ability to jointly consider the q output variables within the same model, which is the case for the example of NN, and also regression trees [24].

In classical regression trees, the output variable y is used to compute the deviance, that is, the splitting criterion of a node t:

$$Deviance\left(t\right)=\sum _{i\in t}{(y_i-{\overline{y}}_t)}^2$$

where ${\overline{y}}_t$ is the empirical mean of y in node t (which corresponds to a subset of the original sample). When y is multidimensional, $y\in R^q$, the deviance is easily generalized [24]:

$$Deviance\left(t\right)=\sum _{i\in t}{\parallel y_i-{\overline{y}}_t\parallel }_2^2$$

where ${\parallel .\parallel }_2$ stands for the Euclidean norm in $R^q$. In this case, the values assigned to each node and each leaf of the tree are the q dimensional vectors ${\overline{y}}_t$. The rest of the algorithm is similar to the one-dimensional output. We will denote this approach as $CARTmv$.

The main advantage of direct multiple-output approaches is that they may account for dependencies between the different outputs. $CARTmv$ and its variants have been widely used in various applications.

3.7. Hyperparameters and Their Tuning

The performance of any ML algorithm highly depends on the choice of its hyperparameters. Each approach uses several hyperparameters, from one to almost ten. In most cases, these parameters have to be optimized, which is often achieved using cross-validation. Tuning may require excessive computation times, depending on the number of hyperparameters to be adjusted. Details about the tuning process and the hyperparameters used will be given in the experiments section.

3.8. Metrics Used to Compare the Models

Analysis of variance is generally used to evaluate the performance of polynomial models in RSM. Depending on the problem under study, it can be observed that the variation in outputs may not be due to the variation in inputs. Different criteria may be used to assess and compare the accuracy of the prediction obtained using any regression model. Let n be the size of the sample used to evaluate the models, $y_i$ be the observed value of the response for observation i, ${\overline{y}}_i$ be the average of responses in the sample, and ${\widehat{y}}_i$ be the predicted value of the response for observation i. Among the most used metrics, we can find:

• Root Mean Square Error, $RMSE=\sqrt{\frac{{\sum }_{i=1}^n{(y_i-{\widehat{y}}_i)}^2}{n}}$
• Mean Absolute Error, $MAE=\frac{1}{n}{\sum }_{i=1}^n|y_i-{\widehat{y}}_i|$
• Mean Absolute Percentage Error, $MAPE=\frac{1}{n}{\sum }_{i=1}^n\left|\frac{y_i-{\widehat{y}}_i}{y_i}\right|$
• Determination Coefficient, $R^2=1-\frac{{\sum }_{i=1}^n{(y_i-{\widehat{y}}_i)}^2}{{\sum }_{i=1}^n{(y_i-\overline{y})}^2}$
• Akaike Information Criterion, $AIC=2k-2ln\left(L\right)$, where k is the number of parameters to be estimated in the model and L is the maximum likelihood function of the model

Other, less well-known or used metrics also appear in the literature:

• The explained variance by the model is the proportion of the variance due to the factors, $EV=1-\frac{var(y-\widehat{y})}{var\left(y\right)}$
• The Nash and Sutcliffe Efficiency is equivalent to the $R^2$ but uses absolute differences rather than quadratic, $NSE=1-\frac{{\sum }_{i=1}^n|y_i-{\widehat{y}}_i|}{{\sum }_{i=1}^n|y_i-\overline{y}|},-\infty <NSE\le 1$
• The agreement index d is a standardized measure of the degree of model prediction error, $d=1-\frac{{\sum }_{i=1}^n|y_i-{\widehat{y}}_i|}{{\sum }_{i=1}^n\left(|y_i-\overline{y}|+|{\widehat{y}}_i-\overline{y}|\right)},0<d\le 1$
• The average absolute deviations from a central point (this metric is defined and used in [8], but the name given by the authors does not seem appropriate for us as it is not in line with their definition), $AAD(\%)=100\times \frac{1}{n}{\sum }_{i=1}^n\frac{{\widehat{y}}_i-y_i}{y_i}$

The objective is to minimize $RMSE$, $MAE$, $MAPE$, $AIC$, and $AAD$, and to maximize $R^2$, $EV$, $NSE$, and d. The results encountered in the literature show that, in general, when different metrics are used for comparisons, they are very often concordant with respect to the objectives.

When the output y is multidimensional, any of these criteria may be used by computing its average over all the dimensions.

4. Comparisons between RSM and ML Approaches

A comparison of these response surface models with ML has been made, mainly in the field of mechanics and materials development.

ML models were compared to the traditional RSM polynomial approximation on a mechanical engineering case study [1]. The authors studied 63 continuous factors and 8 continuous responses combined to produce a one-dimensional output, with 56,512 observations. Polynomial models of degree 1 and 2, Least Absolute Shrinkage Selection Operator [25], Generalized Linear Model [26] and nonlinear ML models (Random Forest, Gradient Boosting Decision Tree [17], Multiple Layer Perceptrons and Support Vector Regression) were tested. Three evaluation criteria were used for the comparisons, as follows: Explained Variance (EV), Mean Absolute Percentage Error (MAPE) and Root Mean Square Error (RMSE). The results obtained show that all ML models outperform the polynomial models of the RSM approach. The authors also note that for larger training set sizes, nonlinear ML models were more accurate. A simulation of the data by the MC method is proposed to improve the estimation of the response surface function. Comparisons made in this work are based on a real dataset with large samples; the true underlying relation between the response and the factors is unknown. Meanwhile, eight original responses were combined to get a one-dimensional output.

In [2], ML models were used to predict coating thickness in a nonlinear electrostatic spray deposition process and compared with the conventional RSM model. Three continuous factors and one continuous response were considered with 30 observations. A polynomial model of degree 2 (RSM), NN (Back-Propagation algorithm [27]) and Support Vector Machine models were used. MAPE was calculated to evaluate the performance of these models. The results suggest that the SVM model gives the best prediction accuracy.

The objective of [3] was to predict the viscosity of nano-polymers used in an enhanced oil recovery method, using the response surface methodology and supervised machine learning approaches. Five continuous factors and one continuous response were available, with 57 observations. The authors applied a polynomial model of degree 2 and analysis of variance, which did not show inadequacy for this model. However, the covariance matrix analysis showed that there is no linear relationship between the factors. Therefore, the authors also used nonlinear ML models, namely ridge regression [28], lasso regression, Support Vector Machine, Decision Tree, random forest, Extra tree regression [29], Gradient Boosting Regression and eXtreme Gradient Boosting [22]. Akaike Information Criterion (AIC), Mean Absolute Error (MAE), RMSE and coefficient of determination were used as evaluation and comparison criteria. The results show that the ensemble models give more accurate results than the RSM. Furthermore, the eXtreme Gradient Boosting model was the best model for response prediction.

In another study, where the objective was to predict the microhardness of a synthesized electroless Ni-P-TiO${}_2$ coated aluminium composite, five continuous factors and one continuous response were considered with 36 observations [4]. The authors tested a polynomial model of degree 2 and four ML models (NN, SVM, rf and Extra Trees). Mean Square Error (MSE), MAE and determination coefficient were used as evaluation criteria to compare these models. The results show that the ET model presents the lowest MSE and MAE values. This model also had the greatest $R^2$.

Hybrid regression and ML models were tested to predict the ultimate condition of fiber-reinforced polymer-confined concrete [5]. The authors used an open database with eight continuous factors, two continuous responses and 765 observations. The authors compared some existing physical empirical models developed by other authors with the RSM and SVR models, together with a hybrid model combining SVR and RSM models. The following metrics were used to evaluate these models: RMSE, MAE, Nash and Sutcliffe Efficiency (NSE) and agreement index d. The authors show that the hybrid model presents a lower RMSE, lower MAE, higher NSE and higher d than other models.

Similar comparisons with ML approaches may also be found in other fields, such as pharmaceutical development. Ref. [6] worked on the effect of the core/shell technique on improving powder compactability. In this paper, two continuous factors, two binary factors and two continuous responses were measured in 28 experiments. The authors used the RSM approach, adjusting one model for each combination of the two binary factors levels, thus, four models for each response. For the ML approach, the authors compared an SVR model and four types of NN models: Backpropagation Neural Network (BPNN), Genetic Algorithm Based BPNN (GA-BPNN) [30], Mind Evolutionary Algorithm Based BPNN (MEA-BPNN) [31]), and Extreme Learning Machine (ELM) [32]. The evaluation criteria were the variance coefficient of the RMSE and RMSE for all models, and AIC for NN models only. The results show that NN models provide better prediction accuracy than other models.

The objective of the authors of [10] was to apply RSM and ML combined with data simulation to estimate metal recovery from freshwater sediments. In this work, three continuous factors were studied, six continuous responses were measured and 18 experiments were carried out. A polynomial model of degree 2 was estimated and used for data simulation. ML models, namely Lazy KStar [33] and rf algorithms, were also tested. A comparison of the models was made by means of RMSE and coefficient of determination. The results show that RSM models overestimate the observed responses by 19% compared to ML models.

Ref. [8] compared RSM and ML approaches (NN) to predict the efficient extraction of artemisinin (a precursor molecule of the most powerful antimalarial drugs on the market). In this work, three continuous factors were studied, one continuous response was measured and a central composite design was constructed with 20 experiments. For the RSM approach, ANOVA was calculated and shows that the variability of the response cannot be adequately predicted by the RSM model depending on the factors studied. This result can be explained by a complex relationship between variables. Multiple Layer Perceptrons (feed forward NN) were also tested. A comparison between these two models was performed by means of Absolute Average Deviation (AAD). The authors concluded that the NN model has better prediction accuracy.

Ref. [9] analysed NN models as an alternative modeling technique for datasets showing nonlinear relationships, using data from a tablet compression study. In this work, six continuous factors were studied, two continuous responses were measured, and 102 experiments were performed. The authors calculated a polynomial model of degree 2 with only important terms (p-value < 0.05) for the RSM approach and calculated a “generalized feed forward multiple layer perceptron network” for the ML approach. A comparison of the determination coefficients of these two models suggests that the NN model can more accurately predict the variation of the response.

A summary of these papers comparing RSM and ML approaches is given in

Table 1. Summary of RSM vs. ML studies—PM: Polynomial Model, LASSO: Least Absolute Shrinkage Selection Operator, GLM: Generalized Linear Model, rf: random forest, GBDT: Gradient Boosting Decision Tree, MLP: Multiple Layer Perceptrons, SVR: Support Vector Regression, BPNN: Back-Propagation Neural Network, SVM: Support Vector Machine, CART: Classification and Regression Tree, ET: Extra Tree regression, GBR: Gradient Boosting Regression, xgboost: eXtreme Gradient Boosting, NN: Neural Network, GA-BPNN: Genetic Algorithm-Back Propagation Neural Network, MEA-BPNN: Mind Evolutionary Algorithm Based BPNN, ELM: Extreme Learning Machine, EV: Explained Variance, MAPE: Mean Absolute Percentage Error, RMSE: Root Mean Square Error, AIC: Akaike Information Criterion, MAE: Mean Absolute Error, $R^2$: coefficient of determination, MSE: Mean Square Error, NSE: Nash and Sutcliffe Efficiency, d: agreement index, CV RMSE: Coefficient of Variance of the RMSE, AAD: Absolute Average Deviation.

References	# Factors	# Responses	n	Models	Criteria
[1]	63	1	56,512	PM (1) PM (2) Linear ML model- LASSO, GLM rf, GBDT MLP, SVR	EV MAPE RMSE
[2]	3	1	30	PM (2) BPNN, SVM	MAPE
[3]	5	1	57	PM (2) Ridge regression LASSO, SVM CART, rf ET, GBR xgboost	AIC MAE RMSE $R^2$
[4]	5	1	36	PM (2) NN, SVM rf, ET	MSE MAE $R^2$
[5]	8	2	765	PM (2), SVR RSM-SVR ${}^1$	RMSE MAE NSE d
[6]	2 (+2) ${}^2$	2	28	PM (2) BPNN GA-BPNN, MEA-BPNN ELM	CV RMSE RMSE AIC (for NN)
[10]	3	6	18	PM (2) Lazy KStar rf	RMSE, $R^2$
[8]	3	1	20	PM (2) MLP	AAD
[9]	6	2	102	PM (2) MLP	$R^2$

¹ hybrid model. ² binary factors.

.

5. Experiments

In this section, we compare all of the ML approaches described in Section 3 to the polynomial model over a simulated dataset and a real dataset. We describe the datasets that we used, and how the different models are tuned and compared. All of the experiments were conducted using R software together with the RStudio Team [34] and the package “caret” [35].

5.1. Simulated Model

The simulated model that we use is based on two independent continuous factors, $X_1$ and $X_2$, following a uniform distribution over $[-1,1]$. The three responses—denoted as $Y_1$, $Y_2$, and $Y_3$—are computed as follows:

$$\begin{array}{cc}{Y}_1=& 20-X_1-X_2+18X_1^2+X_2^2+5X_1{X}_2+ϵ\hfill \\ Y_2=& X_1^3\times exp(sin\left(X_2\right))+ϵ\hfill \\ Y_3=& |X_1-0.5|+|X_2-0.5|+ϵ\hfill \end{array}$$

where $ϵ$ is a random noise with zero mean and standard deviation $\sigma$. The responses were chosen deliberately to present various types of function, as follows: polynomial of degree 2 ($Y_1$), nonlinear ($Y_2$), and nonlinear with discontinuities ($Y_3$). A graphical representation of these responses is shown in Figure 3.

Using this simulated model, we generated a training sample of size n and a test sample of the same size. Different values of n were tested, as follows: 15, 20, 30 and 50 observations. Small samples were used for conformity with true DoE applications. For each simulated dataset of size n, we computed the RMSE metric. Simulations were repeated $K=50$ times, and RMSE values were averaged over the different runs. We also varied the variance of the random noise, testing two values, $\sigma =1$ and $\sigma =2$.

5.1.1. Tuning the Models

We compared the RSM approach with all of the ML approaches described in Section 3: knn, CART, rf, gbm, xgboost, svmPoly, svmRadial, NN, CARTmv and NNmv. The models were trained using the model matrix as input, thus, the true factors together with their squares and interactions.

All ML models depend on several hyperparameters. To optimize the values of the hyperparameters, we applied the grid search approach. Thus, a grid was defined for each model’s hyperparameter, and three-fold cross-validation with five repeats was used for the optimization. The optimal values of the hyperparameters were then used to learn the model over the training set and evaluate it on the test sample.

Table 2. Hyperparameter values tested for each method. knn: k-nearest neighbors, rf: random forest, svmPoly: support vector machine with polynomial kernel, svmRadial: support vector machine with radial basis function kernel, CART: Classification And Regression Tree, gbm: gradient boosting machine, xgboost: eXtreme Gradient Boosting, NN: Neural Network, CARTmv: Classification and Regression Tree with multivariate responses, NNmv: Neural Network with multivariate responses.

Method	Hyperparameter Grid	Description
knn	k: {1, …, 10}	number of neighbors
CART	minsplit: {1, 5, 10} cp: {0.002, 0.005, 0.01, 0.015, 0.02, 0.03} minbucket: {1, 5, 10}	minimum number of observations to attempt a split complexity parameter minimum number of observations in a leaf
rf	minsize: {2, 5, 10} mtry: {1, …, 5}	minimum size of a node number of variables to test for each split
gbm	n.trees: {50, 100, 150, 200, 250} interaction.depth: {1, 2, 3, 4, 5} shrinkage: 0.1 n.minobsinnode: 10	total number of trees to fit maximum depth of trees learning rate minimum number of observations in leaves
xgboost	nrounds: {50, 100, 150, 200, 250} max_depth: {1, 2, 3, 4, 5} eta: {0.3,0.4} gamma: 0 colsample_bytree: {0.6, 0.8} min_child_weight: 1 subsample: [0.5, 0.625, 0.750, 0.875, 1}	maximum number of boosting iterations maximum depth of a tree control the learning rate minimum loss reduction subsample ratio of columns when constructing each tree minimum sum of instance weight needed in a child subsample ratio of the training instance
svmPoly	degree: {0.25, 0.5, 1, 2, 4} scale: {0.001, 0.01, 0.1, 1, 10} $C_{poly}$: {1, 2, 3}	polynome degree polynomial scaling factor control parameter
svmRadial	sigma: constant value $C_{rad}$: {0.25, 0.5, 1, 2, 4, 8, 16, 32}	bandwidth of kernel function control parameter
NN	layer1: {2, 4, 6, 8, 10} layer2: {2, 4, 6, 8, 10} layer3: {2, 4, 6, 8, 10}	number of hidden neurons in each layer
CARTmv	minsplit: {1,5,10} cp: {0.002, 0.005, 0.01, 0.015, 0.02, 0.03} minbucket: {1, 5, 10}	minimum number of observations to attempt a split complexity parameter minimum number of observations in a leaf
NNmv	layer1: {2, 4, 6, 8, 10} layer2: {2, 4, 6, 8, 10} layer3: {2, 4, 6, 8, 10}	number of hidden neurons in each layer

lists the hyperparameters for each method, together with a short description and the grid used for its optimization.

The average of the optimal values of the hyperparameters and their standard deviation for each approach and each response are given in Appendix A.1.

5.1.2. Results

Table 3. RMSE and rank for each method and each response (n = 50). Gray cells highlight the best model.

Method	${\mathit{S}}_1$	${\mathit{S}}_2$	${\mathit{S}}_3$	${\mathit{S}}_{\mathit{ave}}$	${\mathit{R}}_1$	${\mathit{R}}_2$	${\mathit{R}}_3$	${\mathit{R}}_{\mathit{ave}}$
lm	$1.067\pm 0.118$	$1.070\pm 0.136$	$1.116\pm 0.116$	$1.072$	$2$	$5$	$5$	$2$
knn	$1.629\pm 0.190$	$1.063\pm 0.112$	$1.093\pm 0.108$	$1.262$	$7$	$3$	$7$	$7$
CART	$1.715\pm 0.246$	$1.155\pm 0.154$	$1.192\pm 0.130$	$1.354$	$9$	10	11	$9$
rf	$1.338\pm 0.177$	$1.092\pm 0.138$	$1.123\pm 0.126$	$1.184$	$5$	$7$	$8$	$5$
gbm	$1.296\pm 0.161$	$1.067\pm 0.133$	$1.085\pm 0.132$	$1.149$	$4$	$4$	$6$	$4$
xgb	$1.390\pm 0.198$	$1.121\pm 0.147$	$1.150\pm 0.141$	$1.220$	$6$	$8$	$9$	$6$
svmpoly	$1.085\pm 0.120$	$1.054\pm 0.127$	$1.056\pm 0.129$	$1.065$	$3$	$1$	$2$	$1$
svmrad	$1.706\pm 0.626$	$1.072\pm 0.121$	$1.066\pm 0.119$	$1.281$	$8$	$6$	$3$	$8$
NN	$6.102\pm 1.870$	$1.061\pm 0.127$	$1.047\pm 0.109$	$2.737$	10	$2$	$1$	10
CARTmv	$8.331\pm 0.676$	$1.324\pm 0.147$	$1.151\pm 0.154$	$3.602$	11	11	10	11
NNmv	$1.063\pm 0.117$	$1.122\pm 0.130$	$1.070\pm 0.112$	$1.085$	$1$	$9$	$4$	$3$

,

Table 4. RMSE and rank for each method and each response (n = 30). Gray cells highlight the best model.

Method	${\mathit{S}}_1$	${\mathit{S}}_2$	${\mathit{S}}_3$	${\mathit{S}}_{\mathit{ave}}$	${\mathit{R}}_1$	${\mathit{R}}_2$	${\mathit{R}}_3$	${\mathit{R}}_{\mathit{ave}}$
lm	$1.177\pm 0.184$	$1.138\pm 0.184$	$1.123\pm 0.189$	$1.146$	$2$	$8$	$8$	$3$
knn	$2.036\pm 0.527$	$1.088\pm 0.129$	$1.067\pm 0.149$	$1.397$	$8$	$2$	$4$	$7$
CART	$2.003\pm 0.451$	$1.167\pm 0.192$	$1.140\pm 0.166$	$1.437$	$7$	$9$	$9$	$9$
rf	$1.587\pm 0.385$	$1.121\pm 0.135$	$1.117\pm 0.152$	$1.275$	$5$	$7$	$7$	$5$
gbm	$1.550\pm 0.327$	$1.116\pm 0.140$	$1.100\pm 0.159$	$1.255$	$4$	$5$	$6$	$4$
xgb	$1.601\pm 0.380$	$1.216\pm 0.155$	$1.194\pm 0.193$	$1.337$	$6$	10	11	$6$
svmpoly	$1.199\pm 0.207$	$1.100\pm 0.189$	$1.055\pm 0.135$	$1.118$	$3$	$4$	$2$	$1$
svmrad	$2.057\pm 0.798$	$1.093\pm 0.152$	$1.059\pm 0.146$	$1.403$	$9$	$3$	$3$	$8$
NN	$6.248\pm 1.930$	$1.084\pm 0.144$	$1.038\pm 0.142$	$2.790$	10	$1$	$1$	10
CARTmv	$8.104\pm 0.895$	$1.335\pm 0.222$	$1.184\pm 0.203$	$3.541$	11	11	10	11
NNmv	$1.164\pm 0.172$	$1.117\pm 0.165$	$1.081\pm 0.152$	$1.121$	$1$	$6$	$5$	$2$

,

Table 5. RMSE and rank for each method and each response (n = 20). Gray cells highlight the best model.

Method	${\mathit{S}}_1$	${\mathit{S}}_2$	${\mathit{S}}_3$	${\mathit{S}}_{\mathit{ave}}$	${\mathit{R}}_1$	${\mathit{R}}_2$	${\mathit{R}}_3$	${\mathit{R}}_{\mathit{ave}}$
lm	$1.205\pm 0.260$	$1.203\pm 0.233$	$1.196\pm 0.212$	$1.201$	$2$	$9$	10	$3$
knn	$2.309\pm 0.549$	$1.110\pm 0.195$	$1.064\pm 0.174$	$1.494$	$9$	$1$	$4$	$9$
CART	$2.091\pm 0.589$	$1.179\pm 0.231$	$1.139\pm 0.210$	$1.470$	$7$	$8$	$8$	$8$
rf	$1.761\pm 0.485$	$1.147\pm 0.180$	$1.091\pm 0.177$	$1.333$	$5$	$6$	$6$	$4$
gbm	$1.915\pm 0.444$	$1.141\pm 0.196$	$1.099\pm 0.189$	$1.385$	$6$	$5$	$7$	$6$
xgb	$1.706\pm 0.388$	$1.258\pm 0.197$	$1.174\pm 0.183$	$1.379$	$4$	10	$9$	$5$
svmpoly	$1.236\pm 0.236$	$1.127\pm 0.192$	$1.048\pm 0.164$	$1.137$	$3$	$3$	$3$	$2$
svmrad	$2.207\pm 0.979$	$1.123\pm 0.184$	$1.045\pm 0.149$	$1.458$	$8$	$2$	$2$	$7$
NN	$5.397\pm 2.040$	$1.140\pm 0.191$	$1.014\pm 0.153$	$2.517$	10	$4$	$1$	10
CARTmv	$7.959\pm 1.100$	$1.404\pm 0.254$	$1.270\pm 0.252$	$3.545$	11	11	11	11
NNmv	$1.172\pm 0.241$	$1.154\pm 0.196$	$1.064\pm 0.177$	$1.130$	$1$	$7$	$5$	$1$

and

Table 6. RMSE and rank for each method and each response (n = 15). Gray cells highlight the best model.

Method	${\mathit{S}}_1$	${\mathit{S}}_2$	${\mathit{S}}_3$	${\mathit{S}}_{\mathit{ave}}$	${\mathit{R}}_1$	${\mathit{R}}_2$	${\mathit{R}}_3$	${\mathit{R}}_{\mathit{ave}}$
lm	$1.436\pm 0.491$	$1.371\pm 0.512$	$1.293\pm 0.314$	$1.367$	$3$	10	11	$3$
knn	$2.670\pm 0.818$	$1.094\pm 0.192$	$1.063\pm 0.199$	$1.611$	$8$	$1$	$3$	$8$
CART	$2.424\pm 0.693$	$1.220\pm 0.302$	$1.151\pm 0.193$	$1.598$	$6$	$8$	$8$	$6$
rf	$2.149\pm 0.730$	$1.133\pm 0.218$	$1.079\pm 0.170$	$1.454$	$5$	$4$	$5$	$4$
gbm	$2.697\pm 0.741$	$1.187\pm 0.287$	$1.100\pm 0.161$	$1.661$	$9$	$6$	$7$	$9$
xgb	$1.910\pm 0.530$	$1.289\pm 0.290$	$1.221\pm 0.209$	$1.473$	$4$	$9$	$9$	$5$
svmpoly	$1.401\pm 0.425$	$1.194\pm 0.426$	$1.045\pm 0.199$	$1.213$	$2$	$7$	$2$	$2$
svmrad	$2.576\pm 1.080$	$1.162\pm 0.274$	$1.087\pm 0.229$	$1.608$	$7$	$5$	$6$	$7$
NN	$5.624\pm 2.260$	$1.102\pm 0.174$	$1.007\pm 0.182$	$2.577$	10	$2$	$1$	10
CARTmv	$8.267\pm 1.370$	$1.437\pm 0.282$	$1.247\pm 0.292$	$3.650$	11	11	10	11
NNmv	$1.332\pm 0.455$	$1.109\pm 0.185$	$1.077\pm 0.193$	$1.173$	$1$	$3$	$4$	$1$

show the average and standard deviation of the RMSE scores computed over the test samples for each response and averaged over $K=50$ repetitions (columns, $S_1$, $S_2$ and $S_3$), and the average RMSE over the responses (column $S_{ave}$). We also reported the rank of each model with respect to its RMSE for each response ($R_1$, $R_2$ and $R_3$) and for the average over responses ($R_{ave}$). A model has rank 1 for a response if it has the smallest RMSE, and thus, is the best. Each table corresponds to a different sample size n. Boxplots of the RMSE over the $K=50$ runs for all the models and the outputs are shown in Appendix A.2.

For $n=50$ (Table 3), the best models for the three responses were multivariate NN (for $Y_1$), svm with polynomial kernel (for $Y_2$) and NN (for $Y_3$). The linear model had positions (2, 5, 5).

For $n=30$ (Table 4), the best models for the three responses were, respectively: multivariate NN (for $Y_1$) and NN (for $Y_2$ and $Y_3$). With respect to the average of all response errors, svm with polynomial kernel is the best model in this case. Linear regression is at position 2 for $Y_1$, but at position 8 for the other two responses. These first results seem to be in agreement with the state of the art: ML approaches may provide better prediction estimates for the responses.

We obtained similar results for smaller sample sizes, $n=20$ (Table 5), and $n=15$ (Table 6): multivariate NN (for $Y_1$), knn (for $Y_2$) and NN (for $Y_3$) were the best models. Linear regression gets the worst results as the sample size decreases, particularly for $Y_2$ and $Y_3$, which are nonlinear responses.

Globally, the RSM approach seems to be inefficient when compared to ML approaches, mainly with small sample sizes and for responses that are originally nonlinear in the factors. Even for large samples ($n=50$, compared to what is available in DoE), RSM is outperformed in our experiments for linear response. Finally, multivariate NNs show excellent performance in various situations, mainly for the linear response ($Y_1$) where they outperform the RSM for any sample size.

5.2. Use Case

We used the pharmaceutical application that is described in [16]. This article is based on the development of the high-performance liquid chromatography analytical method to quantify verapamil hydrochloride (which is a chemical molecule involved in headaches) and its impurities in tablets. In this use case, three continuous factors (buffer pH, ammonium hydroxide concentration in mobile phase and injection volume of test solutions) and five continuous responses (capacity factor for the first eluted peak denoted $Y_1$, resolution between two impurities denoted $Y_2$, signal-to-noise for an impurity denoted $Y_3$, resolution between two other impurities denoted $Y_4$, and retention time difference between oxidative impurity peak and the closest impurity denoted $Y_5$) were available. The authors used the RSM approach to determine optimal chromatographic conditions.

For the real case study, we conducted similar experiments as for the simulated dataset, replacing random generation with random splitting. Thus, for each of the 50 repetitions, the original dataset was randomly split using proportions $0.7$ and $0.3$ for learning and testing, respectively. The results that we obtained are given in

Table 7. RMSE and rank for each method and each response (DOE case study). Gray cells highlight the best model.

Method	${\mathit{S}}_1$	${\mathit{S}}_2$	${\mathit{S}}_3$	${\mathit{S}}_4$	${\mathit{S}}_5$	${\mathit{S}}_{\mathit{ave}}$	${\mathit{R}}_1$	${\mathit{R}}_2$	${\mathit{R}}_3$	${\mathit{R}}_4$	${\mathit{R}}_5$	${\mathit{R}}_{\mathit{ave}}$
lm	$0.274\pm 0.115$	$7.69\pm 3.060$	$13.35\pm 5.080$	$5.59\pm 2.740$	$0.843\pm 0.338$	$5.55$	$8$	11	11	10	11	11
knn	$0.268\pm 0.081$	$3.15\pm 0.832$	$8.33\pm 1.540$	$4.80\pm 2.000$	$0.310\pm 0.078$	$3.37$	$6$	$8$	$2$	$6$	$1$	$4$
CART	$0.292\pm 0.094$	$3.10\pm 0.903$	$9.85\pm 2.130$	$6.30\pm 1.940$	$0.378\pm 0.109$	$3.98$	$9$	$6$	10	11	$9$	10
rf	$0.258\pm 0.086$	$2.54\pm 0.767$	$9.38\pm 1.880$	$4.99\pm 1.640$	$0.350\pm 0.075$	$3.50$	$5$	$3$	$7$	$7$	$7$	$8$
gbm	$0.256\pm 0.078$	$2.17\pm 0.778$	$8.90\pm 2.500$	$5.49\pm 1.460$	$0.376\pm 0.099$	$3.44$	$4$	$1$	$4$	$8$	$8$	$5$
xgb	$0.198\pm 0.085$	$2.23\pm 0.903$	$9.20\pm 2.140$	$4.28\pm 1.730$	$0.343\pm 0.095$	$3.25$	$1$	$2$	$6$	$3$	$5$	$2$
svmpoly	$0.254\pm 0.087$	$3.36\pm 1.510$	$9.20\pm 1.720$	$4.33\pm 1.770$	$0.347\pm 0.129$	$3.50$	$3$	10	$5$	$4$	$6$	$7$
svmrad	$0.242\pm 0.077$	$2.80\pm 0.985$	$8.89\pm 1.550$	$4.40\pm 1.550$	$0.329\pm 0.093$	$3.33$	$2$	$4$	$3$	$5$	$3$	$3$
NN	$0.273\pm 0.072$	$3.18\pm 1.090$	$9.77\pm 2.430$	$3.70\pm 1.770$	$0.316\pm 0.060$	$3.45$	$7$	$9$	$9$	$1$	$2$	$6$
CARTmv	$0.311\pm 0.077$	$2.89\pm 1.090$	$9.54\pm 1.600$	$5.52\pm 1.310$	$0.339\pm 0.088$	$3.72$	10	$5$	$8$	$9$	$4$	$9$
NNmv	$0.491\pm 0.616$	$3.13\pm 1.280$	$6.98\pm 1.940$	$3.98\pm 1.660$	$0.614\pm 0.467$	$3.04$	11	$7$	$1$	$2$	10	$1$

. The best models with respect to the RMSE were xgb for $Y_1$, gbm for $Y_2$, multivariate NN for $Y_3$, NN for $Y_4$ and knn for $Y_5$. For the overall error, multivariate NN had the smallest error. For this real dataset, and for all responses, ML models gave better results than the RSM approach.

6. Conclusions

In this work, using extensive simulations and a real use case, we have demonstrated that ML approaches present a very interesting alternative to response surface modeling in the DoE. We have tested a large panel of ML approaches, together with direct multi-output regression models (decision trees and NNs), which had never previously been used in this context. Clearly, various ML approaches outperformed RSM in all our experiments. RSM is very simple to use compared to ML approaches, due to the fact that several hyperparameters must be tuned correctly in ML algorithms. This requires optimization procedures over grids. The caret package is a good wrapper for this task and for most of the approaches that we have used, except for multi-output regression, where we implemented the grid search algorithm directly. Another constraint of caret is that not all the hyperparameters involved in each approach may be tuned, and thus it may be necessary to use other packages for this issue.

In our implementation, we used the factors together with their squares and interactions in the ML approaches. Very few papers are clear about this choice. We also tested the models including only the factors, and in some cases a loss in performance appeared for some ML algorithms, but the main conclusion was observed: several ML approaches outperformed classical RSM.

Many ML and deep learning approaches were not explored in this work (like [36,37,38,39]) and may have some advantages and drawbacks compared to those we have used. Due to the very large choice of such approaches, it is impossible to be exhaustive, so we have selected the ones most used and cited in the literature. In addition, further higher-polynomial-order factors could be included as input in the different models. Together with feature selection approaches, this may give better performance for various ML approaches.