Numerical Non-Linear Modelling Algorithm Using Radial Kernels on Local Mesh Support

Abstract

Estimation problems are frequent in several fields such as engineering, economics, and physics, etc. Linear and non-linear regression are powerful techniques based on optimizing an error defined over a dataset. Although they have a strong theoretical background, the need of supposing an analytical expression sometimes makes them impractical. Consequently, a group of other approaches and methodologies are available, from neural networks to random forest, etc. This work presents a new methodology to increase the number of available numerical techniques and corresponds to a natural evolution of the previous algorithms for regression based on finite elements developed by the authors improving the computational behavior and allowing the study of problems with a greater number of points. It possesses an interesting characteristic: Its direct and clear geometrical meaning. The modelling problem is presented from the point of view of the statistical analysis of the data noise considered as a random field. The goodness of fit of the generated models has been tested and compared with some other methodologies validating the results with some experimental campaigns obtained from bibliography in the engineering field, showing good approximation. In addition, a small variation on the data estimation algorithm allows studying overfitting in a model, that it is a problematic fact when numerical methods are used to model experimental values.

1. Introduction

Obtaining a numerical or analytical representation of a given unknown relationship is a frequent problem in the study and modelling of systems, with application in a wide variety of fields ranging from experimental and social sciences to engineering. In addition, in most cases, the only information available is a set of experimental data from the variables that define the relationship. In other words, without loss of generality, given

$$y=f(x^1,\dots ,x^d)$$

(1)

with

$$\overrightarrow{x}=(x^1,\dots ,x^d)\in \mathsf{\Omega }\subset {ℝ}^d$$

(2)

usually only one set of experimental data is known for the variables involved in the relationship (1):

$${\left\{(x_{\left[k\right]}^1,x_{\left[k\right]}^2,x_{\left[k\right]}^{3 },\dots ..,x_{\left[k\right]}^d)\right\}}_{k=1,2,\dots .,p}$$

(3)

These values are usually affected by several factors that induce variations between the real and the observed quantities. That is, for any k:

$$y_{\left[k\right]}=f(x_{\left[k\right]}^1+{\epsilon }_{\left[k\right]}^1, x_{\left[k\right]}^2+{\epsilon }_{\left[k\right]}^2,\dots \dots ., x_{\left[k\right]}^d+{\epsilon }_{\left[k\right]}^d)+ {ϵ}_{\left[k\right]} = f(\overrightarrow{x_{\left[k\right]}}+\overrightarrow{{\epsilon }_{\left[k\right]}})+ {ϵ}_{\left[k\right]}.$$

(4)

If $\mathsf{\Omega }$ is a bounded domain of ${ℝ}^d$, and $f\in C^1$, developing the function $f(\overrightarrow{x})$ in $\overrightarrow{x_{\left[k\right]}}=(x_{\left[k\right]}^1,x_{\left[k\right]}^2,x_{\left[k\right]}^{3 },\dots ..,x_{\left[k\right]}^d)$, from (4):

$$y_{\left[k\right]}=f(x_{\left[k\right]}^1, x_{\left[k\right]}^2,\dots \dots ., x_{\left[k\right]}^d)+{\sum }_i\frac{\partial f}{\partial x^i}({\xi }_{\left[k\right]}^1, {\xi }_{\left[k\right]}^2,\dots \dots ., {\xi }_{\left[k\right]}^d)·{\epsilon }_{\left[k\right]}^{\mathrm{i}}+ {ϵ}_{\left[k\right]}$$

(5)

where ${\xi }_{\left[k\right]}=({\xi }_{\left[k\right]}^1, {\xi }_{\left[k\right]}^2,\dots \dots ., {\xi }_{\left[k\right]}^d)=\overrightarrow{x_{\left[k\right]}}+\lambda ·\overrightarrow{{\epsilon }_{\left[k\right]}}$, with $\mathsf{\lambda }\in \left[0,1\right]$.

Defining a new value $e_{\left[k\right]}$ that includes all perturbations:

$$e_{\left[k\right]}={\sum }_i\frac{\partial f}{\partial x^i}({\xi }_{\left[k\right]}^1, {\xi }_{\left[k\right]}^2,\dots \dots ., {\xi }_{\left[k\right]}^d)·{\epsilon }_{\left[k\right]}^{\mathrm{i}}+{ϵ}_{\left[k\right]}$$

(6)

Therefore,

$$y_{\left[k\right]}=f(x_{\left[k\right]}^1, x_{\left[k\right]}^2,\dots \dots ., x_{\left[k\right]}^d)+e_{\left[k\right]}$$

(7)

Expression (7) is frequently assumed implying that y is the only variable affected by the error, while the variables ${\left\{x^i\right\}}_{i=1}^d$ correspond to the exact values. It is also frequent to include some considerations on the distribution of probability that generates the values of the perturbation, especially the assumption of normality.

Sometimes the form of the relations is known as occurs in linear regression where the equation corresponding to the studied system is:

$$y=\alpha +{\lambda }_1 x^1+{\lambda }_2 x^2 +\cdots ..+{\lambda }_d{x}^d$$

(8)

The problem to determine the values of lambda parameters, that give the best fit between the linear expression and the data, is now reduced minimizing a previously defined error function.

In real cases, the effects of non-linearity dominate the dynamics of the system and far from the equilibrium points linear relations as in (8) are not useful. In the cases when the relation is not linear but known:

$$y=z(x^1,x^2,\dots .,x^d,\dots ..,{\lambda }_1,{\lambda }_2,\dots ..,{\lambda }_Q)$$

(9)

The problem consists in the calculation of the unknown parameters $({\lambda }_1,{\lambda }_2,\dots ..,{\lambda }_Q)$ using the data.

In (8) and (9), determining the values of the parameters of the model can be done by minimizing an error function defined over the parameters. One common method is to consider the mean square error (MSE) of the estimated values on the experimental points.

$$e({\lambda }_1,{\lambda }_2,\dots ..,{\lambda }_Q)={\sum }_{k=1}^p{(y_{\left[k\right]}-z(x_{\left[k\right]}^1,\dots .x_{\left[k\right]}^d,{\lambda }_1,{\lambda }_2,\dots ..,{\lambda }_Q))}^2$$

(10)

However, in real practice, the expression for the relationship is unknown and the problem is to obtain an analytical expression or a numerical approximation to the relation (1).

The finite element method (FEM) is a numerical method used initially for solving structural problems in civil and aeronautical engineering that is applied in a wide variety of fields, see [1,2]. This method allows finding approximate solutions to differential equations and boundary problems in partial differential equations.

Given a differential equation defined through a differential operator, $D$:

$$D(f)=v$$

(11)

where $f,v \in V$ with $V$ a function space the finite elements method replaces $V$ by a finite dimensional subspace $V_h\subset V$ whose elements are continuous piecewise polynomial functions of degree K (this is a Sobolev Space). In addition, the domain of the problem, Ω is divided in $N_e$ parts called elements ${\mathsf{\Omega }}_i$.

$$\mathsf{\Omega }={\cup }_{i=1}^{N_e}{\mathsf{\Omega }}_i$$

(12)

The original Equation (11) changes now to:

$$D(f_h)=v_h$$

(13)

where $f_{h, }{v}_h ϵ V_h$.

If $\dim V_h=Q$, taking a basis of $V_h$:

$$V_h=\langle {\phi }_1(x),{\phi }_2(x),\dots \dots .,{\phi }_Q(x)\rangle$$

(14)

the solution can be approximated as:

$$f_h(x)={\sum }_{i=1}^Q{u}_i·{\phi }_i(x)$$

(15)

The values of $u_i$ must be determined to get a good approximation for the solution to the differential equation.

The basis (14) is selected considering the behaviour of their functions in a set of Q points called nodes $({\zeta }_1, {\zeta }_2,\dots ..,{\zeta }_Q)$ that satisfy the following conditions:

$${\phi }_i({\zeta }_j)={\delta }_{ij}$$

(16)

In addition, an important property is that ${\phi }_i(x)=0$ if the node ${\zeta }_i$ does not form part of the element containing the point x.

The functions of the basis are named shape functions and they are used to interpolate in points different from the nodes.

The selection of the nodes is a crucial point for the precision of the results. The operation that construct the set of nodes and elements is called meshing and is determined by the set of elements and nodes $({\left\{{\zeta }_r\right\}}_{r=1}^Q,{\left\{{\mathsf{\Omega }}_i\right\}}_{i=1}^{N_e})$. The resulting mesh can be locally characterized by a parameter related with the size of the corresponding local element and the error of the approximated solution in the element.

The authors in previous papers, using different methodologies, have studied regression methods based on the FEM. In [3], the regression algorithm is based on a least squares approach comparing experimental data and the model generated using FEM. In [4], the model is built from a physical analogy that solves some problems on the linear system. One approach more on the line of the original FEM can be seen in [5] and applied by an example in [6,7]. Although different modifications of the initial algorithm have been developed, see [8] in order to improve the computing speeds for the regression model, the basic methodology can be resumed as follows.

The numerical regression model for the relation consists of the set formed by the nodal values $\left\{u_i\right\}$, which is called a representation model for the system. Then, the value of the relationship in any point can be estimated using the expression:

$$f_h(x)={\sum }_{i=1}^Q{u}_i {\phi }_i(x)$$

(17)

By (16), the sum is restricted to the nodes that determine the element which contains point x. For simplicity in the subsequent calculations the authors have normalised the data in the domain, ${\left[0,1\right]}^d$, and used a mesh formed by regular hyper-cubic elements with edge length $h$. Introducing the complexity c as the number of elements in each dimension, $h=1/c$, the total number of elements is $c^d$ and the total number of nodes is ${(c+1)}^d$.

The key point in FEM-based methods is the determination of the values $\left\{u_i\right\}$ considering the definition of an error function and its minimization. That problem is equivalent to solve a linear system of size, ${(c+1)}^d$. The results and precision have been checked with good performance for low dimensions, but there is a practical problem when the dimension of the problem becomes greater because of the order of the linear system solution’s algorithms.

The problem with this approach is the conversion of a local problem, which is the determination of the function value in a point $x_o$, into a global question solving a system to obtain the ${(c+1)}^d$ nodal values.

As a conclusion, the direct geometric interpretation of FEM regression methods makes them very interesting compared to other approaches. The techniques related with finite elements are equivalent to the determination of the tangent subspace of the manifold defined by $y=f(x)$ at the objective point $x_0$. The natural evolution of the methods previously considered in the introduction is the restriction of the calculations to the boundary of $x_0$, determined by the underlying FEM methodology. The only requirement is the determination of estimations on nodal points. These estimations can be as simple as the k-nearest neighbor (k-NN). However, k-NN presents the problem of determining the optimal value of k, to include as much representative values as possible, minimizing the influence of outliers. This can be solved by introducing the distance-based nearest neighbor.

To do this, the proposed algorithm is based on radial basis functions (and radial basis function networks) and the results on random fields, given that the regression problem on experimental data is equivalent to removing the influence of this stochastic term over the samples generated from the relationship under study.

After the introduction in Section 1, the paper is organised as follows. Section 2.1 considers the main characteristics and use of radial basis functions in problems of interpolation and regression, while Section 2.2 presents an introduction and some results on random fields. In Section 3, the method proposed by the authors is developed and applied to different problems. Section 4 corresponds to the conclusions and analysis of future investigations.

2. Materials and Methods

2.1. Radial Basis Functions

Radial functions are functions, which depend only on the distance between their arguments:

$$\theta (x,y)=\mathsf{\Phi }(\Vert x-y\Vert )$$

(18)

These kinds of functions can be used as a basis of functional space (radial basis function), see [9], introducing an interpolation method for any function $f$ on a set of pairs composed by points called centres and the values of the function on that points, ${\left\{(c_i,f_i)\right\}}_{i=1}^P$ given by:

$$\widehat{f}(x)={\sum }_{i=1}^P{\alpha }_i\mathsf{\Phi }(\Vert x-c_i\Vert )$$

(19)

The values of the alpha-coefficients are calculated imposing P conditions given by:

$$\widehat{f}(c_i)=f_i$$

(20)

This expression can be used to approximate the solution of a differential equation, using by example a Galerking approach [10]. The determination of centres is a central part of the problem to consider, because they are not fixed points as in other methods as finite elements, finite boundaries, etc., so these techniques receive the name of meshless methods. Several works have studied these approaches for different fields (fluid dynamics in [11]) or solving problems related with the corresponding systems as in [12,13].

Another frequent use of radial functions is in the problem of functional where an unknown function f(x) must be approximated from a set of pairs ${\left\{(x_k,y_k)\right\}}_{k=1}^P$ provided by discrete measurements, where x corresponds to a vector and y are values related to the value of the function in the x-points. In addition, usually there exists errors between y and the value of $f(x)$, that is $y_k=f(x_k)+e_k$.

The direct use of radial basics function (RBF) as an interpolation method is not adequate here because of the need of exact interpolation on the available points. For solving this issue, some neural networks models are developed [14] with the addition of radial functions [15,16] because they are faster compared to the classical sigmoid neurons. This better behaviour is due to the improvements of the algorithm [17] or the determinations of radial centres [18,19] because it is capable of obtaining arbitrarily good approximations to the functions [20,21]. An additional advantage of these radial basis function networks (RBFN) is the more transparent interpretation of the results compared to the case of multilayer perceptron (MLP) with the non-linearities associated with the sigmoidal functions, [19,20].

Compared with other techniques there are different assets depending on the problem, from very good results [22] to other studies affirming its inferior behaviour [23]. Other authors have also cited the need of a greater number of points to obtain similar accuracies to sigmoid networks [20].

RBFN and variations on them have been used widely to study time series [24]; signal processing [25]; data fitting [26]; system modelling [16,27,28]; system control [29]; system identification [30]; faults analysis [31]; computer vision [32,33,34], and speak recognition [35].

In a neural network with n neurons, the output is:

$$f_o(x)={\sum }_{i=1}^n{f}_i R_i(x)$$

(21)

Following [24], a necessary and sufficient condition for a function to act as an activation function in a RBFN is not to be an even polynomial. A typical but not the only, see [36], selection for these radial functions is a set of Gaussian functions given by:

$$R_i(x)=e^{-{\left(\frac{\Vert x-c_i\Vert }{{\sigma }_i}\right)}^2}$$

(22)

Figure 1 represents a diagram of this kind of network.

The use of these kinds of functions was initially applied in the study of some structures of the cerebral cortex [37].

The output is obtained from the parameters ${\left\{c_i, {\sigma }_i,f_i\right\}}_{i=1}^n$ and several algorithms have been developed to obtain an optimum selection of those parameters (learning process). Some of them use a preliminary algorithm to determine the centres $c_i$ (genetic algorithms [38,39] or classification trees [40] and the widths ${\sigma }_i$ of each element of the network [19,31]. However, widths are not usually considered as important as the rest of the parameters, so some authors have focused on the development of algorithms to optimize them [41]. Some more recent papers have used a variety of algorithms to improve the RBFN learning capabilities using hybrid approaches [42], including techniques to reduce the dimensionality as the principal component analysis [43]. Between the modifications of the basic version of RBFN the most developed are the addition of a linear term to improve the behaviour on the regions that are out of the cloud of available points [26]. In addition, the use of a normalising operation defines the normalised radial function neural networks (NRFNN) [44,45], where the network output has the expression:

$$f_o(x)=\frac{{\sum }_{i=1}^n{f}_i·R_i(x)}{{\sum }_{i=1}^n{R}_i(x)}$$

(23)

NRBFN present some good properties as the partition of unity, covering the whole input space and greater stability with respect to the selection of the network parameters [46].

The use of NRBFN has increased in the last times involving problems as electricity price forecasting [47], non-stationary time series [48], system control [49], system modelling [46], magnetic fields [50], and integral equations [51].

For a general study of the families of models related to (23), papers [52,53] can be of interest.

2.2. Random Fields

Given a domain $\mathsf{\Omega }$, a random field defined on it is a collection of random variables defined on each point of $\mathsf{\Omega }$.

Random fields are widely used in problems where local correlations exist on data perturbations as speech recognition, computer vision [54,55], statistical analysis of MEG images [56], and magnetic resonances imaging [57]. Some of these treatments are related with the problem of the determination of a threshold value, which has been generalized, as the determination of the frontier in a binary classification from a set of known points [58]. Additional applications are referenced in [59], etc., including particle physics, agriculture, economics, etc.

An interesting further step is the use of random fields and neural networks as considered in [60].

An important part of the investigations is devoted to specific types of random fields: Gaussian random fields, see [61], Markovian random fields, and conditional random fields used in structured learning problems where relationships exist between the regressed variables in different spatial-temporal points, by example, in the case of temporal data [62].

As pointed in [63], some of the considered problems can be seen as an incomplete data problem where part of the relevant data is hidden and must be estimated. Frequently, this estimation is made using maximum-likelihood related algorithms or using the mean field theory approach.

Following [64], let us consider a set of labelled points $L\times ℝ=\left\{(x_1^1,\dots ,x_1^d,y_1),\dots ,(x_l^1,\dots ,x_l^d,y_l)\right\}$ and a set of unlabeled points $U=\left\{(x_1^1,\dots ,x_1^d),\dots ,(x_u^1,\dots ,x_u^d)\right\}$. The considered problem is to determine a real function defined on the set $V=\left\{(x_1^1,\dots ,x_1^d),\dots ,(x_l^1,\dots ,x_l^d),(x_{l+1}^1,\dots ,x_{l+1}^d),\dots ,(x_u^1,\dots ,x_u^d)\right\}$, $f:V\to ℝ$ that will be used to assign labels to the points in U, under the condition $f(x_i)=y_i$ for $x_i\in L$. The introduction of a weight function

$$w_{ij}=exp\left(-{\sum }_{k=1}^d\frac{{(x_i^k-x_j^k)}^2}{{\sigma }_k^2}\right)$$

(24)

to preserve ‘local smoothness’ and the definition of an energy function

$$E(f)=\frac{1}{2}{\sum }_{i,j}{w}_{ij}·{\left(f(x_i)-f(x_j)\right)}^2$$

(25)

gives as a result the harmonic property for the minimum energy function calculated from E(f), that is, the value of f at any unlabeled point can be written as an average of the rest of the points

$$f(x_j)=\frac{1}{{\delta }_j}·{\sum }_{i\ne j}{w}_{ij}·f(x_i)$$

(26)

In [65], the problem of calculating the regression of a random field from a set of observations is considered. Under the conditions of a finite domain and distribution of the points over that domain, the authors suggest considering the problem as a continuous case.

Other studies of local regression of random fields are considered in [66], where given a realization of a random field $\left\{(x_i,y_i):i\in {\mathbb{Z}}^N\right\}$ the problem is the determination of the spatial regression function given by $f:x⟼f(x):=E\left[y_i|x_i=x\right]$. An approach to the nonparametric regression can be seen in [67,68], where the regression problem is determining the function f defined on a lattice domain and given by the expression $y_i=f(x_i)+e_i$. The estimation expression for the function f on the domain ${\left[0,1\right]}^d$ is related with the expression:

$$f_N(x)=\frac{{\sum }_i{y}_i·K\left(\frac{x-x_i}{h_N}\right)}{{\sum }_{i}K\left(\frac{x-x_i}{h_N}\right)}$$

(27)

where K is a kernel function and N is a parameter that determines the number of points in the lattice and the value of the parameter $h_N$.

Under some general conditions, the difference between the expected value of $f_N(x)$ and the value of the function, $\left|E\left[f_N(x)\right]-f(x)\right|$ is bounded by the value of $h_N$. Moreover, when $h_N$ is fixed as a determined expression of N, the upper bound for the difference $\left|f_N(x)-f(x)\right|$ is related to ${(\ln N)}^{\alpha }/N^d$. These results are very interesting, but they are obtained using a dataset defined on a lattice.

3. Results

3.1. Radial Kernel Weighted Average

Definition 1.

A parameterized radial function is a function $\mathsf{\Phi }:{ℝ}_{+}\times {ℝ}_{+}\to {ℝ}_{+}$ characterized by a parameter $\omega$ that accomplishes the conditions:

$$\begin{array}{c}\forall \omega \ne 0 \underset{r\to \infty }{\lim }\mathsf{\Phi }(r,\omega )=0\\ \forall r\ne 0 \underset{\omega \to 0}{\lim }\mathsf{\Phi }(r,\omega )=0\end{array}\}$$

(28)

Following the definition, an example of parameterized radial function is the Gaussian:

$$\mathsf{\Phi }(r,\omega )=e^{-{(\frac{r}{\omega })}^2}$$

(29)

Definition 2.

Given a function $f(x)$ defined on a domain, $\mathsf{\Omega }$, the $\mathsf{\Phi }$-weighted average of $f(x)$ in a point $x_o$, $c_{*}(x_o)$, is a number implicitly defined as:

$${\int }_{\mathsf{\Omega }}\left[c_{*}(x_o)-f(x)\right] \mathsf{\Phi }(\Vert x-x_o\Vert ,\omega ) dx=0$$

(30)

where $\mathsf{\Phi }(.,.)$ is a parameterized radial function.

Definition 3.

The J-th moment of the radial function $\mathsf{\Phi }$ is defined as:

$$W^J(x_0)={\int }_{\mathsf{\Omega }}\Vert x-x_0\Vert {}^J ·\mathsf{\Phi }(\Vert x-x_0\Vert ,\omega ) dx$$

(31)

Using this expression, Equation (30) can be written as:

$$c_{*}(x_o)·W^0(x_0)={\int }_{\mathsf{\Omega }}f(x)· \mathsf{\Phi }(\Vert x-x_0\Vert ,\omega ) dx$$

(32)

Definition 4.

Given a domain $\mathsf{\Omega }$, the J-th core of width $\chi$ (represented as ${\mathrm{K}}_{\mathsf{\Omega }}^J(\omega ,\chi )$), is the subdomain ${\mathrm{K}}_{\mathsf{\Omega }}^{\mathrm{J}}(\omega ,\chi )\subset \mathsf{\Omega }$ where $\forall x_0\in {\mathrm{K}}_{\mathsf{\Omega }}^J(\omega ,\chi )$, the function $W^J(x_0)$ satisfies:

$$0\le W^J-W^J(x_0)\le \chi$$

(33)

where $W^J={\int }_{{ℝ}^d} \Vert x{\Vert }^J\cdot \mathsf{\Phi }(\Vert x\Vert ,\omega ) dx$.

Definition 5.

In the same conditions of Definition 4, the axial core of width $\chi$ (represented as $A{K}_{\mathsf{\Omega }}(\omega ,\chi )$), is the subdomain $A{K}_{\mathsf{\Omega }}(\omega ,\chi )\subset \mathsf{\Omega }$ where $\forall x_0\in A{K}_{\mathsf{\Omega }}(\omega ,\chi )$ and for every $i=\left\{1,2,..,d \right\}$:

$$|{\int }_{\mathsf{\Omega }}{(x-x_0)}^i ·\mathsf{\Phi }(\Vert x-x_0\Vert ,\omega ) dx |\le \chi$$

(34)

For practical effects, Expression (33) implies considering $W^J(x)$ as the constant value $W^J$ for points belonging to ${\mathrm{K}}_{\mathsf{\Omega }}^J(\omega ,\chi )$ within the precision given by $\chi$. Whereas, (34) is equivalent to using a zero value for the integral inside $A{K}_{\mathsf{\Omega }}(\omega ,\chi )$ with the same consideration with respect to the precision, properties that will be used to obtain an upper bound for the error of the estimation calculated with the algorithm on points belonging to ${\mathrm{K}}_{\mathsf{\Omega }}^0(\omega ,\chi ) \cap A{K}_{\mathsf{\Omega }}(\omega ,\chi )$.

3.2. Interpolation over a Mesh of Finite Elements

Let the decomposition of the domain in $N_e$ elements $\mathsf{\Omega }={\cup }_{i=1}^{N_e}{\mathsf{\Omega }}_i$ and $Q$ nodes ${\left\{{\zeta }_r\right\}}_{r=1}^Q$ with the associated shape functions ${\left\{N_r(x)\right\}}_{r=1}^Q$. Then, the following relations are true:

$$\begin{array}{c}\sum _{r=1}^Q{N}_r(x) =1\\ N_r({\zeta }_S)={\delta }_{rS}\end{array}\}$$

(35)

Given a point $x_oϵ {\mathsf{\Omega }}_l\subset \mathsf{\Omega }$ with nodes ${\left\{{\zeta }_r\right\}}_{r=1}^Q$ and a function $f(x)$, the interpolated value of the function in the point $\widehat{f}(x_o)$ is defined as:

$$\widehat{f}(x_o)={\sum }_{r=1}^Q{N}_r(x_o) ·f({\zeta }_r)$$

(36)

Definition 6.

Given a function $f(x)$ defined on $\mathsf{\Omega }$, and a point $x_oϵ {\mathsf{\Omega }}_l$ the regression estimation of $f(x)$ on $x_o$ associated to the radial kernel $\Phi$ and the mesh $({\left\{{\zeta }_r\right\}}_{r=1}^Q,{\left\{{\mathsf{\Omega }}_i\right\}}_{i=1}^{N_e})$ is defined as the interpolation of the weighted averages of f on the nodes of ${\mathsf{\Omega }}_l$:

$$\widehat{z}(x_o)={\sum }_{r=1}^Q{N}_r(x_o)\cdot c_{*}({\zeta }_r)$$

(37)

This expression is closely related with the approaches presented in (23)–(25) and (27).

Definition 7.

Given a function $f(x)$ defined on a domain $\Omega$ and a random field $e(x)~N(0,\sigma (x))$ defined on $\mathsf{\Omega }$, an experimental realisation of f associated with a sample $e^S(x)$ of $e(x)$ is a function $y^s:\mathsf{\Omega }\to ℝ$, given by:

$$y^s(x)=f(x)+e^S(x)$$

(38)

Proposition 1.

The expected value (denoted as $E[\dots ]$) of the weighted average corresponding to the values $y^s$ in any point $x_o$ is:

$$E\left[c_{*}^s(x_o)\right]=c_{*}(x_o)$$

(39)

Proof. 

Following Expression (30), the regression of the experimental realisation on each node is:

$${\int }_{\mathsf{\Omega }}\left[c_{*}^s(x_o)-y^s(x)\right]. \mathsf{\Phi }(\Vert x-x_o\Vert ,\omega ) dx=0$$

(40)

Using (38), the last expression can be written as:

$${\int }_{\mathsf{\Omega }}\left[c_{*}^s(x_o)-f(x)\right]. \mathsf{\Phi }(\Vert x-x_o\Vert ,\omega ) dx= {\int }_{\mathsf{\Omega }}{e}^s(x). \mathsf{\Phi }(\Vert x-x_o\Vert ,\omega ) dx$$

(41)

Therefore, by (30) and using the definition (31) of $W^0(x_o)$:

$$(c_{*}^s(x_o)-c_{*}(x_o))·W^0(x_o)= {\int }_{\mathsf{\Omega }}{e}^s(x)·\mathsf{\Phi }(\Vert x-x_o\Vert ,\omega ) dx$$

(42)

Under the conditions of Definition 7, $E\left[e^S(x)\right]=0$, and the expected value of (42) is:

$$E\left[c_{*}^s(x_o)-c_{*}(x_o)\right]\cdot W^0(x_o)=0$$

(43)

Therefore,

$$E\left[c_{*}^s(x_o)\right]=c_{*}(x_o)$$

(44)

 □

Proposition 2.

Let $e(x)$ be a distribution not correlated for different points of $\mathsf{\Omega }$, $E\left[e^S(x)·e^S(y)\right]={\sigma }^2(x).\delta (x-y)$. Then, the variance of the regression corresponding to the values $y^s$ is given by:

$$Var(c^s(x_o)-c_{*}(x_o)). {(W^0(x_o))}^2={\int }_{\mathsf{\Omega }}{\sigma }^2(x) {\mathsf{\Phi }}^2(\Vert x-x_o\Vert ,\omega ) dx$$

(45)

Proof. 

Taking (42) to the square and calculating the expected values:

$$\begin{array}{c}E\left[{(c_{*}^s(x_o)-c_{*}(x_o))}^2\right]\cdot {(W^0(x_o))}^2={\int }_{\mathsf{\Omega }}{\int }_{\mathsf{\Omega }}E\left[e^S(x)\cdot e^S(y)\right]\cdot \mathsf{\Phi }(\Vert x-x_o\Vert ,\omega ) ·\\ \mathsf{\Phi }(\Vert y-x_o\Vert ,\omega )dx dy= {\int }_{\mathsf{\Omega }}{\sigma }^2(x) \cdot {\mathsf{\Phi }}^2(\Vert x-x_o\Vert ,\omega ) dx\end{array}$$

(46)

That is:

$$Var (c_{*}^s(x_o)-c_{* }(x_o)). {(W^0(x_o))}^2={\int }_{\mathsf{\Omega }}{\sigma }^2(x)·{\mathsf{\Phi }}^2(\Vert x-x_o\Vert ,\omega ) dx$$

(47)

 □

Proposition 3.

(Upper bound of the error on inner points). Let f be a function $fϵC^2(\mathsf{\Omega })$ defined on $\mathsf{\Omega }\subset {ℝ}^d$, $x_o\in { \mathrm{K}}_{\mathsf{\Omega }}^0(\omega ,\chi ) \cap A{K}_{\mathsf{\Omega }}(\omega ,\chi )$ and $y^S$ a Gaussian experimental realisation of $f$. Then:

$$E\left[\left|{\widehat{z}}^S(x_o)-f(x_o)\right|\right] \le M·d·h^2+\sqrt{\frac{2}{\pi }} ·\sigma$$

(48)

Proof. 

From Equation (37):

$$W^0(x_0)·\left[{\widehat{z}}^S(x_o)-f(x_o)\right]=W^0(x_0)·\left[{\sum }_{r=1}^Q{N}_r(x_o)\cdot c_{*}^s({\zeta }_r)-f(x_o)\right]$$

(49)

If the element containing $x_o$ accomplishes that ${\mathsf{\Omega }}_i\subset {\mathrm{K}}_{\mathsf{\Omega }}^0(\omega ,\chi )$ up to precision $\chi$, $W^0(x)\approx W^0({\zeta }_r)\approx W^0$, and using (32) and (41):

$$\begin{array}{c}{W}^0(x_0)\cdot \left[{\widehat{z}}^S(x_o)-f(x_o)\right]=W^0({\zeta }_r)\cdot \left[{\sum }_{r=1}^Q{N}_r(x_o)\cdot c_{*}^s({\zeta }_r)-f(x_o)\right]=\\ {\sum }_{r=1}^Q{N}_r(x_o)\cdot {\int }_{\mathsf{\Omega }}\left[y^s(x)-f(x_o)\right]\cdot \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega )dx={\sum }_{r=1}^Q{N}_r(x_o)\cdot \\ {\int }_{\mathsf{\Omega }}\left[f(x)-f(x_o)\right]\cdot \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega )dx+{\sum }_{r=1}^Q{N}_r(x_o)\cdot {\int }_{\mathsf{\Omega }}{e}^S(x)\cdot \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega )dx\end{array}$$

(50)

Developing now $f(x)$ around $x=x_o$, the last expression takes the form:

$$\begin{array}{c}{W}^0(x_o)·({\widehat{z}}^S(x_o)-f(x_o))={\sum }_{r=1}^Q{N}_r(x_o)\cdot {\int }_{\mathsf{\Omega }}(\nabla f(x_o)\cdot (x-x_o)+\frac{1}{2}\cdot D^{\alpha }f(p_r(x))\cdot \\ {(x-x_o)}^{\alpha })· \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega ) dx +{\sum }_{r=1}^Q{N}_r(x_o)\cdot {\int }_{\mathsf{\Omega }}{e}^S(x)· \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega ) dx\end{array}$$

(51)

where $\alpha$ is a two-dimensional multi-index. Writing $x-x_o=x-{\zeta }_r+{\zeta }_r-x_o$, the first part of the integral takes the form:

$$\begin{array}{c}{\int }_{\mathsf{\Omega }}{\nabla }_{i}f(x_o)\cdot {(x-x_o)}^i· \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega ) dx ={\nabla }_{i}f(x_o)\cdot {\int }_{\mathsf{\Omega }}{(x-{\zeta }_r)}^i·\\ \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega ) dx +{\nabla }_{i}f(x_o)\cdot {\int }_{\mathsf{\Omega }}{({\zeta }_r-x_o)}^i· \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega ) dx\end{array}$$

(52)

If that ${\zeta }_r\in A{K}_{\mathsf{\Omega }}(\omega ,\chi )$ the first integral is approximately zero, while for the second:

$$\begin{array}{c}{\sum }_{r=1}^Q{N}_r(x_o) ·{\nabla }_{i}f(x_o)\cdot {({\zeta }_r-x_o)}^i·W^0({\zeta }_r)=W^0(x_o)·{\nabla }_{i}f(x_o)·{\sum }_{r=1}^Q{N}_r(x_o) \cdot \\ {({\zeta }_r-x_o)}^i=0\end{array}$$

(53)

That is:

$$\begin{array}{c}{W}^0(x_o)·({\widehat{z}}^S(x_o)-f(x_o))={\sum }_{r=1}^Q{N}_r(x_o)\cdot \frac{1}{2}\cdot {\int }_{\mathsf{\Omega }}{D}^{\alpha }f(p_r(x))\cdot {(x-x_o)}^{\alpha }· \\ \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega ) dx +{\sum }_{r=1}^Q{N}_r(x_o)\cdot {\int }_{\mathsf{\Omega }}{e}^S(x)· \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega ) dx\end{array}$$

(54)

Taking the absolute value:

$$\begin{array}{c}\left|W^0(x_o)·({\widehat{z}}^S(x_o)-f(x_o))\right|\le \frac{1}{2}\cdot {\sum }_{r=1}^Q{N}_r(x_o)\cdot {\int }_{\mathsf{\Omega }}\Vert D^{\alpha }f(p_r(x))\Vert ·\Vert {(x-x_o)}^{\alpha }\Vert ·\\ \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega ) dt + {\sum }_{r=1}^Q{N}_r(x_o)\cdot {\int }_{\mathsf{\Omega }}\left|e^s (x)\right|· \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega )dx\end{array}$$

(55)

If M is an upper bound for the norm of the hessian matrix:

$$\begin{array}{c}{W}^0·\left|{\widehat{z}}^S(x_o)-f(x_o)\right|\le \frac{1}{2}\cdot {\sum }_{r=1}^Q{N}_r(x_o)\cdot {\int }_{\mathsf{\Omega }}M·\Vert x-x_o\Vert {}^2· \mathsf{\Phi }(\Vert x-\\ {\zeta }_r\Vert ,\omega ) dt \le W^0·\frac{1}{2}\cdot M·d·h^2+ {\sum }_{r=1}^Q{N}_r(x_o)\cdot {\int }_{\mathsf{\Omega }}\left|e^s (x)\right|· \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega )dx\end{array}$$

(56)

For the last integral, the expected value is:

$$E[{\int }_{\mathsf{\Omega }}\left|e^s (x)\right|· \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega )dx]={\int }_{\mathsf{\Omega }}{\int }_0^{\infty }\left|ϵ\right|·\pi (x,\left|ϵ\right|)· \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega )de dx$$

(57)

where $\pi (x,\left|ϵ\right|)$ is the probability of an error $ϵ$ in the point given by x. In the case when this probability is independent of the point $\pi (x,\left|ϵ\right|)=\pi (\left|ϵ\right|)$:

$$E[{\int }_{\mathsf{\Omega }}\left|e^s (x)\right|· \mathsf{\Phi }(\Vert x-{\zeta }_r\Vert ,\omega )dt]=W^0({\zeta }_r)·{\int }_0^{\infty }\left|ϵ\right|·\pi (\left|ϵ\right|)dϵ=W^0·{\int }_0^{\infty }\left|ϵ\right|·\pi (\left|ϵ\right|)dϵ$$

(58)

In the case of a Gaussian distribution of $ϵ$:

$$E\left[\left|{\widehat{z}}^S(x_o)-f(x_o)\right|\right]\le \frac{1}{2}\cdot M·d·h^2+ \sqrt{\frac{2}{\pi }}·\sigma$$

(59)

 □

3.3. Computational Algorithm

A natural selection for the parameter $\omega$ of the radial function is the value corresponding to the element width h. In addition, in real cases, complete sets of values of $y^S(x)$ are not available, so one must use just a subset of P points. Then, from this subsample, the integrals are calculated using finite sums, so:

$${\int }_{\mathsf{\Omega }} \mathsf{\Phi }(\Vert x-x_o\Vert ,\omega ) dx\cong \frac{\mu (\mathsf{\Omega })}{P}·{\sum }_{k=1}^P\mathsf{\Phi }(\Vert x_k-x_o\Vert ,h)$$

(60)

$${\int }_{\mathsf{\Omega }} y^s(x)·\mathsf{\Phi }(\Vert x-x_o\Vert ,\omega ) dx\cong \frac{\mu (\mathsf{\Omega })}{P}·{\sum }_{k=1}^P{y}_k^S·\mathsf{\Phi }(\Vert x_k-x_o\Vert ,h)$$

(61)

Now, the finite sample version of (40) is:

$${\sum }_{k=1}^P\left[c_S(x_o)-y_k^S\right]· \mathsf{\Phi }(\Vert x_k-x_o\Vert ,h)=0$$

(62)

Following Expression (62), the proposed algorithm can be condensed in the next schema (Algorithm 1):

Algorithm 1.

1. Begin

2. Step 1: for i =1:P do

3. Initialize the point Q_1 with X[i].    4. Initialize estimation[i] with 0.

5. for k = 1:2dimension do

6. Initialize the point Q_node with Node[k].      7. Initialize dist and nodeEstim with 0.

6. for j = 1:P do

7. Initialize the point Q_2 with X[j].

8. Increment dist with Radial_Kernel(Q_node,Q_2)

9. Increment nodeEstim with Z[j] * Radial_Kernel(Q_node,Q_2).

10. end for j

11. end for k

12. Update the estimate node as nodeEstim = nodeEstim/dist.

13. Increment estimation[i] with nodeEstim * ShapeFunction(k,Q_1).

14. end for i

15. End.

The calculation of radial kernels in lines 9 and 10 for a radial function as that in Equation (29), dimension d implies the sums of the d-coordinate differences $r^i=x^i-x_k^i$. The loop from 7 to 10 iterates over the number of points P, so given that these steps are repeated in the $2^d$ nodes of the finite element for each objective points, the computational time of the algorithm is $t_c~O(P^2·2^d·\mathrm{d})$. The memory complexity corresponds to saving the experimental points, that is, $m~O(P)$. An analysis of the relation between execution times and number of points will be considered in the applications section.

The analysis developed above shows clearly the improvement of the current algorithm compared with those presented in [4,5] where the dominant computational orders were $O({(c+1)}^{3d})$ and $O({(c+1)}^d)$. Remembering that the complexity is related with the precision of the estimation (without considering overfitting problems), the previous methodologies were highly limited in their application by the dimensionality of the problem. However, this limit disappears in this new approach, with the number of points being the main limiting factor.

Let us consider a normalised dataset. Then, the models are determined by the parameter $h=\frac{1}{c}$ that represents the length of the edge of the hypercubes that form the discretization of the domain $\mathsf{\Omega }$. Each value of c (complexity) determines a unique model, but following (37), when $h\to 0$, the regression estimation accomplishes that:

$$\widehat{z}(x_o)\to c_{*}^s(x_o)$$

(63)

In that limit, $\widehat{z}(x_o)$ corresponds to a weighted average of the K-nearest points to $x_o$, with K depending on the parameter $\omega$ in $\mathsf{\Phi }$, which has been fixed as $\omega =h$.

Let us consider the case of an estimation on a point $(x_0^1,x_0^2,..,x_0^d)$, based on the model defined by a dataset ${\left\{(x_k^1,x_k^2,..,x_k^d;y_k)\right\}}_{k=0}^P$ containing $x_o$. For commodity, also let the dataset be ordered with respect to the distance to $x_o$, with ${\mathsf{\Phi }}_i=\mathsf{\Phi }(\Vert x_i-x_o\Vert ,h)$ the parameterized radial function. The weighted average regression is then obtained as:

$$c_{*}^s(x_o)·(1+{\sum }_{k=1}^P {\mathsf{\Phi }}_k)=y_0^S+{\sum }_{k=1}^P{y}_k^S·{\mathsf{\Phi }}_k$$

(64)

However, if the estimation is calculated without using the self $x_o$, the restricted weighted average regression $c_{r*}^s$ takes the form:

$$c_{r*}^s(x_o)·{\sum }_{k=1}^P {\mathsf{\Phi }}_k={\sum }_{k=1}^P{y}_k^S· {\mathsf{\Phi }}_k$$

(65)

Therefore, there exists a relationship between both expressions:

$$c_{*}^s(x_o)·(1+{\sum }_{k=1}^P {\mathsf{\Phi }}_k)=y_0^S+c_{r*}^s(x_o)·{\sum }_{k=1}^P {\mathsf{\Phi }}_k$$

(66)

If the model dataset has a low density, the sum vanishes compared with the constant terms, obtaining:

$$c_{*}^s(x_o)\approx y_0^S$$

(67)

Joining (63) and (67), in the limit $h\to 0$, the result of the estimation tends to $\widehat{z}(x_o)\to y_0^S$, so there will be a trend to the overfitting as the complexity grows in the full model.

The restricted model obtained from Equation (65) does not incorporate the information about the experimental value, if present, on the objective point, so the model is not using all the available information and it is obviously under fitted.

3.4. Applications

To test the algorithm behaviour, several problems have been selected from the UCI Machine Learning Datasets [69] to compare the results with those presented in other papers obtained using different techniques. UCI is a well-known repository and widely used data source for problems related to machine learning, regression, and time series prediction.

The fit of a model can be measured using different parameters, where $y_i$ is the observed and $\widetilde{y_i}$ is the estimated value:

• R² coefficient: The strict use of R² is limited to the linear regression. Out of this case, the value of R² can vary in the interval $\left]-\infty ,1\right]$. However, the expression that defines R² is related to the other family of quality indicators as Legates-McCabe, Nash and Sutcliffe, and Willmott:

  $$R^2=1-\frac{{\sum }_{i=1}^P{(y_i-\widetilde{y_i})}^2}{{\sum }_{i=1}^P{(y_i-E\left[y\right])}^2}$$
• Mean squared error (MSE): This indicator is associated to the second moment of the error, incorporating information about its variance and bias. Its principal problem is the overweighting of outliers, so usually, other parameters as MAE are preferred:

  $$MSE=\frac{1}{P}·\sum _{i=1}^P{(y_i-\widetilde{y_i})}^2$$
• Root mean squared error (RMSE): Squared root of MSE. Its main advantage, with respect to it, is that it is measured in the same units of the predicted variable. However, it suffers of the same problems with respect to the outlier influence:

  $$RMSE=\sqrt{\frac{1}{P}·\sum _{i=1}^P{(y_i-\widetilde{y_i})}^2}$$
• Mean absolute error (MAE): The best characteristic of MAE compared to RMSE is the equal weight of the deviations on the error. The ratio RMSE/MAE serves as an indication of the outlier presence on the sample:

  $$MAE=\frac{1}{P}·\sum _{i=1}^P\left|y_i-\widetilde{y_i}\right|$$
• Mean absolute percentage error (MAPE): This parameter is frequently used by its simple interpretation as a relative error. However, it presents a bias that tends to select the model with smaller forecasts. One option could be the symmetric MAPE, but is not so widely used:

  $$MAPE=\frac{100}{P}·\sum _{i=1}^P\left|\frac{y_i-\widetilde{y_i}}{y_i}\right|$$
• Regression error characteristic (REC) curve: It is a curve that obtains plotting the error tolerance on the X-axis versus the percentage of points predicted within the tolerance on the Y-axis.

The proposed methodology is a deterministic algorithm and the obtained model (and in consequence the quality parameters) depends only on the available experimental data, so there is no difference in them between several runnings of the model.

3.4.1. Air foil Self Noise

The study of self-noise is an active topic in aerodynamics with a great number of research papers devoted to its prediction or improving the design of air foils and other structures. The phenomena related to this problem are complex involving vortex, turbulence, and flows in the linear-limit.

The NASA “Air foil Self Noise” dataset [70,71] was obtained from a series of aerodynamic and acoustic tests of two and three-dimensional air foil sections conducted in an anechoic wind tunnel. The data set comprises different size NACA 0012 air foils (with sizes from 2.5 to 61 cm) at various wind tunnel speeds (up to Mach 0.21) and angles of attack (from 0 to 25.2°). The span of the air foil and the observer position were the same in all of the experiments. The measurements were made in a wind tunnel.

The data is composed of 1503 rows with five independent variables to determine the sound pressure level as indicated in

Table 1. Input and output variables for the air foil self noise.

Input Variables	Output Variable
1. Frequency (Hz).	Scaled sound pressure level (dB)
2. Angle of attack (deg).
3. Chord length (m).
4. Free-stream velocity (m/s).
5. Suction side displacement thickness (m).

.

Table 2. Parameters of quality for models of air foil self noise dataset in previous papers.

Research	Model	R2	MSE	RMSE	MAE
[71]	LR	0.558	22.129	4.704	3.672
	DT	0.871	6.457	2.541	1.788
	SVMR	0.751	12.483	3.533	2.674
	RF	0.935	3.283	1.812	1.313
	ABR	0.725	13.796	3.714	3.078
	XGBR	0.946	2.707	1.645	1.074
	Baseline NN	0.676	16.236	4.029	3.078
	Deeper NN	0.802	9.916	3.149	2.474
[72]	RF	0.929
	SGTB	0.969
[73]	NN	0.952 (R2 of linear regression predicted/real)
[74]	NN	0.894
[75]	NN	0.943/0.838	(several models)

shows the parameters of quality for the models obtained in previous research using this dataset. The techniques are linear regression (LR), decision tree (DT), support vector machine regression (SVMR), random forest regression (RF), AdaBoost regression (ABR), XGBoost regression (XGBR), several types of neural networks (NN), and Stochastic Gradient Tree Boosting (SGTB).

The first step is to study the behaviour of a cross validation using (80% of the data as train and 20% as test sets) and 100 iterations. The result for different values of the complexity is shown in Figure 2.

Apart from the expected result of a smaller error in the train subsets, an important behaviour can be observed in the MAE corresponding to the test subsamples with a stable value further than complexity 90 (with a mean MAE of 1.958). This value corresponds to a complexity close to 30 on the train subsamples.

The next step is the study of the models generated by the full dataset to test if this trend is verified. To do this, for each complexity two estimations have been done for the points of the dataset, corresponding to the full and the restricted estimations introduced in Equations (64) and (65). The results of MAE, MAPE, and R² coefficients are shown in Figure 3.

The behaviour is consistent with the one observed in Figure 2, with a stable value in MAE around complexity 100 and MAPE from complexity 80. In addition, it is clear how the full model has diminishing values for MAE and MAPE coefficients when the complexity grows (making evident in that case the existence of overfitting because improvements in the full model are not parallel to improvements in the values of the restricted model).

Let us then study the characteristics of the model with complexity 90.

Figure 4 shows the errors as pairs observed/estimated for the full and restricted models for complexity 90.

The parameters of quality for this model are shown in

Table 3. Parameters of the model with complexity 90 for the air foil self noise dataset.

Parameter	Full Model	Restricted Model
R2	0.964	0.887
MAE	0.755	1.679
MAPE	0.60%	1.35%
MSE	1.711	5.387
RMSE	1.308	2.321

.

The results are comparable to those presented in Table 2 corresponding to previous papers.

The computation time over 10 iterations has been measured for different complexities, obtaining a result shown on Figure 5. Given the expression of the computational order $O(P^2·2^d·d)$ no dependence on complexity is expected, a result that is ratified by the plot results.

To confirm the quadratic dependence on the number of points $O(P^2·2^d·d)$, a study of the computing time for different values of P in the dataset, for a high complexity (in this study complexity 100 has been selected) has been realised. The result for the value of the variable has been obtained, the computing time divided by the square of the number of points is shown in Figure 6.

The constant behaviour at a greater number of points of the last figure certifies the hypothesis introduced on the dependence $t_c~P^2$.

3.4.2. Combined Cycle Power

The problem of effective prediction of power output for electric power plants is of great importance in order to optimize the economic profit by the generated megawatt. For a gas turbine, the relationship between the power available under full load conditions and variables as temperature, pressure, relative humidity, and the exhaust vacuum measured from steam turbine, is complex enough to be studied using machine learning methods. Specifically, given the diversity in the values of the different periods (summer and winter) regression and prediction methods based on locality will have a better performance. This makes this dataset a good benchmark for the current methodology.

The dataset contains 9568 data points collected from a combined Cycle Power Plant over six years (2006–2011), when the power plant was set to work with a full load. The variables that correspond to the hourly average values are presented in

Table 4. Input and output variables for the combined cycle power dataset.

Input Variables	Output Variable
1. Temperature	Net hourly electrical energy output
2. Ambient pressure
3. Relative Humidity
4. Exhaust Vacuum

.

For additional details, consider the original papers [76,77]. The models considered in [76] are: (i) Least median square (LMS); (ii) support vector poly kernel regression (SMOReg); (iii) KStar (K *); (iv) bagging REP tree (BREP); (v) model trees rules (M5R); (vi) model trees regression (M5P) and; (vii) reduced error pruning (REP) trees. The results presented in

Table 5. Parameters of quality for the models of combined cycle power dataset in previous research.

Research	Model	MSE	RMSE	MAE
[76]	LMS	20.903	4.572	3.621
	SMOReg	20.821	4.563	3.620
	K *	14.907	3.861	2.882
	BREP	14.341	3.787	2.818
	M5R	17.040	4.128	3.172
	M5P	16.704	4.087	3.140
	REP	17.733	4.211	3.133
[77]	k-NN + ANN	19.990
	k-means + ANN	15.430

for [77] are: (i) k-nearest neighbour (k-NN) smoother + feedforward error backpropagating ANN and; (ii) the best of several combinations of K-means clustering + feedforward error backpropagating ANN.

Following the steps done in the first example, the cross validation results can be seen in Figure 7.

Now, the value of the stabilised MAE for the test subsamples (calculated from complexities above 70) is 2.479, corresponding to a complexity of 50 in the train subsample. Running the full and restricted models for complexities between 10 and 200, it can be seen that the stable behaviour for MAE and MAPE starts before that value, and can be fixed around complexity 70. Figure 8 shows the comparative of the quality parameters (MAE, MAPE, and R²) for the full and restricted models on the dataset, for different complexities from 10 to 200.

To be conservative with respect to the possibility of overfitting, complexity 60 is selected to run the final model. The corresponding plot of estimated versus experimental values can be seen in Figure 9.

The parameters of quality for this model are shown in

Table 6. Parameters of the model with complexity 70 for the combined cycle power dataset.

Parameter	Full Model	Restricted Model
R2	0.979	0.957
MAE	1.741	2.572
MAPE	0.39%	0.57%
MSE	6.102	12.592
RMSE	2.470	3.548

.

The computation time has been also obtained from 10 iterations for complexities between 10 and 200, showing its independence in Figure 10.

Again, a representation of $t_c/P^2$ for complexity 100 is shown in Figure 11.

4. Discussion

Several questions must be considered in this step. Firstly, the presented algorithm continues the line of work developed by the authors in previous papers, see [3,4,78], solving the common bottleneck caused by the dependence of the time of computation on the complexity of the model at orders $O({(c+1)}^{3d})$ and $O({(c+1)}^d)$. Now, the dependence on the dimension has an expression of $d·2^d$. This represents an important advance, allowing the treatment of a greater number of problems.

Secondly, given that for complexities less than 100, (for the most part of the studied problems this value is usually around 70–80), there exists a direct relationship between the complexity of the model and the precision of the estimation, the independence of the computing time of current algorithm from this parameter is an important result. However, the problem has been now displaced to the number of points, which contribution to the total computing time was hidden in the previous methodologies. Therefore, the two new challenges for future investigations will be the use of more efficient algorithms and data structures to improve the dependence of the number of points and the dimensionality of the problem that is present as $d·2^d$.

On the other side, the characteristics of the methodology allow a cheap parallel calculation of the estimated values for points belonging to the sample that determines the model in two different ways: The full and the restricted models. The study of their behaviour for different complexities allows the inclusion of a benchmark to capture the overfitting in the model. Overfitting of models is a very common problem that appears where the model tends to treat in-model and out-of-model points in a different way causing the estimations to be useless. Following in this direction, considering the development of algorithms for automatic detection of overfitting, and the selection of optimum complexity will be another pending task for future investigations. Considering that the process of selecting the most adequate complexity for the model is equivalent to optimising the error of a weighted combination of both models, its automation seems to be a feasible problem.

However, at the moment, obtaining an adequate combination of the full and restricted models introduced in 4.3 is a complex problem. This makes the comparisons with other available methodologies diffuse because the quality parameters of the optimal model would be an intermediate case between those corresponding to the full and restricted cases.

Another characteristic that could be considered as a distinctive aspect of the methodology is its ‘liquidity’, in the sense of the absence of a model as such, that is, the methodology does not generate any additional structure beyond the estimations and the optimum value for the h parameter. Thus, it can also be viewed as a positive aspect.

Finally, the proposed methodology has been used on well-known problems in order to compare the results with those obtained in other papers and methods.

For the Air Foil Self Noise dataset, the quality parameters of the restricted model, considered as the worst prediction are outperformed by decision trees, XGBoost regression, and some models of NN. However, the full model, considered as the most optimistic model (in occasions presenting overfitting) obtain results similar to the best available models.

In the case of Combined Cycle Power dataset, both models (full and restricted) obtain better quality parameters than the methodologies previously considered. This very interesting result should be studied more in depth to understand it, especially in the case of the restricted model.

In addition, a wider use of the regression method must be faced by increasing the number of studied datasets, and comparing the results with other available methods, especially those based on radial basis functions, to compare their performance.