Nonparametric Estimation of Continuously Parametrized Families of Probability Density Functions—Computational Aspects

Abstract

We consider a rather general problem of nonparametric estimation of an uncountable set of probability density functions (p.d.f.’s) of the form: $f(x;r)$, where r is a non-random real variable and ranges from $R_1$ to $R_2$. We put emphasis on the algorithmic aspects of this problem, since they are crucial for exploratory analysis of big data that are needed for the estimation. A specialized learning algorithm, based on the 2D FFT, is proposed and tested on observations that allow for estimate p.d.f.’s of a jet engine temperatures as a function of its rotation speed. We also derive theoretical results concerning the convergence of the estimation procedure that contains hints on selecting parameters of the estimation algorithm.

1. Introduction

Consider a family $f(x;r)$ of functions such that for every $r\in [R_1,R_2]\subset \mathbb{R}$ $f(.;r):\mathbb{R}\to \mathbb{R}$ is a probability density function (p.d.f.) on real line $\mathbb{R}$, while non-random parameter r takes values from a finite interval $[R_1,R_2]$. Assume that we have observations $({\kappa }_l,r_l)$, $l=1,2,\dots d$ at our disposal, where for $r_l\in [R_1,R_2]$ observation ${\kappa }_l$ is drawn at random according to p.d.f. $f(x;r_l)$, $x\in \mathbb{R}$. Our aim is to propose, under mild assumptions, a nonparametric estimator of the whole continuum set of p.d.f.’s $\mathcal{F}=\{f(.;r),r\in [R_1,R_2]\}$, assuming that the number of data $d\to \infty$ and that $r_l$’s cover $[R_1,R_2]$ more and more densely as $d\to \infty$. Later on, we shall refer to the above stated problem as the $\mathcal{F}$-estimation problem.

In this paper, we concentrate mainly on the algorithmic aspects of the $\mathcal{F}$-estimation problem, since it is computationally demanding. However, we also provide an outline of the proof that the proposed learning algorithm is convergent in the integrated mean squared error (IMSE) sense.

Before providing motivating examples, we briefly indicate similarities, differences, and a generality of this problem among other nonparametric estimation tasks that were considered from the 1950s [1,2,3,4]:

• The $\mathcal{F}$-estimation problem has certain similarities to the nonparametric estimation problem of a bivariate p.d.f. Notice, however, the important difference, namely in our case r is a non-random parameter. In other words, our sub-task is to select $r_l$’s in an appropriate way (or to confine ourselves to such an interval $[R_1,R_2]$ which is covered densely by passive observations of pairs $({\kappa }_l,r_l)$, $l=1,2,\dots d$).
• One can also notice a formal similarity of our problem and the problem of nonparametric estimation of a non-stationary p.d.f. $f(x;t)$, say that was introduced to the statistical literature by Rutkowski (see [5,6,7] ) and continued in the papers on the concept drift tracking (see [8,9,10]). There is, however, a fundamental difference between the time t and parameter r. Namely, time is not reversible and we usually do not have an opportunity to repeat observations at instants preceding present t. On the contrary, in the $\mathcal{F}$-estimation problem, we allow the next observation to be done at $r_{l+1}<r_l$. Furthermore, we allow also for repeated observations for the same value of r.
• The estimation of several p.d.f.’s was considered in [11], where it was pointed out that it is a computationally demanding problem, because all of these densities should be estimated simultaneously. However, the goal of this paper is quite different than ours. Namely, in [11], the goal was to compare several densities that are not necessarily indexed by the same additional parameter, which does not arise as a parameter of an estimator.
• Denote by $\widehat{f}(.;r)$ nonparametric estimators of $f(.,r)$, $r\in [R_1,R_2]$. Having them at our disposal, we immediately obtain nonparametric estimators of a regression function: ${\int }_{-\infty }^{\infty }x\widehat{f}(x;r)dx$ with r as the input variable.
• Similarly, calculating the median of $\widehat{f}(.;r)$, we obtain an estimator of the median regression on r. Analogously, estimators of other quantile regression functions can be obtained.
• When we allow that x and/or r if $f(x;r)$ are vectors, then the $\mathcal{F}$-estimation problem covers also multivariate density and regression estimation problems. In this paper, we do not follow these generalizations, since even for univariate x and r we obtain computationally and data demanding problem. On the other hand, we propose double orthogonal expansion as the base for solving the $\mathcal{F}$-estimation problem. Replacing orthogonal functions of x and r by their multivariate counterparts, we obtain an estimator that formally covers also multivariate cases, but it is still non-trivial to derive a computationally efficient algorithm and to establish its asymptotic properties.

Below, we outline examples of possible applications of the $\mathcal{F}$-estimation:

• The temperature of a jet engine x depends on the rotating speed r of its turbine and on many other non controllable factors. It is relatively easy to collect a large number of observations $({\kappa }_l,r_l)$, $l=1,2,\dots d$ from proper and improper operating conditions of the engine. For diagnostic purposes, it is desirable to estimate the continuum set of p.d.f.s of the temperatures for rotating speeds $r\in [R_1,R_2]$. This is our main motivating example that is discussed in detail in Section 7.1.
• Consider the fuel consumption x of a test exemplar of a new car model. The main impact on x comes from the car speed r, but x also depends on the road surface, the type of tyres and many other factors. It would be of interest for users to know the whole family $\mathcal{F}$ p.d.f.’s in addition to the mean fuel consumption.
  In a similar vain, it would be of interest to estimate $\mathcal{F}$ when x is the braking distance of a car running at speed r.
• Cereal crops x depend on an amount r of a fertilizer applied to a unit area as well as on soil valuation, weather conditions, etc. Estimating $\mathcal{F}$ would allow for selecting r which provides a compromise between a high yield and its robustness against other conditions.

Taking into account a rapidly growing amount of available data and increasing computational power, it would be desirable to extend many other examples of nonparametric regression applications to estimating the whole $\mathcal{F}$.

Our starting point for constructing an estimator for $\mathcal{F}$ is nonparametric estimation of p.d.f.’s by orthogonal series estimators. We refer the reader to the classic results in this field [1,3,12,13,14,15]. We shall need also some results on the influence of rounding errors on these estimators (see [16,17]).

The paper is organized as follows: the derivation of the algorithm is presented, a fast method of computation is proposed. Subsequently, tests of synthetic data are preformed and the convergence of the method is shown. Finally, a real world problem regarding jet turbine temperature is presented as well as other possible applications. As an appendix, a detailed proof of convergence is given.

2. Derivation of the Estimation Algorithm

Let us define $\left(X\right(r),r)$ as a generic pair, where—for fixed $r\in [R_1,R_2]$—random variable (r.v.) $X\left(r\right)$ has p.d.f $f(x;r)$. We use a semicolon; in order to indicate that r is a non-random variable. For simplicity of the exposition, we assume that $X\left(r\right)$ have bounded supports, $[S_1,S_2]\subset \mathbb{R}$ say, which is the same for every $r\in [R_1,R_2]$. Thus, the family $\mathcal{F}$ of p.d.f.’s is defined on $[S_1,S_2]\times [R_1,R_2]$. We additionally assume that f is squared integrable, i.e., $f(.;.)\in L_2([S_1,S_2]\times [R_1,R_2])$.

2.1. Preparations—Orthogonal Expansion

Consider two orthogonal and normalized (orthonormal) sequences $v_k\left(x\right),x\in [S_1,S_2]k=1,2,\dots$ and $T_j\left(r\right),r\in [R_1,R_2],j=1,2,\dots$ that are complete in $L_2(S_1,S_2])$ and $L_2\left([R_1,R_2]\right)$, respectively. Then, $f(x;r)$ can be represented by the series (convergent in $L_2[S_1,S_2]$) with the following coefficients:

$$a_k\left(r\right)={\int }_{S_1}^{S_2}{v}_k\left(x\right)f(x;r)dx,k=1,2,\dots$$

(1)

Notice that $a_k\left(r\right)$’s can be interpreted as follows:

$$a_k\left(r\right)=E_r\left[v_k\left(X\left(r\right)\right)\right],$$

(2)

where $E_r$ stands for the expectations with respect to random variable $X\left(r\right)$, having p.d.f $f(x;r)$. Furthermore, each $a_k\left(r\right)$ can be represented by the series:

$$a_k\left(r\right)=\sum _{j=1}^{\infty }{\alpha }_{kj}{T}_j\left(r\right)$$

(3)

with constant coefficients ${\alpha }_{kj}$, defined as follows:

$${\alpha }_{kj}={\int }_{R_1}^{R_2}{a}_k\left(r\right)T_j\left(r\right)dr,k,j=1,2,\dots$$

(4)

Series (3) is convergent in $L_2\left([R_1,R_2]\right)$. By back substitution, we obtain the following series representation of f:

$$f(x;r)=\sum _{k=1}^{\infty }\sum _{j=1}^{\infty }{\alpha }_{kj}{v}_k\left(x\right)T_j\left(r\right)$$

(5)

The series in (5), convergent in in $L_2([S_1,S_2]\times [R_1,R_2])$, forms a base for constructing estimators for $\mathcal{F}$. They differ in the way of estimating ${\alpha }_{kj}$’s and in the way of resolving the so called bias-variance dilemma. The latter can be resolved by appropriate down-weighting ${\alpha }_{kj}$’s estimators. In this paper, we confine ourselves to the simplest way of down-weighting, namely, to the truncation of the both sums in a way described later on.

2.2. Intermediate Estimator

The simplest, from the computational point of view, estimator $\widehat{f}(x;r)$ for the family $f(x;r),r\in [R_1,R_2]$, we obtain when we additionally assume that the observations of $X\left(r\right)$’s are made on an equidistant grid ${\rho }_1<{\rho }_2<\dots <{\rho }_M$ that splits $[R_1,R_2]$ into non-overlapping intervals of the length ${\Delta }_r>0$, which cover all $[R_1,R_2]$ in such a way that $R_1={\rho }_1-{\Delta }_r/2$ and $R_2={\rho }_M+{\Delta }_r/2$. In this section, we tentatively impose the restriction that only repeated, but independent and identically distributed (i.i.d.) observations of $X\left({\rho }_m\right)$, $m=1,2,\dots ,M$ are available. In the asymptotic analysis at the end of this paper, we shall assume that M grows to infinity with the number of observations d. Then, also positions of ${\rho }_m$’s and ${\Delta }_r$ will be changing, but we do not display this fact in the notations, unless necessary.

At each of ${\rho }_m$, $m=1,2,\dots ,M$, $n_m>0$ observations $(X_l\left({\rho }_m\right),{\rho }_m)$, $l=1,2,\dots ,n_m$ are made, keeping the above-mentioned assumptions on mutual independence in force. Additionally, the mutual independence of the following lists of r.v.’s is postulated:

$$\{X_l\left({\rho }_m\right),l=1,2,\dots ,n_m\},m=1,2,\dots ,M$$

(6)

Then, one can estimate ${\alpha }_{kj}$’s by

$${\widehat{\alpha }}_{kj}={\Delta }_r\sum _{m=1}^M{\widehat{a}}_k\left({\rho }_m\right)T_j\left({\rho }_m\right)$$

(7)

where

$${\widehat{a}}_k\left({\rho }_m\right)=\frac{1}{n_m}\sum _{l=1}^{n_m}{v}_k\left(X_l\left({\rho }_m\right)\right)$$

(8)

Estimators (7) are justified by (4). Notice that in (7) the simplest quadrature formula is used for approximating the integral ${\int }_{R_1}^{R_2}$. When an active experiment is applicable, then one can select ${\rho }_m$’s at nodes of more advanced quadratures with weights, in the same spirit as in [17,18] for nonparametric regression estimators, where the bias reduction was proved. In turn, estimators (8) are motivated by replacing the expectations in (2) by the corresponding empirical means.

Truncating series (5) at K-th and J-th terms and substituting ${\widehat{\alpha }}_{kj}$ instead of ${\alpha }_{kj}$, we obtain the following estimator if the family $f(x;r),r\in [R_1,R_2]$

$$\widehat{f}(x;r)=\sum _{k=1}^K\sum _{j=1}^J{\widehat{\alpha }}_{kj}{v}_k\left(x\right)T_j\left(r\right).$$

(9)

Later on, K and J will depend on the number of observations, but it is not displayed, unless necessary.

Estimator (9) is quite general in the sense that one can select (and mix) different orthonormal bases $v_k$’s and $T_j$’s. In particular, the trigonometric system, Legendre polynomials, Haar system, and other orthogonal vawelets can be applied. The reduction of computational complexity of (9) is possible when, for a given orthogonal system, its discrete and fast counterpart exists. We illustrate this idea in the next section, by selecting $v_k$’s and $T_j$’s as the trigonometric bases, applying discrete Fourier transform (DFT) and the fast Fourier transform (FFT) as its fast implementation.

3. Efficient Learning Algorithm

Our aim in this section is to propose an algorithm for fast calculations of ${\widehat{\alpha }}_{kj}$’s in (9), using the FFT method, which is necessary to learn a proper selection of K and J or a proper selection those of ${\widehat{\alpha }}_{kj}$’s that are essentially different from zero.

3.1. Data Preprocessing

The FFT algorithm operates on data on a grid. Thus, our first step is to attach the set of raw observations ${\mathcal{D}}_d\stackrel{def}{=}\{({\kappa }_l,r_l),l=1,2,\dots d\}$ to a grid.

We already have one axis of the grid, namely, points ${\rho }_m$, $m=1,2,\dots ,M$. Define ${\mathcal{B}}_m$, $m=1,2,\dots ,M$ in the following way. For $j=1,2,\dots ,d$ check:

$$x_{mj}\in {\mathcal{B}}_m\mathrm{iff}\exists ({\kappa }_l,r_l)\in {\mathcal{D}}_d:r_l\in [{\rho }_m-{\Delta }_r/2,{\rho }_m+{\Delta }_r/2),x_{mj}={\kappa }_l.$$

(10)

Notice that the contents of each ${\mathcal{B}}_m$’s depends on ${\Delta }_r$, but it is not displayed in the notation. Denote by ${\widehat{n}}_m$ the cardinality of ${\mathcal{B}}_m$. Clearly, we must have ${\sum }_{m=1}^M{\widehat{n}}_m=d$. Informally, one can consider $x_{mj}$, $j=1,2,\dots ,{\widehat{n}}_m$ in bin ${\mathcal{B}}_m$ as slightly distorted realizations of r.v.’s in (6).

The second split of the grid goes along the x-axis. Denote by ${\chi }_1<{\chi }_2<\dots {\chi }_N$, $N>1$ equidistant points such that for ${\Delta }_x={\chi }_2-{\chi }_1$ the intervals $[{\chi }_n-{\Delta }_x,{\chi }_n+{\Delta }_x)$, $n=1,2,\dots ,N$ cover the support $[S_1,S_2]$ and $S_1={\chi }_1-{\Delta }_x/2$, $S_N={\chi }_N+{\Delta }_x/2$.

Now, we are ready to define the number of observations $q_{mn}$ that are attached to each grid point $({\chi }_n,{\rho }_m)$. Namely, $q_{mn}$ is the number of observations $x_{mn}$ that are contained in bin ${\mathcal{B}}_m$ and simultaneously take values in $[{\chi }_n-{\Delta }_x,{\chi }_n+{\Delta }_x)$, $n=1,2,\dots ,N$, $m=1,2,\dots ,M$. Let us define $M\times N$ matrix P with elements:

$$p_{mn}=q_{mn}/\sum _{l=1}^N{q}_{ml},n=1,2,\dots ,N,m=1,2,\dots ,M.$$

(11)

Clearly, $p_{m.}$ sum up to 1. Notice, however, that—strictly speaking— $p_{m.}$’s are not histograms of r.v.’s $X\left({\rho }_m\right)$’s, since they are based on the observations contained in bins ${\mathcal{B}}_m$. Nevertheless, we shall later interpret them as such because – as ${\Delta }_r\to 0$ – $p_{m.}$’s converge to $f(x;{\rho }_m)$, assuming that also ${\Delta }_x\to 0$ in an appropriate way (see [13] for precise statements).

3.2. Fast Calculations and Smoothing

The crucial, mostly time-consuming step is smoothing preprocessed data contained in matrix P. Therefore, it is expedient to apply 2D FFT in order to calculate the DFT of P:

$$G=FF{T}_{2D}(P;M,N),$$

(12)

where the resulting $M\times N$ matrix G has complex elements $g_{mn}$, $n=1,2,\dots ,N$, $m=1,2,\dots ,M$ (see, e.g., [19] for the definition of 2D DFT and its implementation as 2D FFT).

Obviously, the inverse transform provides $P=FF{T}_{2D}^{-1}(G;M,N)$. Thus, in order to smooth P, we have to remove high frequency components from matrix G, retaining only its, appropriately chosen, $K\times J$ sub-matrix $\widehat{G}$, $1<K<M$, $1<J<N$ and setting to zero other elements of G Instead of setting zeros, one can apply at this stage so-called windowing, e.g., using the Hamming window that provides a mild way of approaching zero.

Remark 1.

The appropriate choice of $K\times J$ sub-matrix $\widehat{G}$, with elements ${\widehat{g}}_{kj}$, means that we have to take into account that the complex valued matrix G has the component corresponding to $(0,0)$ frequency, which is placed as $g_{11}$ (or $g_{MN}$). Analogously, other components of G, corresponding to low frequencies, are placed at the "corners" of this matrix. Hence, in order to cancel high frequencies, we have to reshape matrix G in order to have $(0,0)$ frequency in its middle and other low frequencies nearby. Then, to put zeros at the positions corresponding to high frequencies, select sub-matrix $\widehat{G}$ and reshape it in the order reverse to that of the reshaping G. This last step is necessary so as to obtain a smoothed version of the P matrix, which would be a $K\times J$ matrix.

Remark 2.

It is crucial for further considerations to observe that ${\widehat{\alpha }}_{kj}$’s are directly calculable from ${\widehat{g}}_{jk}$’s and their conjugates by adding or subtracting their real and imaginary parts.

Remark 3.

The choice of K, J or the choice of indexes k, j for which ${\widehat{alpha}}_{kj}$ is left as nonzero, is crucial for proper functioning of the estimation algorithm, since their choice dictates the amount of smoothness of the resulting surface. Below, we discuss possible ways of learning the algorithm to select them properly.

Although the $\mathcal{F}$ estimation problem differs from the one of estimating bi-variate p.d.f.’s, we may consider the methods elaborated in this field as candidates that might be useful also here.

 1.

The cross-validation approach—see [20], where the asymptotic optimality of this method is proved for the trigonometric and the Hermite series density estimators,

 2.

The Kronmal and Tarter rule [21], which—in our case—reads as follows. For $m=1,2,\dots M$ and $l=1,2,\dots N$, check the following condition:

$$|g_{kl}{|}^2>c/MN,$$

(13)

where $c>0$ is a preselected constant. According to derivations in [21], $c=1$, but in the same paper it is advocated to use $c=0.5$. From our simulation experiments, it follows that for moderate M and N $c=1.5$ is appropriate, while, for larger M, N, even larger constants are better. If for $g_{kl}$ condition (13) holds, then leave it in matrix G as a nonzero element. Otherwise, set $g_{kl}=0$ in this matrix. Set $\widehat{G}=G$. Notice that in this case matrix $\widehat{G}$ is of the same dimensions as G, but it has many zeros as its elements.

Selection of ${\Delta }_r$ and ${\Delta }_x$ (or equivalently M and N) as functions of the number of observations is also very important for the proper estimation. We give some hints on their choice in the section before last, where the asymptotic analysis is discussed.

The performance of Algorithm 1 is illustrated in the next section.

Algorithm 1 Estimation and learning algorithm.

Input: Raw observations $({\kappa }_l,r_l)$, $l=1,2,\dots ,d$.

Step 1: Select parameters M, N and $c>0$ in (13).

Step 2: Perform data preprocessing, as described in Section 3.1, in order to obtain matrix P.

Step 3: Calculate matrix $G=FF{T}_{2D}(P;M,N)$.

Step 4: Select elements of matrix $\widehat{G}$ either by selecting K and J, using cross-validation, or by the Kronmal-Tarter rule.

Step 5: Calculate matrix $\widehat{P}=FF{T}_{2D}^{-1}(P;K,J)$ when in Step 4 the cross-validation is used or as $\widehat{P}=FF{T}_{2D}^{-1}(P;M,N)$ when the Kronmal-Tarter rule is applied.

Output:$\widehat{P}$ is the output of the algorithm, if it suffices to have the estimates of $f(x;r)$ on the grid. If one needs the estimates of $f(x;r)$ at intermediate points, then calculate ${\widehat{\alpha }}_{kl}$’s from the corresponding elements of $\widehat{G}$ matrix (see Remark 2) and used them in (9).

4. Test on Synthetic Data

The first test can be made using synthetic data. These data are obtained from the family of probability density functions:

$$f_s(x;r)=N_{(\sqrt{r},1.)}\left(x\right),$$

(14)

where $N_{(\mu ,{\sigma }^2)}$ is normal distribution with mean $\mu =r_i$ and variation ${\sigma }^2=1$. The data are generated with 200 points in $r_i=0.,0.25,\dots ,50.$ and 300 random points for each $r_i$. Those data are binned in order to obtain the matrix of probabilities (11) with size $200\times 56$.

The similarities between two p.d.f.’s can be calculated using distances which are defined in many ways. Here, we would use Hellinger distance and Kullback–Leibler divergence.

For a specific r, the Hellinger distance is defined as follows:

$$H{\left(r\right)}^2=\frac{1}{2}\int {\left(\sqrt{f_s(x;r)}-\sqrt{\widehat{f}(x,r)}\right)}^{2}dx$$

(15)

Another integration along r is required in order to obtain distance for all r’s:

$$\frac{1}{R_2-R_1}{\int }_{r_1}^{r_2}{\int }_{-\infty }^{\infty }{\left(\sqrt{f_s(x;r)}-\sqrt{\widehat{f}(x,r)}\right)}^{2}dxdr.$$

(16)

The Kullback–Leibler divergence for a specific r can be defined as follows:

$$D_{KL}=\int f_s(x;r)(log{f}_s(x;r)-log\widehat{f}(x,r))dx.$$

(17)

Again, additional integration along r is required

$$\frac{1}{R_2-R_1}{\int }_{r_1}^{r_2}{f}_s(x;r)(log{f}_s(x;r)-log\widehat{f}(x,r))dxdr.$$

(18)

Due to inherent randomness, the calculations were carried out 100 times. The results are presented in

Table 1. Result of distance calculation between synthetic data and reconstruction.

Method	Mean	Standard Deviation
Hellinger	0.00007	$3.7\times {10}^{-6}$
Kullback–Leibler	0.00014	$5.9\times {10}^{-6}$

. Observe that Kullback–Leibler divergence is not symmetric. Its symmetrized version provides almost the same results as in Table 1.

The reconstruction using the presented algorithm can be seen in Figure 1.

5. Convergence of the Learning Procedure

In this section, we prove the convergence of the learning procedure in a simplified version similar to that described in Section 2.2, but without the discretization with respect to r. Then, we shall discuss additional conditions for the convergence of its discretized version.

Let $X\left(r\right),X_1\left(r\right),\dots ,X_n$ be independent, identically distributed random variables with parameter r, with p.d.f. $f(.,r)\in L^2\left([R_1,R_2]\right)$, where—for simplicity—we assume that n is the same for each r. Then, f has the representation

$$f(x,r)=\sum _{k=1}^{\infty }{a}_k\left(r\right)v_k\left(x\right),$$

(19)

which is convergent in the $L^2\left([S_1,S_2]\right)$ norm, where

$$a_k\left(r\right)={\int }_{S_1}^{S_2}{v}_k\left(x\right)f(x,r)dx=E_r\left(v_k\left(X\left(r\right)\right)\right).$$

(20)

Then, we estimate $a_k\left(r\right)$’s as follows:

$${\widehat{a}}_k\left(r\right)=\frac{1}{n}\sum _{i=1}^n{v}_k\left(X_i\left(r\right)\right).$$

(21)

Lemma 1.

For every $r\in [R_1,R_2]$ $\widehat{a_k}\left(r\right)$ is the unbiased estimator of $a_k\left(r\right)$.

Proof. 

Indeed,

$$E_r\left(\widehat{a_k}\left(r\right)\right)=\frac{1}{n}\sum _{i=1}^n{E}_r\left[v_k\left(X_i\left(r\right)\right)\right]=a_k\left(r\right).$$

(22)

 □

As an estimator of $f(.,r)$, we take the truncated version of (19):

$${\tilde{f}}_n(x,r)=\sum _{k=1}^{K\left(n\right)}{\widehat{a}}_k\left(r\right)v_k\left(x\right),$$

(23)

where the truncation point K depends on n. It may also depend on r, but, for the asymptotic analysis, we take this simpler version.

The standard way of assessing the distance between f and its estimator ${\tilde{f}}_n$ is the mean integrated squared error ($MISE$). Notice, however, that, in our case, the MISE additionally depends on r, which is further denoted as $MISE\left(r\right)$. Thus, in order to have a global error, we consider the integrated $MISE\left(r\right)$ that is defined as follows:

$$IMIS{E}_n={\int }_{R_1}^{R_2}MIS{E}_n\left(r\right)dr.$$

(24)

The $MISE\left(r\right)$ is defined as follows:

$$MIS{E}_n\left(r\right)=E_r{\int }_{S_1}^{S_2}{[f(x,r)-{\tilde{f}}_n(x,r)]}^{2}dx,$$

(25)

where the expectation w.r.t. $f(.,r)$ concerns all $X_i\left(r\right)$’s that are present in (23).

Using the orthonormality of $v_k$’s, we obtain:

$$MIS{E}_n\left(r\right)=E_r{\int }_{S_1}^{S_2}{[\sum _{k=1}^{K\left(n\right)}{(a_k\left(r\right)-{\widehat{a}}_k\left(r\right))}^2{v}_k\left(x\right)-\sum _{k=K\left(n\right)+1}^{\infty }{a}_k\left(r\right)v_k\left(x\right)]}^{2}dx$$

(26)

Continuing (26), we obtain for each $r\in [R_1,R_2]$:

$$MIS{E}_n\left(r\right)=\underset{Var(r,K(n\left)\right)}{\underbrace{E_r\left[\sum _{k=1}^{K\left(n\right)}{(a_k\left(r\right)-{\widehat{a}}_k\left(r\right))}^2\right]}}+\underset{Bia{s}^2(r,K\left(n\right))}{\underbrace{\sum _{k=K\left(n\right)+1}^{\infty }{a}_k{\left(r\right)}^2}}.$$

(27)

It is known that that for squared integrable f we have: ${\sum }_{k=1}^{\infty }{a}_k^2\left(r\right)<\infty$. Thus, if $K\left(n\right)\to \infty$ when $n\to \infty$, then for every $r\in [R_1,R_2]$ we obtain:

$$Bia{s}^2(r,K\left(n\right))=\sum _{k=K\left(n\right)+1}^{\infty }{a}_k^2\left(r\right)\to 0\mathrm{when}n\to \infty .$$

(28)

This result is not sufficient to prove the convergence of $IMIS{E}_n$ to zero as $n\to \infty$. To this end, we need a stronger result, namely an upper bound on $Bia{s}^2\left(rK\left(n\right)\right)$, which is independent of r.

Lemma 2.

Let us assume that $\frac{\partial }{\partial x}f(x,r)$ exists and it is a continuous function of x in $[S_1,S_2]$ for each $r\in [R_1,R_2]$. Furthermore, there exists constant $0<U<\infty$, which does not depend on r, and such that

$$\forall x\in [S_1,S_2]\forall r\in [R_1,R_2]\left|\frac{\partial }{\partial x}f(x,r)\right|\le U.$$

(29)

If $K\left(n\right)\to \infty$ when $n\to \infty$, then – for n sufficiently large we have:

$$\forall r\in [R_1,R_2]Bia{s}^2(r,K\left(n\right))\le \frac{U^2{(S_2-S_1)}^2}{K\left(n\right)}.$$

(30)

Proof. 

It is known that

$$|a_k\left(r\right)|\le k^{-1}{\int }_{S_1}^{S_2}\left|\frac{\partial }{\partial x}f(x,r)\right|dx\le k^{-1}U(S_2-S_1).$$

(31)

Thus, $Bia{s}^2\left(rK\left(n\right)\right)\le U^2{(S_2-S_1)}^2{\sum }_{k=K\left(n\right)}^{\infty }{k}^{-2}$, which finishes the proof, since it is known that for sufficiently large $K\left(n\right)$ we have ${\sum }_{k=K\left(n\right)}^{\infty }{k}^{-2}=K^{-1}\left(n\right)$.  □

For evaluating the variance component, we use Lemma 1:

$$Var(K\left(n\right),r)=\sum _{k=1}^{K\left(n\right)}{E}_r[{(a_k\left(r\right)-{\widehat{a}}_k\left(r\right))}^2]=\sum _{k=1}^{K\left(n\right)}\underset{Va{r}_r\left({\widehat{a}}_k\left(r\right)\right)}{\underbrace{E_r{({\widehat{a}}_k\left(r\right)-E_r{\widehat{a}}_k\left(r\right))}^2}},$$

(32)

where $Va{r}_r(.)$ is the variance of an r.v. having the p.d.f. $F(x,r)$.

Let us assume that the orthonormal and complete system $v_k$’s is uniformly bounded, i.e., there exists p being a non-negative integer and $C_p$ such that

$$\underset{x\in [S_1,S_2]}{sup}\left|v_k\left(x\right)\right|\le C_p{k}^p,k=1,2,\dots .$$

(33)

Notice that (33) holds for the trigonometric system with $p=0$

Lemma 3.

If (33) holds, then

$$Va{r}_r(K\left(n\right),r)\le C_p\frac{K^{2p+1}\left(n\right)}{n}$$

(34)

Proof. 

$X_i\left(r\right)$’s are i.i.d. and (33) holds. Thus,

$$Va{r}_r\left({\widehat{a}}_k\left(r\right)\right)=\frac{1}{n^2}\sum _{i=1}^{n}Va{r}_r\left(v_k\left(X_i\left(r\right)\right)\right)=\frac{1}{n}Va{r}_r\left(v_k\left(X_1\left(r\right)\right)\right)\le \frac{C_p^2{K}^{2p}\left(n\right)}{n}.$$

(35)

Using this fact in (32) finishes the proof.  □

Notice that the bound in (34) does not depend on r.

Theorem 1.

Let the assumptions of Lemmas 2 and 3 hold. If the sequence $K\left(n\right)$ is selected in such a way that the following conditions hold:

$$K\left(n\right)\to \infty \mathrm{and}\frac{K^{2p+1}\left(n\right)}{n}\to 0\mathrm{as}\mathrm{a}n\to \infty ,$$

(36)

then the estimation error $IMIS{E}_n\to 0$ as $n\to \infty$.

Proof. 

By Lemmas 2 and 3, we have uniform (in r) bounds for the variance and for the squared bias, respectively. Thus,

$$MIS{E}_n\left(r\right)\le C_p\frac{K^{2p+1}\left(n\right)}{n}+\frac{U^2{(S_2-S_1)}^2}{K\left(n\right)}.$$

(37)

Hence,

$$IMIS{E}_n\le (R_2-R_1)\left[C_p\frac{K^{2p+1}\left(n\right)}{n}+\frac{U^2{(S_2-S_1)}^2}{K\left(n\right)}\right],$$

(38)

which finishes the proof by applying (36).  □

Observe that for $v_k$’s being the trigonometric system we have $p=0$ and the r.h.s. of (38) is, roughly, of the form: $c_{1}K\left(n\right)/n+c_2/K\left(n\right)$, $c1>0$, $c_2>0$, which is minimized by $K\left(n\right)=c_3\sqrt{n}$ for a certain constant $c_3>0$. This implies that $IMIS{E}_n=O\left(n^{-1/2}\right)$.

6. Proposed Algorithm

Let us define $\left(x\right(r),r)$ as a generic par that for fixed r has p.d.f $f(x;r)$. We use semicolon; in order to indicate that r is a non-random variable.

Observations:

$$(X_i\left(r_i\right),r_i),i=1,2,\dots ,n$$

(39)

we admit that $X_i\left(r_i\right)$ and $X_j\left(r_j\right)$ are independent random variables even when $r_i=r_j$ for $i\ne j$. In general, r.v.’s in (39) are assumed to be mutually independent.

A family of p.d.f.’s is defined on $[{\kappa }_1,{\kappa }_2]\times [R_1,R_2]$ whre ${\kappa }_1<{\kappa }_2,R_1<R_2$ are real numbers. For each $r\in [R_1,R_2]f(x;r)$ is a p.d.f of a random variable $X\left(r\right)$.

Consider two orthinormal sequences $v_k\left(x\right),x\in [{\kappa }_1,{\kappa }_2]k=1,2,\dots$ and $T_j\left(r\right),r\in [R_1,R_2],j=1,2,\dots$ that are complete in $L_2\left([{\kappa }_1,{\kappa }_2]\right)$ and $L_2\left([R_1,R_2]\right)$, respectively. Then, $f(x;r)$ can be represented by the series (convergent in $L_2[{\kappa }_1,{\kappa }_2]$

$$a_k\left(r\right)={\int }_{{\kappa }_1}^{{\kappa }_2}{v}_k\left(x\right)f(x;r)dx,k=1,2,\dots$$

(40)

Notice that $a_k\left(r\right)$’s can be interpreted as

$$a_k\left(r\right)=E_r{v}_k\left(X\left(r\right)\right),$$

(41)

where $E_r$ stands for the expectations with respect to random variable $X\left(r\right)$ with p.d.f $f(x;r)$. Furthermore, each $a_k\left(r\right)$ can be represented by the series:

$$a_k\left(r\right)=\sum _{j=1}^{\infty }{\alpha }_{kj}{T}_j\left(r\right)$$

(42)

that is convergent in $L_2\left([R_1,R_2]\right)$ for constant coefficents ${\alpha }_{kj}$ defined as

$${\alpha }_{kj}={\int }_{R_1}^{R_2}{a}_k\left(r\right)T_j\left(r\right)dr,k,j=1,2,\dots$$

(43)

By the back substitution, we obtain the following series representation (in $L_2([{\kappa }_1,{\kappa }_2]\times [R_1,R_2])$)

$$f(x;r)=\sum _{k=1}^{\infty }\sum _{j=1}^{\infty }{\alpha }_{kj}{v}_k\left(x\right)T_j\left(r\right)$$

(44)

The simplest from the computational point of view, estimator $\widehat{f}(x;r)$ or the family $f(x;r),r\in [R_1,R_2]$ we obtain, when we additionally assume that the observations of $X\left(r\right)$’s are made on an equidistant grid ${\rho }_1<{\rho }_2<\dots <{\rho }_M$ that splits $[R_1,R_2]$ into nonoverlapping intervals of the length $\Delta >0$.

At each of ${\rho }_m,m=1,2,\dots M$, $n_m$ observations $(X_k\left({\rho }_m\right),{\rho }_m),l=1,2,\dots ,n_m$ are made, keeping the above-mentioned assumptions on mutual independence in force.

Then, one can estimate ${\alpha }_{kj}$’s by

$${\widehat{\alpha }}_{kj}=\Delta \sum _{m=1}^M{\widehat{a}}_k\left({\rho }_m\right)T_j\left({\rho }_m\right)$$

(45)

where

$${\widehat{a}}_k\left({\rho }_m\right)=\frac{1}{n_m}\sum _{l=1}^{n_m}{v}_k\left(x_l\left({\rho }_m\right)\right)$$

(46)

Estimators (46) are justified by (42). Notice that in (46) the simplest quadrature formula is used for approximating the integral ${\int }_{R_1}^{R_2}$. When an active experiment is applicable, then one can select ${\rho }_m$’s at nodes of more advanced quadratures with weights, in the same spirit as in [17] for nonparameteric regression estimators.

Truncating series (44) at K-th and J-t terms and substituting ${\widehat{\alpha }}_{kj}$ instead of ${\alpha }_{kj}$, we obtain the following estimator if the family $f(x;r),r\in [R_1,R_2]$

$$\widehat{f}(x;r)=\sum _{k=1}^K\sum _{j=1}^J{\widehat{\alpha }}_{kj}{v}_k\left(x\right)T_j\left(r\right).$$

(47)

If as $T_j\left(r\right)$ a trigonometric series is used on $[R_1,R_2]$, then ${\widehat{\alpha }}_{kj}$ can be calculated using FFT. In addition, $v_k$ can be trigonometric too.

7. Estimating Jet Engine Temperature Distributions

7.1. Subject Overview

A jet turbine is a well-known engine known for at least 90 years The typical application is that of an aircraft power-plant, but also in some ground applications. The main examples are firefighting and snow removal. In recent years, some companies have started to develop engines optimized for thermal power rather than thrust.

In a very simplistic view (see Figure 2), an already running turbine has one input parameter that is fuel flow. As outputs, we have its rotational speed (given in rpm—r) and temperature (in ${}^{\circ }$C—T). In ideal conditions, there should exist a simple relationship between T and r. Since many factors vary and not all of them can be directly measured then we can think about this relationship as probability, which puts us in the framework of the problem stated in the introduction.

7.2. Data Preparation

The orginal process data from the server have a form of JSON file of a database table. In this table, we are interested only in two columns, namely the turbine rotation speed (in rpm) and the turbine temperatue. Since the resulting temperature is dependent on many other factors, not all of them measured or even known, we treat the value as a random variable X and the rotation speed as known (so not random) r. The amount of relevant data are 71,268.

These two columns have to be changed into frequencies for each $r_i$. Groups are formed on a rectangular grid. An example of such a grid can be seen in Figure 3. The resulting number of samples in each box of the grid is shown in Figure 4. They should be converted into frequencies by simple divison. In order not to obscure the image, a 32 × 32 grid was used.

As a result, we obtain a matrix containing the observations near points on the grid ${\rho }_1,\dots ,{\rho }_m$

$$\left[\begin{array}{cccc}f(X_{11},{\rho }_1)& f(X_{21},{\rho }_1)& \dots & f(X_{n1},{\rho }_1)\\ f(X_{12},{\rho }_2)& f(X_{22},{\rho }_1)& \dots & f(X_{n2},{\rho }_2)\\ \dots & \dots & \dots & \dots \\ f(X_{1m},{\rho }_m)& f(X_{2m},{\rho }_m)& \dots & f(X_{nm},{\rho }_m)\end{array}\right]$$

(48)

7.3. The Estimation Process

From the matrix obtained in the previous section, a two-dimensional Discrete Fourier Transform is calculated using the FFT algorithm. The result is a matrix $F_{mn}$ of equal size but containing complex values. In literature regarding the Fourier transform, many properties can be found. A good example is book [22]. We use symmetry and antisymmetry of the resulting matrix. General inversion of the Fourier transform is defined in the following way:

$$f(i,j)=\frac{1}{\sqrt{m·n}}\sum _{k=0}^m\sum _{l=0}^n{F}_{k,l}exp(\mathbf{j}\frac{k·i}{m}·\frac{l·j}{n})$$

(49)

where ${\mathbf{j}}^2=-1$.

Equation (49) is defined only for the original points of the matrix (48). The continuous surface can be obtained by changing the therms $\frac{i}{m}\in [0,1],\frac{j}{n}\in [0,1]$ into $\frac{r}{max{r}_i}\in [0,1],\frac{T}{max{T}_i}\in [0,1]$. We cannot guarantee that between grid points the result would be real. The sensible solution is to use the absolute value of a possibly complex number.

We can ask the question of why use the FFT, if we reconstruct the same data again. The reason is smoothing the result. The reconstruction can use only a selected part of the spectrum—obviously lower frequencies. As mentioned before, they reside in the center of the matrix:

$$f(r,T)=\left|\frac{1}{\sqrt{m·n}}\sum _{k=K}^{m/2+K}{\gamma }_k\sum _{l=L}^{n/2+L}{F}_{k,l}exp(\mathbf{j}\frac{k·r}{r_{max}}·\frac{l·T}{T_{max}})\right|,$$

(50)

where ${\gamma }_k$ is the correction factor necessary for compensating for the omitted data. Its values should be selected so the reconstruction result is still p.d.f. and integrates to 1. Obviously, the size of part of the FFT matrix taken is $4·K·L$, the careful selection of $K,L$ is crucial. When those numbers are too small (see Figure 5), the result loses any resemblance to the original data (Figure 4). On the other hand, when the numbers are too large, this gives a detailed reconstruction (Figure 6) with all unnecessary details.

The acceptable reconstruction in Figure 7 is made with $K=16$ and $L=4$. It is obvious that smaller size means faster calculations.

The exact values can be obtained experimentally (as in this example) or by using some method like the Kronmal–Tarter rule.

Another heuristic approach is to reduce the abount of the entry points (probabilities) $f(X,\rho )$ by removing perimeter data. The result of such an approach can be seen in Figure 8. The removal of peripherial data results in reduction of over-fitting.

8. Possible Applications

8.1. Process Simulation

The simulation is an important tool in both design and then subsequent maintenance of the system, especially if physical device is cumbersome, costly, or dangerous to use. The obvious method of simulating the engine temperature at specific rotational speed is to generate random numbers from a distribution specific for that temperature. The simplest method is the so-called acceptance-rejection method. In this method, a random number from distribution $f\left(x\right)\in [a,b]$ is generated as follows (in general using any dense random number generator, but typically uniform is used):

• generate random $x\in [a,b]$,
• generate $y\in [0,1]$,
• if $y>f\left(x\right)$ then go to 1, otherwise the result is x.

We can easily obtain distribution for a specific parameter r. Using the example from Section 7.1, with r = 50,000 we obtain a distribution as in Figure 9.

If a random generation algorithm is used directly, the resulting process path would be highly irregular and perceived as not real. To avoid this, a simple filter can be used

$${\widehat{T}}_{i+1}=\tau ·\tilde{T}+(1-\tau ){\widehat{T}}_i,$$

(51)

where $\tilde{T}$ is the resulting new random number, ${\widehat{T}}_{i+1}$ is the new process value, and ${\widehat{T}}_i$ is the old process value. The result with parameter $\tau =0.05$ can be seen in Figure 10.

8.2. Process Diagnostics

From the resulting function $f(r,T)$, for specific r, we can calculate expected the value of $\overline{T}$ and also some specified interval where $\alpha \in [0,1]$ of realizations can be found—the equivalent of a confidence interval. If this parameter $\alpha$ is selected carefully, we can discern whether the actual pair $r,T$ form the process in or outside it. Other similar diagnostic methods can be seen in [23,24].

9. Convergence of the Learning Procedure

In this section, we prove the convergence of the learning procedure in a simplified version similar to that described in Section 2.2, but without the discretization with respect to r. Then, we shall discuss additional conditions for the convergence of its discretized version.

Let $X\left(r\right),X_1\left(r\right),\dots ,X_n$ be independent, identically distributed random variables with parameter r, with p.d.f. $f(.,r)\in L^2\left([R_1,R_2]\right)$, where—for simplicity—we assume that n is the same for each r. Then, f has the representation

$$f(x,r)=\sum _{k=1}^{\infty }{a}_k\left(r\right)v_k\left(x\right),$$

(52)

which is convergent in the $L^2\left([S_1,S_2]\right)$ norm, where

$$a_k\left(r\right)={\int }_{S_1}^{S_2}{v}_k\left(x\right)f(x,r)dx=E_r\left(v_k\left(X\left(r\right)\right)\right).$$

(53)

Then, we estimate $a_k\left(r\right)$’s as follows:

$${\widehat{a}}_k\left(r\right)=\frac{1}{n}\sum _{i=1}^n{v}_k\left(X_i\left(r\right)\right).$$

(54)

Lemma 4.

For every $r\in [R_1,R_2]$$\widehat{a_k}\left(r\right)$ is the unbiased estimator of $a_k\left(r\right)$.

Proof. 

Indeed,

$$E_r\left(\widehat{a_k}\left(r\right)\right)=\frac{1}{n}\sum _{i=1}^n{E}_r\left[v_k\left(X_i\left(r\right)\right)\right]=a_k\left(r\right).$$

(55)

 □

As an estimator of $f(.,r)$, we take the truncated version of (52):

$${\tilde{f}}_n(x,r)=\sum _{k=1}^{K\left(n\right)}{\widehat{a}}_k\left(r\right)v_k\left(x\right),$$

(56)

where the truncation point K depends on n. It may also depend on r, but, for the asymptotic analysis, we take this simpler version.

The standard way of assessing the distance between f and its estimator ${\tilde{f}}_n$ is the mean integrated squared error ($MISE$). Notice, however, that, in our case, the MISE additionally depends on r, which is further denoted as $MISE\left(r\right)$. Thus, in order to have a global error, we consider the integrated $MISE\left(r\right)$, which is defined as follows:

$$IMIS{E}_n={\int }_{R_1}^{R_2}MIS{E}_n\left(r\right)dr.$$

(57)

The $MISE\left(r\right)$ is defined as follows:

$$MIS{E}_n\left(r\right)=E_r{\int }_{S_1}^{S_2}{[f(x,r)-{\tilde{f}}_n(x,r)]}^{2}dx,$$

(58)

where the expectation w.r.t. $f(.,r)$ concerns all $X_i\left(r\right)$’s that are present in (56).

Using the orthonormality of $v_k$’s, we obtain:

$$MIS{E}_n\left(r\right)=E_r{\int }_{S_1}^{S_2}{[\sum _{k=1}^{K\left(n\right)}{(a_k\left(r\right)-{\widehat{a}}_k\left(r\right))}^2{v}_k\left(x\right)-\sum _{k=K\left(n\right)+1}^{\infty }{a}_k\left(r\right)v_k\left(x\right)]}^{2}dx$$

(59)

Continuing (59), we obtain for each $r\in [R_1,R_2]$:

$$MIS{E}_n\left(r\right)=\underset{Var(r,K(n\left)\right)}{\underbrace{E_r\left[\sum _{k=1}^{K\left(n\right)}{(a_k\left(r\right)-{\widehat{a}}_k\left(r\right))}^2\right]}}+\underset{Bia{s}^2(r,K\left(n\right))}{\underbrace{\sum _{k=K\left(n\right)+1}^{\infty }{a}_k{\left(r\right)}^2}}.$$

(60)

It is known that that for squared integrable f we have: ${\sum }_{k=1}^{\infty }{a}_k^2\left(r\right)<\infty$. Thus, if $K\left(n\right)\to \infty$ when $n\to \infty$, then for every $r\in [R_1,R_2]$, we obtain:

$$Bia{s}^2(r,K\left(n\right))=\sum _{k=K\left(n\right)+1}^{\infty }{a}_k^2\left(r\right)\to 0\mathrm{when}n\to \infty .$$

(61)

This result is not sufficient to prove the convergence of $IMIS{E}_n$ to zero as $n\to \infty$. To this end, we need a stronger result, namely an upper bound on $Bia{s}^2\left(rK\left(n\right)\right)$, which is independent of r.

Lemma 5.

Let us assume that $\frac{\partial }{\partial x}f(x,r)$ exists and it is a continuous function of x in $[S_1,S_2]$ for each $r\in [R_1,R_2]$. Furthermore, there exists constant $0<U<\infty$, which does not depend on r, and such that

$$\forall x\in [S_1,S_2]\forall r\in [R_1,R_2]\left|\frac{\partial }{\partial x}f(x,r)\right|\le U.$$

(62)

If $K\left(n\right)\to \infty$ when $n\to \infty$, then—for n sufficiently large, we have:

$$\forall r\in [R_1,R_2]Bia{s}^2(r,K\left(n\right))\le \frac{U^2{(S_2-S_1)}^2}{K\left(n\right)}.$$

(63)

Proof. 

It is known that

$$|a_k\left(r\right)|\le k^{-1}{\int }_{S_1}^{S_2}\left|\frac{\partial }{\partial x}f(x,r)\right|dx\le k^{-1}U(S_2-S_1).$$

(64)

Thus, $Bia{s}^2\left(rK\left(n\right)\right)\le U^2{(S_2-S_1)}^2{\sum }_{k=K\left(n\right)}^{\infty }{k}^{-2}$, which finishes the proof, since it is known that for sufficiently large $K\left(n\right)$ we have ${\sum }_{k=K\left(n\right)}^{\infty }{k}^{-2}=K^{-1}\left(n\right)$.  □

For evaluating the variance component, we use Lemma 1:

$$Var(K\left(n\right),r)=\sum _{k=1}^{K\left(n\right)}{E}_r[{(a_k\left(r\right)-{\widehat{a}}_k\left(r\right))}^2]=\sum _{k=1}^{K\left(n\right)}\underset{Va{r}_r\left({\widehat{a}}_k\left(r\right)\right)}{\underbrace{E_r{({\widehat{a}}_k\left(r\right)-E_r{\widehat{a}}_k\left(r\right))}^2}},$$

(65)

where $Va{r}_r(.)$ is the variance of a r.v. having the p.d.f. $F(x,r)$.

Let us assume that the orthonormal and complete system $v_k$’s is uniformly bounded, i.e., there exists p being a nonnegative integer and $C_p$ such that

$$\underset{x\in [S_1,S_2]}{sup}\left|v_k\left(x\right)\right|\le C_p{k}^p,k=1,2,\dots .$$

(66)

Notice that (66) holds for the trigonometric system with $p=0$

Lemma 6.

If (66) holds, then

$$Va{r}_r(K\left(n\right),r)\le C_p\frac{K^{2p+1}\left(n\right)}{n}$$

(67)

Proof. 

$X_i\left(r\right)$’s are i.i.d. and (66) holds. Thus,

$$Va{r}_r\left({\widehat{a}}_k\left(r\right)\right)=\frac{1}{n^2}\sum _{i=1}^{n}Va{r}_r\left(v_k\left(X_i\left(r\right)\right)\right)=\frac{1}{n}Va{r}_r\left(v_k\left(X_1\left(r\right)\right)\right)\le \frac{C_p^2{K}^{2p}\left(n\right)}{n}.$$

(68)

Using this fact in (65) finishes the proof.  □

Notice that the bound in (67) does not depend on r.

Theorem 2.

Let the assumptions of Lemmas 5 and 6 hold. If the sequence $K\left(n\right)$ is selected in such a way that the following conditions hold:

$$K\left(n\right)\to \infty \mathrm{and}\frac{K^{2p+1}\left(n\right)}{n}\to 0\mathrm{as}\mathrm{a}n\to \infty ,$$

(69)

then the estimation error $IMIS{E}_n\to 0$ as $n\to \infty$.

Proof. 

By Lemmas 2 and 6, we have uniform (in r) bounds for the variance and for the squared bias, respectively. Thus,

$$MIS{E}_n\left(r\right)\le C_p\frac{K^{2p+1}\left(n\right)}{n}+\frac{U^2{(S_2-S_1)}^2}{K\left(n\right)}.$$

(70)

Hence,

$$IMIS{E}_n\le (R_2-R_1)\left[C_p\frac{K^{2p+1}\left(n\right)}{n}+\frac{U^2{(S_2-S_1)}^2}{K\left(n\right)}\right],$$

(71)

which finishes the proof by applying (69).  □

Observe that for $v_k$’s being the trigonometric system we have $p=0$ and the r.h.s. of (71) is, roughly, of the form: $c_{1}K\left(n\right)/n+c_2/K\left(n\right)$, $c1>0$, $c_2>0$, which is minimized by $K\left(n\right)=c_3\sqrt{n}$ for a certain constant $c_3>0$. This implies that $IMIS{E}_n=O\left(n^{-1/2}\right)$. Notice that this rate is known (see [25]) to be the best possible when a bivariate, continuously differentiable regression function is estimated by any nonparametric method.

However, this convergence rate was obtained under an idealized assumption that we have observations of $X_j\left(r\right)$’s for each r. Further discussion of this topic is outside the scope of this paper. We only mention that in [17] it was proved that orthogonal series estimators of p.d.f.’s retain consistency under mild assumptions concerning grouped observations. In particular, in the notation of Section 3, bin width ${\Delta }_x$ should depend on the number of observations d as follows: ${\Delta }_x\left(d\right)\to 0$ as $d\to \infty$, but in such a way that ${\Delta }_x^2\left(d\right)$ multiplied by the qube power of the number of spanning orthogonal terms should also approach zero. One can expect that, for the trigonometric series, ${\Delta }_r$ should also fulfill similar conditions.

10. Conclusions

In this paper, a method for the estimation of families of density functions is presented along with an efficient learning algorithm. It gives insight into the relation between probability density function and external factors that influence it. The method was used in a real-world problem to simulate and diagnose a jet turbine.

Additional, significant possible applications include the estimation of quality indexes for decision-making and optimal control, especially for repetitive and/or spatially distributed processes (see [26,27] for most recent contributions).

From the formal point of view, the method presented can easily be extended into a multidimensional case. Namely, if r is multivariate $\overline{r}=[r^{\left(1\right)},r^{\left(2\right)},\dots ,r^{\left(d\right)}]$, say, then it suffices to replace orthogonal system $T_j\left(r\right)$, $j=1,2,\dots$ by the tensor product of $T_j$’s, i.e., by all possible products of the following form:

$$T_{j_1}(r^{\left(1\right)})T_{j_2}(r^{\left(2\right)})·\dots ·T_{j_d}(r^{\left(d\right)}),$$

(72)

where $j_i$ takes all the values in $\{1,2,\dots \}$, $i=1,2,\dots ,d$. As a consequence, one has to replace all the sums over j by the multiple sums over $j_i$’s. When $T_{j_i}$’s form the trigonometric system, then one can apply the multidimensional $FFT$ algorithm. Clearly, a much larger number of observations is also necessary for a reliable estimation, but a family of estimated p.d.f.’s $\widehat{f}(x,\overline{r})$ provides much more information than a regression function of $\overline{r}$.