Second-Moment/Order Approximations by Kernel Smoothers with Application to Volatility Estimation

Abstract

Volatility estimation and quantile regression are relevant active research areas in statistics, machine learning and econometrics. In this work, we propose two procedures to estimate the local variances in generic regression problems by using kernel smoothers. The proposed schemes can be applied in multidimensional scenarios (not just for time series analysis) and easily in a multi-output framework as well. Moreover, they enable the possibility of providing uncertainty estimation using a generic kernel smoother technique. Several numerical experiments show the benefits of the proposed methods, even compared with the benchmark techniques. One of these experiments involves a real dataset analysis.

1. Introduction

Regression analysis can be considered as a set of methodologies for estimating the relationships between a dependent variable y (often called an ‘output’ or ‘response’) and a vector $\mathbf{x}$ of independent variables (often called an ‘input’). Essentially, the main goal in a regression problem is to obtain an approximation of the trend defined as the expected value of y given $\mathbf{x}$, i.e., $E\left[y\right|\mathbf{x}]$; that is, the first (non-central) moment of the conditional density $p\left(y\right|\mathbf{x})$. We can assert that the most complete regression problem consists of approximating the whole conditional density $p\left(y\right|\mathbf{x})$, whereas the simplest task in a regression problem consists of estimating only the first moment $E\left[y\right|\mathbf{x}]={\int }_{\mathcal{Y}}yp\left(y\right|\mathbf{x})dy$. Intermediate scenarios appear in different applications, where other moments (higher than one) are of interest and hence are also approximated.

Volatility estimation (intended as local variance or local standard deviation) and quantile regression analysis are currently important tasks in statistics, machine learning and econometrics. The problem of volatility estimation has particular relevance in financial time series analysis, where the volatility represents the degree of variation in a trading price series over time, usually measured by the standard deviation of logarithmic returns. Indeed, it is important to remark that an inherent problem in the volatility estimation is that the conditional variance is latent and hence is not directly observable when real data are analyzed. It is still an important active area of research [1,2,3,4].

In the literature, several schemes have been proposed. A well-known family among the proposed approaches is formed by the generalized autoregressive conditional heteroskedasticity (GARCH) models [5,6]. They are considered the benchmark for time series analysis. Other important classes of techniques are the stochastic volatility models [7] ([8] Section 7.4) and the exponentially weighted moving average models [9]. They have also been extended for multivariate scenarios, but the application is much more complex than the method that we propose here. Moreover, other criticisms have been detected in the literature: as stated by some authors [6,10,11], most of the latent volatility models fail to satisfactorily describe several stylized facts that are observed in financial time series. Several schemes are based on state-space models ([8] Section 7.4). Some important review papers on this topic can be found in the following [12,13,14,15]. In these relevant works, the authors have also focused on the theoretical foundations of the estimators that have been proposed recently.

Other approaches have been considered. For instance, the local linear regression methods (which can resemble a kernel smoother approach) have already been proposed for volatility estimation [16,17,18]. They share the notion of “locality” with the kernel smoother approach and perform several linear regressions (applied to the residual errors for estimating the variance). Generally, for all these schemes, the extensions for the multivariate and/or multi-output cases are not straightforward. Moreover, most of them are parametric methods (here, we consider non-parametric techniques where the complexity of the model is related to the number of observed data). In mathematical finance and financial engineering, another class of methods is formed by the so-called local volatility models, which are generalizations of the Black–Scholes model. Local volatility models are similar and related to stochastic volatility models, where the instantaneous volatility is itself a random variable [19]. Other authors have studied the high-frequency features of the data, which contain market microstructure noise, in order to estimate the instantaneous volatility (e.g., [20]). Quantile regression models study the relationship between an independent input vector $\mathbf{x}$ and some specific quantile or moment of the output variable y [21,22,23]. Therefore, in quantile regression analysis, the goal is to estimate the higher-order moments of the response/output variable given an input $\mathbf{x}$ [21,22,23,24,25,26].

On the other hand, it is important to remark that many advanced and benchmark regression methods in the literature (such as Gaussian processes, etc.) consider a constant variance as the initial assumption of the output given the input, $\mathrm{var}\left[y\right|\mathbf{x}]={\sigma }_e^2$; i.e., it does vary with the input variable $\mathbf{x}$ [27,28].

In this work, we consider an extended approach (with respect to the assumptions used in classical regression methods) where $\mathrm{var}\left[y\right|\mathbf{x}]=v(\mathbf{x})$; i.e., the local variance varies with the input $\mathbf{x}$. More precisely, in this work, we propose two procedures for estimating the local variance in a regression problem and using a kernel smoother approach ([28] Section 9) [29]. Note that the kernel smoother schemes contain several well-known techniques as special cases, such as the fixed radius nearest neighbors approach, as an example [30]. The resulting solution is the non-parametric method; hence, both the complexity and flexibility of the solution increase with the number of data N. This ensures having the adequate flexibility to analyze the data.

The first proposed method is based on the Nadaraya–Watson derivation of a linear kernel smoother [27,28]. The second proposed approach draws inspiration from divisive normalization, a function grounded in the activity of brain neurons [31]. This function aims to standardize the neuron activity by dividing it by the activity of the neighboring neurons. It has demonstrated favorable statistical properties [32,33,34] and has been utilized in various applications [35,36]. Other important and related schemes, containing relevant theoretical results, can be found in the literature [37,38,39].

The proposed methods can be applied to time series analysis and/or in more general regression problems where the input variables are multidimensional, and with correlated noise. Therefore, our approach can be implemented in spatial statistical modeling and any other inverse problems [23,40]. There are many datasets with multidimensional inputs (e.g., including space and time variables) whose structures involve a substantial change in variance. The decomposition of the seismic data is just an example. Furthermore, another advantage of the proposed scenario is that the generalization for multi-output scenarios can be easily designed. More generally, the proposed approach is easy to apply and also easy to extend and generalize in terms of flexibility: different regression techniques or different kernel functions can be considered (possibly different for trend and variance estimation), and the number of hyper-parameters can be decided by the user.

From another point of view, this work provides another relevant contribution. The proposed schemes enable performing uncertainty estimation with a generic kernel smoother technique. Indeed, a generic kernel smoother method is not generally supported by a generative probabilistic derivation that also yields an uncertainty estimation. The Gaussian processes (GPs) and relevance vector machines (RVMs) regression methods are relevant well-known and virtually unique exceptions [28,41].

We have tested the proposed methods in five numerical experiments. Four of them involve the application of the different schemes to time series analysis and comparison with GARCH models, which are considered benchmark techniques for volatility estimation in time series [42]. The last numerical experiment addresses a more general regression problem, with a multidimensional input variable $\mathbf{x}$ of dimension 122, considering a real (emo-soundscapes) database [43,44].

The paper is structured as follows. The main methodology employed here for regression, based on kernel smoothers, is described in Section 2. Several features and relevant characteristics are also discussed. The two proposed variance estimation procedures are introduced in Section 3. The different extensions, generalizations, and variants are provided in Section 4. Numerous numerical experiments covering different scenarios (and with several comparisons) are provided in Section 5. Finally, Section 6 is devoted to the conclusions and some possible ideas for future work.

2. Approximating the Trend

Let us consider a set of N data pairs, such as ${\{{\mathbf{x}}_i,y_i\}}_{i=1}^N$, where ${\mathbf{x}}_i\in {\mathbb{R}}^D$, with $D\ge 1$ and $y_i\in \mathbb{R}$. First of all, we are interested in obtaining a regression function (a.k.a., “local mean” trend), i.e., removing the noise in the signal obtaining an estimator $\widehat{f}\left(\mathbf{x}\right)$ for all possible $\mathbf{x}$ in the input space. One possibility is to employ a linear kernel smoother. More specifically, we consider the Nadaraya–Watson estimator [27,28] that has the following form,

$$E\left[y\right|\mathbf{x}]\approx \widehat{f}\left(\mathbf{x}\right)=\sum _{n=1}^N\frac{h(\mathbf{x},{\mathbf{x}}_n)}{{\sum }_{j=1}^{N}h(\mathbf{x},{\mathbf{x}}_j)}{y}_n=\sum _{n=1}^N\phi (\mathbf{x},{\mathbf{x}}_n)y_n,$$

(1)

where $\phi (\mathbf{x},{\mathbf{x}}_n)=\frac{h(\mathbf{x},{\mathbf{x}}_n)}{{\sum }_{j=1}^{N}h(\mathbf{x},{\mathbf{x}}_j)}$ and $h(\mathbf{x},\mathbf{z})$ is a function (which is often called ‘kernel’) decided by the user and defined in ${\mathbb{R}}^D\times {\mathbb{R}}^D$. Note that, by this definition, the nonlinear weights $\phi (\mathbf{x},{\mathbf{x}}_n)$ are normalized, i.e.,

$$\sum _{n=1}^N\phi (\mathbf{x},{\mathbf{x}}_n)=1,\forall \mathbf{x}.$$

(2)

As an example, we could consider

$$h(\mathbf{x},\mathbf{z})=h(\mathbf{x},\mathbf{z}|\lambda )=exp\left(-\frac{\left|\right|\mathbf{x}-{\mathbf{z}\left|\right|}^2}{\lambda }\right),$$

where $\lambda$ is a hyper-parameter that should be tuned. Clearly, we also have

$$\phi (\mathbf{x},\mathbf{z})=\phi (\mathbf{x},\mathbf{z}|\lambda ).$$

The form of this estimator above is quite general and contains other different well-known methods as special cases. For instance, it contains the k-nearest neighbors algorithm (kNN) for regression as a specific case (to be more specific, the fixed-radius near neighbors algorithm). Indeed, with a specific choice of $h(\mathbf{x},{\mathbf{x}}_n|\lambda )$ (as a rectangular function), the expression above can represent the fixed-radius near neighbors estimator [28,30].

Remark 1. 

The resulting regression function is a flexible non-parametric method. Both complexity and flexibility of the solution grow with the number of data N since, at each new data point ${\mathbf{x}}_{N+1}$, it corresponds to the use of an additional kernel function $\phi (\mathbf{x},{\mathbf{x}}_{N+1})$ located at ${\mathbf{x}}_{N+1}$, and the final solution $\widehat{f}$ will be a linear combination of $N+1$ values.

Remark 2. 

The characteristics of the solution are determined by the choice of the kernel, which is decided by the user. For instance, the choice of a Gaussian kernel function, $h(\mathbf{x},{\mathbf{x}}_n)$, generates a smooth infinitely differentiable solution. The choice of a Laplacian kernel function yields a solution that is not differentiable at all the inputs ‘${\mathbf{x}}_n$’s in the dataset.

Remark 3. 

The input variables ${\mathbf{x}}_n\in {\mathbb{R}}^D$ are vectors ($D\ge 1$) in general. Therefore, the described methods have much wider applicability than the techniques that can be employed only for time series (where the time index is a scalar number, $x=t\in \mathbb{R}$). Clearly, the methodologies described here can also be employed for analyzing time series. Moreover, even in a time series framework, we can obtain a prediction between two consecutive time instants. For instance, if we have a time series with daily data, with a kernel smoother, we can obtain a prediction at each hour (or minute) within a specific day.

Remark 4. 

It is important to remark that the application to multidimensional input case, ${\mathbf{x}}_n\in {\mathbb{R}}^D$, is in general fast and straightforward, unlike for other local piecewise polynomial solutions (e.g., [17,18]), which would require the construction of multidimensional and expensive grid (to build and evaluate the piecewise regressor properly). Moreover, the proposed framework contains these piecewise solutions as special cases, choosing a kernel function that is non-zero just in a subset of the support (e.g., a rectangular kernel).

• Learning $\mathit{\lambda }$. One possibility for tuning the hyper-parameters of the kernel functions is to use a cross-validation (CV) procedure. In this work, we have employed a leave-one-out cross-validation (LOO-CV) [27].

3. Variance Estimation Procedures

Let us assume that we have already computed the trend (a.k.a., “local mean”), i.e., the regression function $\widehat{f}\left(\mathbf{x}\right)$. Here, we present two methods to obtain an estimation of the local variance (or volatility) at each point $\mathbf{x}$, which is theoretically defined as

$$\begin{array}{cc}\hfill v\left(\mathbf{x}\right)=\mathrm{var}\left[y\right|\mathbf{x}]& ={\int }_{\mathcal{Y}}{(y-E\left[y\right|\mathbf{x}])}^{2}p\left(y\right|\mathbf{x})dy,\hfill \\ \hfill & =E\left[y^2\right|\mathbf{x}]-{\left(E\left[y\right|\mathbf{x}]\right)}^2.\hfill \end{array}$$

(3)

METHOD-1 (M1). If the weights $\phi (\mathbf{x},{\mathbf{x}}_n)$ are adequate for linearly combining the outputs $y_i$ and obtaining an proper approximation of $E\left[y\right|\mathbf{x}]$, one can extend this idea for approximating the second non-central moment $E\left[y^2\right|x]$ as

$$\begin{array}{c}\hfill E\left[y^2\right|\mathbf{x}]\approx \widehat{q}\left(\mathbf{x}\right)=\sum _{n=1}^N\phi (\mathbf{x},{\mathbf{x}}_n)y_n^2.\end{array}$$

(4)

hence

$$\begin{array}{c}\hfill \widehat{v}\left(\mathbf{x}\right)=\widehat{q}\left(\mathbf{x}\right)-{\left(\widehat{f}\left(\mathbf{x}\right)\right)}^2\approx \mathrm{var}\left[y\right|\mathbf{x}],\end{array}$$

(5)

which is an estimator of the instant variance. Note that $\widehat{q\left(\mathbf{x}\right)}$ and $\widehat{f}\left(\mathbf{x}\right)$ are obtained with the same weights

$$\phi (\mathbf{x},{\mathbf{x}}_n)=\phi (\mathbf{x},{\mathbf{x}}_n|\lambda ),$$

with the same value of $\lambda$ (obtained using a LOO-CV procedure). Thus, a variant of this procedure consists of learning another value of ${\lambda }_2$, i.e., obtaining other coefficients ${\phi }_2(\mathbf{x},\mathbf{z})={\phi }_2(\mathbf{x},\mathbf{z}|{\lambda }_2)$, in the Equation (4) considering the signal $y_n^2$ (instead of $y_n$).

• METHOD-2 (M2). Let us define the signal obtained as the estimated square errors

$$v_n={\left(y_n-\widehat{f}\left({\mathbf{x}}_n\right)\right)}^2,n=1,...,N.$$

If $\widehat{f}\left(\mathbf{x}\right)\approx E\left[y\right|\mathbf{x}]$ as assumed, $v_n$ is a one-sample estimate of the variance at ${\mathbf{x}}_n$. Then, the goal is to approximate the trend of this new signal (i.e., new output) $v_n$,

$$\begin{array}{cc}\hfill \widehat{v}\left(\mathbf{x}\right)& =\sum _{n=1}^N{\phi }_2(\mathbf{x},{\mathbf{x}}_n)v_n,\hfill \end{array}$$

(6)

where we consider another parameter ${\lambda }_2$, i.e.,

$${\phi }_2(\mathbf{x},\mathbf{z})={\phi }_2(\mathbf{x},\mathbf{z}|{\lambda }_2),$$

that is tuned again with LOO-CV but considering the new signal $v_n$. Note that $\widehat{v}\left(\mathbf{x}\right)$ can be interpreted as estimation of the instant variance of the underlying signal. As an alternative, completely different kernel functions (as ${\phi }_2$) can also be applied (that differ in the functional form with respect to $\phi$ instead of just for the choice regarding $\lambda$).

Remark 5. 

Again, as for estimating the trend, note that the two procedures above can be applied for multivariate inputs ${\mathbf{x}}_n\in {\mathbb{R}}^D$ and not just for scalar inputs (as in the time series).

Remark 6. 

If one divides $\mathbf{x}$ by $\widehat{v}\left(\mathbf{x}\right)$, we obtain a signal with uniform local variance; i.e., we have removed the (possible) heteroscedasticity. In [34], this procedure was used to define the relation kernels in the divisive normalization and thus equalize the energy of neuron responses locally.

4. Extensions and Variants

This section is devoted to describing several generalizations and variants of the ideas provided in the previous sections.

4.1. Use of Generic Regression Methods

The use of linear kernel smoothers as regression methods is not mandatory. Indeed, the ideas previously described can be employed even applying different regression methods. Below, we provide some general steps in the same fashion of a pseudo-code, In order to clarify the application of possible different regression techniques,

Choose a regression method. Obtain a trend function $\widehat{f}\left(\mathbf{x}\right)\approx E\left[y\right|\mathbf{x}]$ given the dataset ${\{{\mathbf{x}}_i,y_i\}}_{i=1}^N$. Choose one method for estimating the instance variance (between the two below): M1. Choose a regression method (the same as in the previous step or a different one). Consider the dataset ${\{{\mathbf{x}}_i,y_i^2\}}_{i=1}^N$ and obtain $\widehat{q}\left(\mathbf{x}\right)\approx E\left[y^2\right|\mathbf{x}]$. Then, compute $\widehat{v}\left(\mathbf{x}\right)=\widehat{q}\left(\mathbf{x}\right)-{\left(\widehat{f}\left(\mathbf{x}\right)\right)}^2$. M2. Choose a regression method (the same as in the previous step, i.e., for the trend, or a different one). Consider the dataset ${\{{\mathbf{x}}_i,v_i\}}_{i=1}^N$ where $v_i={(y_i-\widehat{f}\left({\mathbf{x}}_i\right))}^2$, and obtain the function $\widehat{v}\left(\mathbf{x}\right)$.

Note that the regression techniques in step 1 and step 2 above can be different.

4.2. Estimation of the Covariance between Two Generic Inputs

In this section, we describe how to approximate the covariance between two generic inputs $\mathbf{x}$ and $\mathbf{z}$, extending the previous proposed procedures. For the sake of simplicity, we just adapt method M2 in order to estimate the covariance $\widehat{c}(\mathbf{x},\mathbf{z})$ between two generic inputs as

$$\begin{array}{cc}\hfill \widehat{c}(\mathbf{x},\mathbf{z})& =\sum _{n=1}^N\sum _{i=1}^N\frac{h(\mathbf{x},{\mathbf{x}}_n)g(\mathbf{z},{\mathbf{x}}_i)}{{\sum }_{j=1}^N{\sum }_{k=1}^{N}h(\mathbf{x},{\mathbf{x}}_j)g(\mathbf{z},{\mathbf{x}}_k)}\left(y_n-\widehat{f}\left({\mathbf{x}}_n\right)\right)\left(y_i-\widehat{f}\left({\mathbf{x}}_i\right)\right),\hfill \\ \hfill & =\sum _{n=1}^N\sum _{i=1}^N\phi (\mathbf{x},\mathbf{z},{\mathbf{x}}_n,{\mathbf{x}}_i)\left(y_n-\widehat{f}\left({\mathbf{x}}_n\right)\right)\left(y_i-\widehat{f}\left({\mathbf{x}}_i\right)\right),\hfill \end{array}$$

(7)

where

$$\phi (\mathbf{x},\mathbf{z},{\mathbf{x}}_n,{\mathbf{x}}_i)=\frac{h(\mathbf{x},{\mathbf{x}}_n)g(\mathbf{z},{\mathbf{x}}_i)}{{\sum }_{j=1}^N{\sum }_{k=1}^{N}h(\mathbf{x},{\mathbf{x}}_j)g(\mathbf{z},{\mathbf{x}}_k)},$$

is a bivariate normalized kernel (and h and g two generic kernel functions). Note that $\left(y_n-\widehat{f}\left({\mathbf{x}}_n\right)\right)\left(y_i-\widehat{f}\left({\mathbf{x}}_i\right)\right)$ is a one-sample estimator of the covariance between ${\mathbf{x}}_n$ and ${\mathbf{x}}_i$. Similar ideas have been introduced in [45]. Hence, we can also estimate the correlation within the noise in the dataset. This is of interest in different applications [23,29,46].

Remark 7. 

This section perhaps clarifies how the kernel estimation approach works in general, for possibly estimating different moments and quantities. The idea is to use one-sample estimators involving the data points ${\{{\mathbf{x}}_n,y_n\}}_{n=1}^N$ as ‘anchors’ and, for extending the results at generic inputs, then use linear combinations of these values, weighting them according to the distance from the generic inputs to these anchor points.

4.3. Multi-Output Scenario and Other Extensions

In a multi-output framework, we have N data pairs, such as ${\{{\mathbf{x}}_i,{\mathbf{y}}_i\}}_{i=1}^N$, where ${\mathbf{x}}_i\in {\mathbb{R}}^D$, but, in this case, the outputs are also vectors ${\mathbf{y}}_i\in {\mathbb{R}}^{D_Y}$. Hence, for the local trend, we also have a vectorial hidden function $\widehat{\mathbf{f}}\left(\mathbf{x}\right):{\mathbb{R}}^D\to {\mathbb{R}}^{D_Y}$ for all possible $\mathbf{x}$ in the input space. Then, we could easily write

$$\widehat{\mathbf{f}}\left(\mathbf{x}\right)=\sum _{n=1}^N\phi (\mathbf{x},{\mathbf{x}}_n){\mathbf{y}}_n.$$

(8)

Regarding the estimations of local variances, follow the same procedures, defining the following vectorial quantities: $\widehat{\mathbf{q}}\left(\mathbf{x}\right)\approx E\left[{\mathbf{y}}^2\right|\mathbf{x}]$ for M1, $\widehat{\mathbf{v}}\left(\mathbf{x}\right)=\widehat{\mathbf{q}}\left(\mathbf{x}\right)-{\left(\widehat{\mathbf{f}}\left(\mathbf{x}\right)\right)}^2$ and ${\mathbf{v}}_n={({\mathbf{y}}_n-\widehat{\mathbf{f}}\left({\mathbf{x}}_n\right))}^2$ for M2.

• One $\mathit{\lambda }$ for each data point. Furthermore, we could also consider different local hyper-parameters $\lambda$, for instance ${\lambda }_n$ with $n=1,...,N$, or, more generally, with $\lambda =\lambda \left(\mathbf{x}\right)$. In this case, we would have different coefficient functions ${\phi }_n(\mathbf{x},{\mathbf{x}}_n)={\phi }_n(\mathbf{x},{\mathbf{x}}_n|{\lambda }_n)$ (one for each input ${\mathbf{x}}_n$), so that trend could be easily expressed as

  $$\widehat{\mathbf{f}}\left(\mathbf{x}\right)=\sum _{n=1}^N{\phi }_n(\mathbf{x},{\mathbf{x}}_n){\mathbf{y}}_n.$$

  (9)

4.4. Adding More Flexibility to the Starting Model

The starting model in Equation (1) is already a flexible non-parametric method. Moreover, if we consider again Equation (9) but in a single output framework, we have

$$\widehat{f}\left(\mathbf{x}\right)=\sum _{n=1}^N{\phi }_n(\mathbf{x},{\mathbf{x}}_n)y_n,with{\phi }_n(\mathbf{x},{\mathbf{x}}_n)={\phi }_n(\mathbf{x},{\mathbf{x}}_n|{\lambda }_n).$$

Clearly, this is even a much more flexible model than Equation (1), with N hyper-parameters to tune (where N is the number of data; namely, the number of hyper-parameters grows with the number of data). Clearly, intermediate scenarios can be designed where we have a unique kernel function depending on two hyper-parameters,

$$\phi (\mathbf{x},{\mathbf{x}}_n)=\phi (\mathbf{x},{\mathbf{x}}_n|{\lambda }_1,{\lambda }_2),$$

or, more generally, on P hyper-parameters, $\phi (\mathbf{x},{\mathbf{x}}_n)=\phi (\mathbf{x},{\mathbf{x}}_n|{\lambda }_1,{\lambda }_2,...,{\lambda }_P)$. It is important to remark that, clearly, all these ideas can also be employed in the estimation of the variance (not just for the trend).

4.5. Joint Use of a Rectangular Kernel and a Least Squares Solution for Time Series Analysis

Let us consider a time series framework; i.e., the input is a scalar time instance, $x=t\in \mathbb{R}$, and then the dataset is formed by the following pairs ${\{t_i,y_i\}}_{i=1}^N$. Let us also consider that the intervals, $t_i-t_{i-1}$, are constant; in this case, we can skip the t index and consider the dataset ${\{t_i,y_i\}}_{i=1}^N$. In this scenario, as an alternative, one can use an autoregressive (AR) model,

$$\begin{array}{c}\hfill y_i=a_1{y}_{i-1}+a_2{y}_{i-2}+...+a_W{y}_{i-W}+{ϵ}_y,\end{array}$$

(10)

where ${ϵ}_y$ is a noise perturbation and the coefficients $a_i$, for $i=1,...,W$, are obtained by a least squares (LSs) minimization and the length of the temporal window W (i.e., the order of the AR filter) is obtained using a spectral information criterion (SIC) [47,48]. Note that temporal window W corresponds to the use of a rectangular kernel in a kernel smoother approach (but this scenario only considers the past samples; see the first ‘Remark’ below). Hence, so far, this variant could be considered as a special case of the generic technique in Section 2. However, here, coefficients that mix the past samples are obtained by an LS procedure (not all equals to the same value). Then, considering the estimated coefficients ${\widehat{a}}_i$ by LS, and $\widehat{W}$ by SIC, we have

$$\begin{array}{c}\hfill \widehat{f}\left(t_i\right)={\widehat{f}}_i={\widehat{a}}_1{y}_{i-1}+{\widehat{a}}_2{y}_{i-2}+...+{\widehat{a}}_{\widehat{W}}{y}_{i-\widehat{W}}.\end{array}$$

(11)

Just as an example, in order to apply M2 for estimating the instant variance, we can set $v_i={(y_i-{\widehat{f}}_i)}^2$ and assume another AR model over $v_i$ with window length now denoted as H,

$$\begin{array}{c}\hfill v_i=b_1{v}_{i-1}+b_2{v}_{i-2}+...+b_H{v}_{i-H}+{ϵ}_v,\end{array}$$

(12)

where again the estimations ${\widehat{b}}_i$ can be obtained by LS, and $\widehat{H}$ by SIC. The resulting instant variance function is

$$\begin{array}{c}\hfill v\left(t_i\right)=v_i={\widehat{b}}_1{v}_{i-1}+{\widehat{b}}_2{v}_{i-2}+...+{\widehat{b}}_{\widehat{H}}{v}_{i-\widehat{H}}.\end{array}$$

(13)

Clearly, the application of M1 could be performed in a similar way.

Remark 8. 

The resulting estimators in this section are still a linear combination of (a portion) of the outputs. Therefore, we still have linear smoothers (or better linear filters since only considering combinations of past samples). However, note that in Equations (11)–(13) the coefficients of the linear combinations are obtained by least squares (LSs) method. Hence, we have a window of length W (and then H), but this differs from the use of a rectangular kernel function for two reasons: (a) the window only considers past samples; (b) the coefficients are not all equal (to the ratio 1 over the number of samples included in the window), but they are tuned by LS.

5. Numerical Experiments

5.1. Applications to Time Series

In this section, we consider 5 different numerical experiments with different (true) generating models. Here, we focus on time series ($x=t\in \mathbb{R}$) for two main reasons: (a) first of all, for the sake of simplicity (we can easily show a realization of the data in one plot), (b) and, last but not least, we have relevant benchmark competitors, such as the GARCH models [42,49,50], that we can also test. Note that GARCH models have been specifically designed for handling volatility in time series.

More specifically, in this section, we test our methodologies and different GARCH models in four easy reproducible examples. These four examples differ for the data generating model. Indeed, the data are generated as the sum of a mean function, $f\left(t\right)$, and an error term, $s\left(t\right)\epsilon \left(t\right)$, i.e.,

$$y\left(t\right)=f\left(t\right)+s\left(t\right)\epsilon \left(t\right),\epsilon \left(t\right)\sim \mathcal{N}(0,1).$$

(14)

where we have used the fact that we are analyzing time series, i.e., $x=t\in \mathbb{R}$. The functions $f\left(t\right)$ and $s\left(t\right)$ are deterministic and represent the trend and the standard deviation of the observations $y\left(t\right)$, whereas $\epsilon \left(t\right)\sim \mathcal{N}(0,1)$ represent a standard Gaussian random noise. Note also that $v\left(t\right)=s{\left(t\right)}^2$. In order to keep the notation of the rest of the paper, note that the dataset ${\{t_n,y_n\}}_{n=1}^N$ will be formed by the N pairs,

$$x_n=t_n,y_n=y\left(t_n\right),$$

where $y_n$ will be generated according to the model above. The four experiments consider the following:

1.

$f\left(t\right)=sin\left(0.5t\right)$, $s\left(t\right)=0.2$.

2.

$f\left(t\right)=t$, $s\left(t\right)=2\left|t\right|$.

3.

$f\left(t\right)=0.5t^2$, $s\left(t\right)=5sin\left(0.3t\right)$.

4.

$f\left(t\right)=0$ and $s\left(t\right)$ generated by a GARCH model.

In all cases, the generated data $y\left(t\right)$ and the underlying standard deviation $s\left(t\right)$ (or variance $v\left(t\right)=s{\left(t\right)}^2$) are known; hence, we can compute errors and evaluate the performance of the used algorithms. For the application of the proposed methods, we consider a nonlinear kernel function as

$$\begin{array}{c}\hfill h(t,t_n)=exp\left(-\frac{{(t-t_n)}^{\alpha }}{\lambda }\right),\end{array}$$

(15)

with $\alpha \in \{1,2,10\}$. The value $\alpha =1$ is the minimum possible value if we desire to have a suitable $L_p$ distance (with $p=\alpha$), satisfying all the properties of a distance. The choice of $\alpha =2$ is just in order to consider the typical Euclidean distance. For larger values of $\alpha$, we set $\alpha =10$, as well as for avoiding possible numerical problems. The parameter $\lambda$ is tuned by LOO-CV. We test 7 different combinations, each one mixing a specific trend approximation method with a specific variance estimation technique:

• Based on kernel smoothers: we consider different regression methods based on Equations (1) and (15). Furthermore, for the variance estimation, we consider both procedures M1 and M2 described in Section 3. More specifically,

  -

  KS-1-1: $\alpha =1$ and M1;

  -

  KS-1-2: $\alpha =1$ and M2;

  -

  KS-2-1: $\alpha =2$ and M1;

  -

  KS-2-2: $\alpha =2$ and M2;

  -

  KS-10-1: $\alpha =10$ and M1;

  -

  KS-10-2: $\alpha =10$ and M2.

• Based on AR models: we apply the method described in Section 4.5, which employs two auto-regressive models, one for the trend and one for the variance, jointly with procedure M2.

Table 1. Proposed methods (specific implementations) tested in the numerical experiments.

Method	KS-1-1	KS-1-2	KS-2-1	KS-2-2	KS-10-1	KS-10-2	AR-Based
For trend estimation	Equations (1)–(12), $\alpha =1$		Equations (1)–(12), $\alpha =2$		Equations (1)–(12), $\alpha =10$		AR—Section 4.5
For variance estimation	M1	M2	M1	M2	M1	M2	AR and M2—Section 4.5

summarizes all seven considered specific implementations. Moreover, we test several GARCH(p,q) models: GARCH(1,1), GARCH(2,2), GARCH(5,5) and GARCH(10,10) (using a Matlab implementation of GARCH models).

Remark 9. 

The GARCH models work directly with the signal without trend, i.e., $|y_n-{\widehat{f}}_n|$. In order to have a fair comparison, we use a kernel smoother with $\alpha =2$, i.e., applying Equations (1)–(15) with $\alpha =2$ (as in the KS-2 methods).

We recall that GARCH models have been specifically designed for making inferences in heteroscedastic time series.

5.1.1. Experiment 1

As previously stated, in this first example, we consider

$$\begin{array}{c}\hfill y\left(t\right)=sin\left(0.5t\right)+0.2\epsilon \left(t\right),\epsilon \left(t\right)\sim \mathcal{N}(0,1),\end{array}$$

i.e., where $f\left(t\right)=sin\left(0.5t\right)$ and $s\left(t\right)=0.2$ (in this case, we have homoscedastic series). We generate $N={10}^4$ data with $t\in [-10,10]$. We apply all the algorithms, compute the mean absolute error (MAE) and repeat and average the results over ${10}^4$ independent runs. Figure 1a depicts one realization of the generated data with the model above.

• Results. Figure 1b,c provide the results in this example. Figure 1b shows the MAE obtained in estimating the trend, and Figure 1c depicts the MAE in estimating the standard deviation. In this case, regarding the trend, we see that the AR-based has an MAE higher than the rest when estimating the trend, but the other MAEs are quite similar. Similar outcomes are also obtained for approximating with the standard deviation, although the smaller MAE values are provided by the GARCH models. Regarding the estimating of the standard deviation, the M1 seems to provide better results than M2 in this example.

5.1.2. Experiment 2

In this second example, we consider

$$\begin{array}{c}\hfill y\left(t\right)=t+2\left|t\right|\epsilon \left(t\right),\epsilon \left(t\right)\sim \mathcal{N}(0,1),\end{array}$$

where $f\left(t\right)=t$ and $s\left(t\right)=2\left|t\right|$ (i.e., in this case, the heteroscedastic scenario). As previously, we generate $N={10}^4$ data with $t\in [-10,10]$. We apply all the algorithms, compute the mean absolute error (MAE) and again we repeat and average the results over ${10}^4$ independent runs. Figure 2a depicts one realization of the generated data according to the model above.

• Results. The results are presented in Figure 2b,c. Again, the MAE for the AR-based method seems to be higher than for the rest of the algorithms when estimating the trend and standard deviation of the time series. Regarding the estimation of the standard deviation, the best results are provided by the kernel methods, which use the M2 (the MAE is particularly small with $\alpha =10$). Thus, it is remarkable that, even in this heteroscedastic scenario, the proposed methods provide similar, and, in same cases, better, results than GARCH models.

5.1.3. Experiment 3

In this third example, we consider a standard deviation that varies periodically, and a second-order polynomial as a trend function, i.e.,

$$\begin{array}{c}\hfill y\left(t\right)=0.5t^2+5sin\left(0.3t\right)\epsilon \left(t\right),\epsilon \left(t\right)\sim \mathcal{N}(0,1),\end{array}$$

where $f\left(t\right)=0.5t^2$ and $s\left(x\right)=5sin\left(0.3t\right)$. Again, we have heterostodastic scenario. As previously, we generate $N={10}^4$ data with $t\in [-10,10]$. We compute the mean absolute error (MAE), and again we repeat and average the results over ${10}^4$ independent runs. Figure 3a depicts one realization of the generated data according to the described model.

• Results. Figure 3b,c depict the boxplots of the MAE in estimating the trend and the standard deviation. As with the rest of the examples, the AR-based provides the worst results when calculating the trend of the time series, while the rest of the algorithms have very similar error values (recall that the GARCH models employ the KS-2 method for the trend estimation). However, the AR-based method provides good performance in the approximation of the standard deviation. The AR-based technique uses M2 for the variance. Moreover, for the standard deviation estimation, the methods KS-1-1, KS-2-1 and KS-10-1 provide smaller errors. Hence, it seems that in this case M1 works better even than GARCH models.

5.1.4. Experiment 4

Let us consider

$$\begin{array}{cc}\hfill y\left(t\right)& =s\left(t\right)\epsilon \left(t\right),\epsilon \left(t\right)\sim \mathcal{N}(0,1),\hfill \\ \hfill y_t& =s_t{ϵ}_t=z_t,\hfill \end{array}$$

i.e., $f\left(t\right)=f_t=0$, and we have denoted $z_t=s_t{ϵ}_t$. The process for the standard deviation $s_t=\sqrt{v_t}$ will be a GARCH($P,Q$), i.e.,

$$\begin{array}{cc}\hfill v_t=& \kappa +{\gamma }_1{v}_{t-1}+\dots +{\gamma }_P{v}_{t-P}+\hfill \\ \hfill & +{\alpha }_1{z}_{t-1}^2+\dots +{\alpha }_Q{z}_{t-Q}^2.\hfill \end{array}$$

In this example, we generate the variance from a GARCH($2,3$)

$$\begin{array}{cc}\hfill v_t=& 2+0.2·v_{t-1}+0.1·v_{t-2}+0.2·z_{t-1}^2+\hfill \\ \hfill & +0.2·z_{t-2}^2+0.2·z_{t-3}^2\hfill \end{array}$$

(16)

Clearly, again, it is a heterostodastic scenario. We generate $N={10}^4$ data and apply the different techniques. We compute the mean absolute error (MAE), and we repeat and average the results over ${10}^4$ independent runs. One realization of the generated data is provided in Figure 4a. Note that the trend is linear, but the volatility presents very fast changes since it generated by a GARCH(2,3) model (see blue line in Figure 4a).

• Results. The results for this example are presented in Figure 4b,c. The best performance in the estimation of $s\left(t\right)$ is obtained by KS-1-2 and all the GARCH models (as expected in this case). The AR-based and KS-2-2 also provided close values of MAE. In this example, clearly, M2 outperforms M1. We remark that, even in this scenario that is particularly favorable for the GARCH models, all the proposed methods provide competitive results. The highest (hence worst) MAEs values are provided by the KS with $\alpha =10$, which is a quite extreme tuning/choice of this parameter, especially when the volatility follows an autoregressive rule. Indeed, a Laplacian kernel with $\alpha =1$ is more adequate in this framework. For further considerations, see ([28] Section 10).

5.1.5. Final Comments about the Applications to Time Series

All the proposed methods based on kernel smoothers (KS) and M1 and M2 always provide competitive results (with closer or smaller MAE values) with respect to the benchemark algorithms, such as GARCH methods, which have been specifically designed for the estimation of the standard deviation in time series.

These numerical experiments show that the proposed M1 and M2 can even outperform GARCH models regarding the estimation of the standard deviation in time series. The GARCH models seem to provide more robust results when the signal has been generated truly by a GARCH model in Section 5.1.4 and, surprisingly, in the homoscedasticscenario in Section 5.1.1. The methods based on kernel smoothers virtually outperform the AR-based approach due to the fact that they incorporate the “future samples” in their estimators. A rectangular window, including both past and future samples, should improve the performance of the AR-based method.

5.2. Application in Higher Input Dimension: Real Data on Soundscape Emotions

GARCH models have been specifically designed for estimating the volatility in time series; hence, $x=t\in \mathbb{R}$. However, the methods proposed in this work can be applied to problems with multidimensional inputs $\mathbf{x}\in {\mathbb{R}}^D$ with $D\ge 1$.

More specifically, here, we focus on analyzing a soundscape emotion database. In the last decade, soundscapes have become one of the most active topics in acoustics. In fact, the number of related research projects and scientific articles grows exponentially [51,52,53]. In urban planning and environmental acoustics, the general scheme consists of (a) soundscapes recording, (b) computing of acoustic and psychoacoustic indicators of the signals, (c) including other context indicators (e.g., visual information [54]) and (d) ranking of soundscapes’ audio signals employing emotional descriptors. Finally, the model can be developed [40].

Here, we consider the emo-soundscapes database (EMO) [43,44], where $N=1200$ and $D=122$, i.e., $\mathbf{x}\in {\mathbb{R}}^{122}$, which is the largest available soundscape database with annotations of emotion labels, and also the most recent up to now [43,44]. Among the 122 audio features that form the inputs $\mathbf{x}$, we can distingush three main groups of features of the audio signals: psychoacoustic features, time-domain features and frequency-domain features. The output y that we consider is called arousal. For further information, the complete database can be found at https://metacreation.net/emo-soundscapes/(accessed on 1 January 2024). We again employ a kernel function of type

$$h(\mathbf{x},\mathbf{z})=h(\mathbf{x},\mathbf{z}|\lambda )=exp\left(-\frac{\left|\right|\mathbf{x}-{\mathbf{z}\left|\right|}^{\alpha }}{\lambda }\right),$$

with $\alpha =2$. Therefore, considering the EMO database ${\{{\mathbf{x}}_i,y_i\}}_{i=1}^{1200}$, we apply M1 and M2 and CV-LOO for tuning $\lambda$. The MAE in estimating the trend is $0.0388$, obtained with an optimal parameter ${\lambda }^{*}=0.12$. Moreover, in both cases, we find a sensible increase in the volatility of the output in the last half of the samples, for i from 621 to 1213. This could be an interesting result, which should be discussed with experts in the field and regarding the EMO database.

6. Conclusions

In this work, we have proposed two methods for estimating the instant/local variance in regression problems based on kernel smoothers. From another point of view, this work can also be viewed as a way to perform quantile regression (at least a second-order quantile regression) using kernel smoothers.

With respect to the other procedures noted in the literature, the proposed methods have much wider applicability than the other techniques since they can be employed only for a time series, such as the well-known GARCH models (where the time index is a scalar number, $x=t\in \mathbb{R}$). Indeed, the proposed schemes can be employed with multidimensional input variables ${\mathbf{x}}_n\in {\mathbb{R}}^D$. Moreover, the proposed techniques can also be applied to time series ($D=1$) and can also obtain a prediction between two consecutive time instants (i.e., with a higher resolution than the received data). More generally, the estimation of the local variance can occur regarding any of the training data ${\mathbf{x}}_i$ but also regarding any generic test input $\mathbf{x}$.

Furthermore, analyzing the time series, the numerical simulations have shown that the proposed schemes provide competitive results even with respect to GARCH models (which are specifically designed for analyzing time series). Applications regarding real data and multidimensional inputs (with dimension 122) have also been provided. As a future research direction, we plan to design kernel smoother functions with local parameters $\lambda$, or more generally with $\lambda =\lambda \left(\mathbf{x}\right)$, to better handle the heteroscedasticity. Moreover, applications regarding multi-output scenarios can be designed and considered.