On the Application of the Stability Methods to Time Series Data

Abstract

The important problem of selecting the predictors in a high-dimensional case where the number of candidates is larger than the sample size is often solved by the researchers from the signal processing community using the orthogonal matching pursuit algorithm or other greedy algorithms. In this work, we show how the same problem can be solved by applying methods based on the concept of stability. Even if it is not a new concept, the stability is less known in the signal processing community. We illustrate the use of stability by presenting a relatively new algorithm from this family. As part of this presentation, we conduct a simulation study to investigate the effect of various parameters on the performance of the algorithm. Additionally, we compare the stability-based method with more than eighty variants of five different greedy algorithms in an experiment with air pollution data. The comparison demonstrates that the use of stability leads to promising results in the high-dimensional case.

1. Introduction

1.1. Motivation

According to the presentation from [1], which considers the evolution of the statistical inference over the past decades, one of the prominent research topics after 1990 was high-dimensional statistical modeling, where the sample size (n) is much smaller than the number of covariates (p). This problem is difficult because the large value of p makes the total number of candidate variables impractically large [2].

For the sake of concreteness, let us assume that we possess the response vector $\mathit{y}={[y_1\cdots y_n]}^{\top }$, where ${(·)}^{\top }$ denotes transposition, and the matrix $\mathit{X}=[{\mathit{x}}_1\cdots {\mathit{x}}_p]$ of p potential predictors. In many cases, the vector $\mathit{y}$ and the columns of $\mathit{X}$ are centered; the columns of $\mathit{X}$ are normalized so that the Euclidean norm is the same for all of them. In signal processing, the matrix $\mathit{X}$ is called a dictionary and the columns of the matrix $\mathit{X}$ are dubbed atoms. We wish to represent $\mathit{y}$ as a sparse linear combination of the columns of $\mathit{X}$, i.e., $\mathit{y}=\mathit{X}\widehat{\mathit{\beta }}$, where most of the entries of the vector $\widehat{\mathit{\beta }}$ are equal to zero. It is evident that the residual sum of squares (RSS) is given by $\parallel \mathit{y}-\mathit{X}\widehat{\mathit{\beta }}{\parallel }_2^2$, where ${\parallel ·\parallel }_2$ is the symbol for the Euclidean norm. However, if $p=100$ and we want vector $\widehat{\mathit{\beta }}$ to have exactly five non-zero entries, the naive approach that considers all possible subsets of five atoms for selecting the best one is totally impractical because the number of subsets that should be evaluated is ${\displaystyle \left(\genfrac{}{}{0pt}{}{100}{5}\right)=75,287,520}$.

1.2. Background and Related Works

1.2.1. Greedy Algorithms

A possible solution can be obtained in the following way. First, construct a sequence of linear models and then choose the best model from this sequence using either cross-validation or an information theoretic (IT) criterion. This strategy relies on greedy algorithms. One of the greedy algorithms often employed in signal processing is the Matching Pursuit Algorithm (MPA) [3]. According to Scopus, the term “matching pursuit algorithm” appears in the title/abstract/keywords of 257 documents that have been published in journals/conference proceedings/book chapters that contain in their titles the term “signal processing”. In MPA, $\widehat{\mathit{\beta }}$ is initialized with the vector of length p whose entries are equal to zero, hence the initial value of RSS equals ${\parallel \mathit{y}\parallel }_2^2$. At each iteration of MPA, the column of $\mathit{X}$ is selected, which leads to the largest reduction of RSS. The entry of $\widehat{\mathit{\beta }}$ that corresponds to the selected column of $\mathit{X}$ is updated, whereas all other entries of $\widehat{\mathit{\beta }}$ remain unchanged. The total number of iterations is large and gives the total number of competing models [4]. The selection of the best model by IT criteria is not straightforward because the criteria for Gaussian linear regression should be altered before applying them to the models generated by MPA. The interested reader can find in [5] a list of 22 IT criteria that are suitable for MPA as well as more technical details about the algorithm itself.

A variant of MPA commonly employed in signal processing is the Orthogonal Matching Pursuit (OMP) [6,7]. An indication of the fact that OMP is more popular than MPA in the signal processing community is provided by the number of Scopus documents found when, in the search mentioned above, we replace “matching pursuit algorithm” with “orthogonal matching pursuit”. This time the result is 592, which is clearly greater than the result produced by the previous search. The technique used by OMP is the same as in MPA, in the sense that, at each iteration, the column of $\mathit{X}$ that leads to the largest reduction of RSS is selected. The main difference is that all the entries of $\widehat{\mathit{\beta }}$ that correspond to the columns of $\mathit{X}$ selected at the current and past iterations are updated by least squares, hence the IT criteria for Gaussian linear regression can be used. For example, in [8], 12 different IT criteria have been employed for selecting the best model from the candidates yielded by OMP.

In the same reference, three other greedy algorithms are presented: Relaxed Matching Pursuit (RMP) [4,7], Frank-Wolfe Algorithm (FWA) [9] and Constrained Matching Pursuit (CMP) [4]. These algorithms are applied less frequently to signal processing problems. Performing a search for “Frank-Wolfe” on Scopus, where the other settings are the same as in the searches made before, returns only 35 documents. At the same time, it is remarkable that FWA and CMP are solvers for the Lasso problem [10], which can be formulated as a penalized estimation with an ${\ell }_1$-penalty. For more details, see [4,11] and the references therein. Note that five algorithms for the Lasso problem are reviewed in [12]. Our search on Scopus (with the settings above) finds “lasso” in 514 documents, which demonstrates that Lasso is extensively used in signal processing research.

1.2.2. Stability Methods

It is known that one of the disadvantages of Lasso is the lack of control over the selection of false or irrelevant variables (see [13] for a discussion on this topic). An option for controlling the false discovery proportion is to consider the stability of the selection under subsampling. In connection with this approach, it was proposed in [14] that instead of applying Lasso to the whole data set of size n, Lasso is applied repeatedly to subsets of size $n/2$, and a variable chosen frequently when running the experiments is deemed to be relevant. For simplicity, we assume that n is even. More importantly, a variable is included in the model only if the empirical probability of being selected in the experiments is greater than a fixed threshold, which is chosen by the practitioner. The major difficulty is that the experiments should be executed for all ${\displaystyle \left(\genfrac{}{}{0pt}{}{n}{n/2}\right)}$ subsets of size $n/2$. Another variant for stability-based variable selection was introduced in [15]. An important difference between the method from [15], which is called Complementary Pairs Stability Selection (CPSS), and the one from [14] is that CPSS does not consider in each experiment a subset of size $n/2$, but a pair of subsets of size $n/2$ whose intersection is the empty set. It is not needed to run experiments for all the pairs with these properties; it is enough to execute the subsampling B times, where B is a tuning parameter. The Lasso selection is applied to each subset in the pair and the statistics concerning how many times a particular variable is selected are computed by taking into account the selection results obtained for each subset, in each experiment. The decision of including a variable in the model is based on the comparison of the computed statistic with a threshold. More interestingly, instead of a bound on false discovery proportion, CPSS asserts the bounds in the following terms: (i) “the expected number of variables chosen by CPSS that have low selection probability under the base selection procedure” and (ii) “the expected number of high selection probability variables that are excluded by CPSS”. In the case that is of interest for us, the base selection procedure is Lasso, but it is evident that the stability methods from [14,15] can be applied to other selection procedures as well. As a continuation of the series of statistics from Scopus that we have presented above, we mention that the reference [15] is cited 171 times on Scopus, but there is no citation in journals/conference proceedings/book chapters that contain in their titles the term “signal processing”. A possible explanation might be that CPSS as well as the stability method from [14] were designed for independent and identically distributed data. It seems that the only work in which CPSS was altered to be suitable for time series is [16]. The key point of the modification of CPSS proposed in [16] consists in sampling from data blocks that are ‘almost’ independent.

According to [17], the stability has been employed in statistical inference for a long time; it was applied not only in the cases where the data perturbation was produced by subsampling, but it was also used in conjunction with other perturbation schemes as jackknife or bootstrap. For instance, in the signal processing literature, the stability was proven to be instrumental in finding the number of groups when performing data clustering (see, for example [18]). A comprehensive analysis of the stability-based methods for clustering that have been proposed during the last decades can be found in [19]. More recently, the stability was used for the identification of differential equations from noisy spatio-temporal data [20]. The results of extensive simulation studies that evaluate the capabilities of the stability methods for high-dimensional biomedical data were reported in [21,22]. The aim of the study conducted in [21] was to compare the abilities of four base selection procedures to correctly identify the true predictors in artificial data when the stability criterion is applied. The results have shown that Lasso can lead to modest results when there is correlation among the significant variables. We should mention that the study did not use the stability methods from [15,16].

1.3. Organization of the Paper and the Main Contributions

In this work, we make an attempt to introduce the stability-based selection of predictors to the signal processing community:

• We consider the algorithm from [16]. The algorithm is conceptually simple and can be regarded as a modification of the method from [15], which does not seem to be currently used in signal processing.
• We relate this algorithm to the problems of interest in multivariate signal processing by applying the algorithm to the selection of predictors in the case of a vector autoregressive (VAR) model [23]. Scopus indicates that vector autoregressive can be found in the title/abstract/keywords of 359 documents published in signal processing venues. As we are interested in estimating an entry of a time series by employing the past observations and (if available) the current observations collected for other time series (see [5] and the references therein), we use a variant of the VAR model, which is called vector autoregressive with exogenous variables (VARX) [16,23]. In Section 2, we justify why the sparse VARX model is appropriate and show how the main algorithm, with Lasso as the base selection procedure, can be applied to find the relevant predictors.
• We conduct experiments with simulated data in order to evaluate the influence of various parameters on the performance of the main algorithm. As the ground truth is known, the performance is evaluated by measuring the feature selection capabilities in terms of the true positive rate and the false positive rate. We also discuss a modified variant of the main algorithm (see Section 3).
• We compare the performance of the stability-based method with the performance of methods that rely on greedy algorithms and IT criteria/cross-validation. The comparison involves more than eighty methods and it is carried out by using air pollution data that were measured in Auckland, New Zealand. As the ’true’ predictors are not known for the real-life data, the comparison of various methods is made by considering the prediction accuracy. Additionally, we analyze the predictors that are selected most often and give an interpretation based on what is known from the environmental chemistry (see Section 4).

Section 5 concludes the paper.

2. Main Algorithm

2.1. Notation

We use bold letters for both vectors and matrices. In particular, $\mathit{I}$ is the identity matrix of appropriate size. The symbol $\mathit{v}\left(i\right)$ denotes the ith entry of an arbitrary vector $\mathit{v}$. The notation for the number of non-zero entries of $\left|\right|\mathit{v}\left|\right|$ is ${\left|\right|v\left|\right|}_0$ and $\mathrm{size}\left(\mathit{v}\right)$ denotes the total number of entries of $\mathit{v}$. The symbol ${\left|\right|\mathit{v}\left|\right|}_{\eta }$ denotes the ${\ell }_{\eta }$-norm, where $\eta \in \{1,2\}$. In the case of an arbitrary matrix $\mathit{M}$, $\mathit{M}(i,j)$ is the entry located at the intersection of the ith row and the jth column. For a set of positive integers J, the symbol $\mathit{M}(J,:)$ denotes the rows of $\mathit{M}$ whose indexes are given by the elements of J and $\mathit{M}(:,J)$ stands for the columns of $\mathit{M}$ whose indexes are given by the elements of J. The operator for transposition is ${(·)}^{\top }$. For any set A, ${\mathbb{1}}_A(·)$ is the indicator function, which has the property that ${\mathbb{1}}_A\left(a\right)=1$ if $a\in A$ and ${\mathbb{1}}_A\left(a\right)=0$ otherwise. The cardinality of A is denoted $\left|A\right|$.

2.2. Problem Formulation and the Lasso Solution

Assume that the time series data ${\mathit{y}}_1,\dots ,{\mathit{y}}_T$, or equivalently ${\left\{{\mathit{y}}_t\right\}}_{t=1}^T$, are available. For explaining how the stability methods can be applied to the time series data, we resort to the well-known VARX model [23]:

$$\begin{array}{c}\hfill {\mathit{y}}_t=\sum _{i=1}^{p_y}{\mathit{A}}_i{\mathit{y}}_{t-i}+\mathit{C}{\mathit{v}}_t+{\mathit{w}}_t,\end{array}$$

(1)

where ${\mathit{y}}_t,{\mathit{w}}_t\in {\mathbb{R}}^{K_y\times 1}$, ${\mathit{A}}_i\in {\mathbb{R}}^{K_y\times K_y}$ for $i\in \{1,\dots ,p_y\}$, $\mathit{C}\in {\mathbb{R}}^{K_y\times p_v}$ and ${\mathit{v}}_t\in {\mathbb{R}}^{p_v\times 1}$. For simplicity of the presentation, we suppose that the data have been padded, hence the identity above holds true for all $t\in \{1,\dots ,T\}$. Note that the model in (1) can be easily extended such that to consider not only the observations ${\mathit{v}}_t$, but also past measurements of the exogenous variables, ${\mathit{v}}_{t-1},{\mathit{v}}_{t-2},\dots$ The vectors ${\left\{{\mathit{w}}_t\right\}}_{t=1}^T$ are independent and identically distributed, and they are drawn from a $K_y$-variate Gaussian distribution with zero mean vector and covariance matrix ${\sigma }_{\nu }^2\mathit{I}$, where the value of ${\sigma }_{\nu }^2$ is unknown.

The identity in (1) can be written more compactly as follows:

$$\begin{array}{c}\hfill \mathit{Y}=\mathit{H}\mathit{Z}+\mathit{W},\end{array}$$

(2)

where $\mathit{Y}=\left[{\mathit{y}}_1\cdots {\mathit{y}}_T\right]$ and $\mathit{H}=\left[{\mathit{A}}_1\cdots {\mathit{A}}_{p_y}\mathit{C}\right]$.

With the convention that ${\mathit{z}}_t={\left[{\mathit{y}}_{t-1}^{\top }\cdots {\mathit{y}}_{t-p_y}^{\top }{\mathit{v}}_t^{\top }\right]}^{\top }$ for all $t\in \{1,\dots ,T\}$, we have $\mathit{Z}=\left[{\mathit{z}}_1\cdots {\mathit{z}}_T\right]$. Additionally, $\mathit{W}=\left[{\mathit{w}}_1\cdots {\mathit{w}}_T\right]$. Observe that $\mathit{Y}\in {\mathbb{R}}^{K_y\times T}$, $\mathit{H}\in {\mathbb{R}}^{K_y\times (K_y{p}_y+p_v)}$, $\mathit{Z}\in {\mathbb{R}}^{(K_y{p}_y+p_v)\times T}$ and $\mathit{W}\in {\mathbb{R}}^{K_y\times T}$.

In many practical situations, the matrix $\mathit{H}$ is assumed to be sparse. Some of the reasons for the presence of zeros in $\mathit{H}$ are:

• In general, the order of the autoregressions is not known and $p_y$ is taken to be an upper bound for the unknown order. It is expected that the estimated order is smaller than this upper bound.
• An important result in the analysis and forecasting of multivariate time series claims that, for some $a,b\in \{1,\dots ,K_y\}$, $y_b$ does not Granger-cause $y_a$ if and only if the entry indexed by $(a,b)$ is zero for all matrix coefficients ${\mathit{A}}_i$, where $i\in \{1,\dots ,p_y\}$ [23]. We note in passing that, there is an increasing interest in novel methods for the identification of vector autoregressive models with Granger and stability constraints (see [24] and the references therein). However, in the literature that is focused on this particular identification problem, the term ‘stability’ refers to the following condition that should be satisfied by the matrix coefficients: The magnitudes of all the eigenvalues of the matrix

  $$\begin{array}{c}\hfill \left[\begin{array}{ccccc}{\mathit{A}}_1& {\mathit{A}}_2& \dots & {\mathit{A}}_{p_y-1}& {\mathit{A}}_{p_y}\\ \mathit{I}& 0& \dots & 0& 0\\ 0& \mathit{I}& \dots & 0& 0\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& \dots & \mathit{I}& 0\end{array}\right]\end{array}$$

  (3)

  are strictly less than one [23].
• There might be exogenous variables in ${\mathit{v}}_t$ that do not influence some of the entries of ${\mathit{y}}_t$.

In our presentation, we consider the problem of selecting the ‘best’ predictors for the first component of ${\mathit{y}}_t$. This is mainly motivated by the real-life prediction problem that we will discuss in Section 4. It follows from (2) that

$$\begin{array}{c}\hfill \mathit{Y}(1,:)=\mathit{H}(1,:)\mathit{Z}+\mathit{W}(1,:).\end{array}$$

For ease of exposition, let us dub $\mathit{y}=\mathit{Y}{(:,1)}^{\top }$ and $\mathit{h}=\mathit{H}{(:,1)}^{\top }$. Note that the total number of predictors is $p=K_y·p_y+p_v$. As it is desirable for the estimated vector of coefficients $\widehat{\mathit{h}}$ to be sparse, we apply the Lasso method. More precisely,

$$\begin{array}{c}\hfill {\widehat{\mathit{h}}}_{\lambda }=\underset{\mathit{h}}{argmin}{\mathcal{L}}_{\lambda }(\mathit{h};\mathit{y},\mathit{Z}),\end{array}$$

(4)

where

$$\begin{array}{c}\hfill {\mathcal{L}}_{\lambda }(\mathit{h};\mathit{y},\mathit{Z})={\displaystyle \frac{\parallel \mathit{y}-{\mathit{Z}}^{\top }{\mathit{h}\parallel }_2^2}{2·\mathrm{size}\left(\mathit{y}\right)}}+\lambda {\parallel \mathit{h}\parallel }_1.\end{array}$$

(5)

It is obvious that $\mathrm{size}\left(\mathit{y}\right)=T$, but we prefer to write $\mathrm{size}\left(\mathit{y}\right)$ instead of T because later on we will employ the expression of ${\mathcal{L}}_{\lambda }(·;·)$ when the response vector and the matrix of predictors are not $\mathit{y}$ and $\mathit{Z}$. Hence, in (5), we emphasize that the quantity in the denominator of the first term is equal to the length of the response vector multiplied by two. To clarify why the input arguments for ${\mathcal{L}}_{\lambda }(·;·)$ are not necessarily $\mathit{y}$ and $\mathit{Z}$ we mention that, according to [16], the Lasso procedure is applied to the entire data set as well as to subsamples of the data set.

In [16], it is recommended to use the entire data set for selecting the value of $\lambda$ in (5). For a fixed integer $q\in (0,p)$, the value of $\lambda$ is varied until ${\widehat{\mathit{h}}}_{\lambda }$ has q non-zero entries, which means that q predictors have been selected. The ratio $\theta =q/p$ is regarded as the average selection probability and all the predictors for which the probability of selection is smaller than $\theta$ are deemed to be irrelevant. Note that the value of $\lambda$ selected in this way is then used in connection with the subsampling procedures that are described below.

2.3. Subsampling

The major difficulty in connection with subsampling stems from the fact the entries of the time series data are not independent. In order to address this issue, it was proposed in [16] to obtain from the time series data a sequence of blocks that are ‘almost’ independent. These blocks are obtained as follows: A partition of $2{\mu }_t$ subsets of the set $\{1,\dots ,T\}$ is defined such that each subset contains exactly $a_T$ elements. We suppose that $T/\left(2a_T\right)$ is an integer. If this condition is not satisfied, then the expression $\lfloor T/\left(2a_T\right)\rfloor$ that involves the floor operator should be used. The subsets of the partition are $O_1,\dots ,O_{{\mu }_T}$ and $E_1,\dots ,E_{{\mu }_T}$. For $j\in \{1,\dots ,{\mu }_T\}$,

$$\begin{array}{ccc}\hfill O_j& =& \{i:2(j-1)a_T+1\le i\le (2j-1)a_T\},\hfill \\ \hfill E_j& =& \{i:(2j-1)a_T+1\le i\le 2j{a}_T\}.\hfill \end{array}$$

(6)

Additionally, we define:

$$\begin{array}{c}\hfill O=\bigcup _{j=1}^{{\mu }_T}{O}_j.\end{array}$$

(7)

It is helpful to employ a different notation for the odd blocks (which are the O-blocks) and the even blocks (which are the E-blocks) because the data corresponding to the odd blocks are used in estimation, whereas the data corresponding to the even blocks are discarded. The key point is that any two odd blocks are separated by at least $a_T$ time points and, because of this feature, the odd blocks are deemed to be independent, especially when $a_T$ is large. Hence, the subsampling is performed only from the odd blocks and not from the entire time series. In order to illustrate how the subsampling works, suppose that we sample without replacement the sequence of blocks $O_1^{\prime },\dots ,O_{{\mu }_T/2}^{\prime }$ from the set of ${\mu }_T$ odd blocks. For ease of notation, we denote $\mathit{y}\left(O^{\prime }\right)$ the vector formed by the entries of $\mathit{y}$ whose indexes belong to the set

$$\begin{array}{c}\hfill O^{\prime }=\bigcup _{j=1}^{{\mu }_T/2}{O}_j^{\prime }.\end{array}$$

(8)

Similarly, $\mathit{Z}(:,O^{\prime })$ is the block of the matrix $\mathit{Z}$ formed by the columns of $\mathit{Z}$ whose indexes belong to the set $O^{\prime }$. Therefore, instead of estimating the vector of linear coefficients by minimizing the cost function in (5), we minimize

$$\begin{array}{c}\hfill {\mathcal{L}}_{\lambda }(\mathit{h};\mathit{y}\left(O^{\prime }\right),\mathit{Z}(:,O^{\prime })).\end{array}$$

(9)

The main difference between (5) and (9) is that in the former we utilize all T data from $\mathit{y}$, whereas in the latter we use only $({\mu }_T/2)a_T$ data from $\mathit{y}$. As it is evident that $({\mu }_T/2)a_T$ is equal to the cardinality of $O^{\prime }$, we prefer to employ the symbol $|O^{\prime }|$ when we refer to the amount of data.

2.4. Stability-Based Selection

The outcome of the Lasso for the optimization problem in (9) provides an estimator for the subset $S\subset \{1,\dots ,p\}$ that comprises the ‘signal’ variables. According to the nomenclature from [15,16], the subset $\{1,\dots ,p\}\setminus S$ contains the ‘noise’ variables. To fix the ideas, we give below the definition of the Lasso-based estimator for S:

$$\begin{array}{c}\hfill {\widehat{S}}_{|O^{\prime }|}=\left\{i:\widehat{\mathit{h}}\left(i\right)\ne 0,\widehat{\mathit{h}}=\underset{\mathit{h}}{argmin}{\mathcal{L}}_{\lambda }\left(\mathit{h};\mathit{y}\left(O^{\prime }\right),\mathit{Z}(:,O^{\prime })\right)\right\}.\end{array}$$

(10)

The presence of the symbol $|O^{\prime }|$ in ${\widehat{S}}_{|O^{\prime }|}$ indicates the amount of data involved in estimation.

Relying on the idea of the complementary pairs from [15], we consider the subset $O^{\prime \prime }=O\setminus O^{\prime }$ [see (7) and (8) for the definitions of O and $O^{\prime }$]. It is straightforward to obtain another estimator, ${\widehat{S}}_{|O^{\prime \prime }|}$, for the subset S of ‘signal’ variables by replacing $O^{\prime }$ with $O^{\prime \prime }$ in (9) and then applying Lasso for solving the optimization problem. Formally, the definition of ${\widehat{S}}_{|O^{\prime \prime }|}$ can be written down by using $O^{\prime \prime }$ instead of $O^{\prime }$ in (10). Bearing in mind that we aim to select the relevant predictors, we calculate the statistic

$$\begin{array}{c}\hfill {\mathbb{1}}_{{\widehat{S}}_{|O^{\prime }|}}\left(k\right)+{\mathbb{1}}_{{\widehat{S}}_{|O^{\prime \prime }|}}\left(k\right)\end{array}$$

(11)

for each index $k\in \{1,\dots ,p\}$. Obviously, the statistic above can take only the values 0, 1 and 2. Following the recipe from [15,16], we execute the subsampling not only once, but B times. We will discuss later on how the value of B can be chosen. Most importantly, at each subsampling, a different pair of sets $\left(O^{\prime },O^{\prime \prime }\right)$ is generated and the statistic in (11) is computed for each predictor. Furthermore, for each $k\in \{1,\dots ,p\}$, we calculate the sum of the statistics (11) obtained for the kth predictor in all B subsamplings and divide the result by $2B$ in order to obtain the estimator ${\widehat{\Pi }}_B^{av}\left(k\right)$. The possible values for ${\widehat{\Pi }}_B^{av}\left(k\right)$ are ${\displaystyle \frac{0}{2B}},{\displaystyle \frac{1}{2B}},{\displaystyle \frac{2}{2B}},\dots ,{\displaystyle \frac{2B}{2B}}$. It allows us to find the block average selection estimator

$$\begin{array}{c}\hfill {\widehat{S}}_{\varphi }^{av}=\left\{k:{\widehat{\Pi }}_B^{av}\left(k\right)\ge \varphi \right\},\end{array}$$

(12)

where $\varphi$ is a threshold that, in general, can take values in the interval $(0.5,0.9]$. The value of $\varphi$ is chosen by resorting to theoretical results on stability selection that provide an upper bound for the expected number of falsely selected predictors. This is the major advantage of the use of stability in comparison with applying only the Lasso base procedure. The main steps of the selection method presented in this section are shown in Algorithm 1. In the next sections, we investigate the performance of the algorithm in experiments with simulated data and air pollution data.

Algorithm 1 Stability selection with the base procedure Lasso (see [16] (Algorithm 1))

Input: $\mathit{y}\in {\mathbb{R}}^{T\times 1}$ [T measurements], $\mathit{Z}\in {\mathbb{R}}^{p\times T}$ [p predictors], $a_T$ [the cardinality for each odd block], $q/p$ [average selection probability], $\varphi$ [threshold], B [number of pairs $(O^{\prime },O^{\prime \prime })$], $\Lambda =\{{\lambda }_1,\dots ,{\lambda }_{100}\}$ [a sequence of penalty parameters] Initialize:  ${\widehat{\Pi }}_B^{av}\left(k\right)=0$, for all $k\in \{1,\dots ,p\}$ Select the penalty parameter: For $\lambda \in \Lambda$, solve $${\widehat{\mathit{h}}}_{\lambda }=\underset{\mathit{h}}{argmin}{\mathcal{L}}_{\lambda }(\mathit{h};\mathit{y},\mathit{Z})\left[\mathrm{see}\left(5\right)\right]$$ and then set $${\lambda }_q=\underset{\lambda \in \Lambda }{argmin}\left\{\parallel {\widehat{\mathit{h}}}_{\lambda }{\parallel }_0=q\right\}$$ for $n=1$ to B do    Sample: From the odd blocks $O_1,\dots ,O_{{\mu }_T}$, sample without replacement the sequence of blocks $O_1^{\prime },\dots ,O_{{\mu }_T/2}^{\prime }$, construct $O^{\prime }$ and set $O^{\prime \prime }=O\setminus O^{\prime }$ [see (7) and (8)]    Estimate: $${\widehat{S}}_{|O^{\prime }|}=\left\{i:\widehat{\mathit{h}}\left(i\right)\ne 0,\widehat{\mathit{h}}=\underset{\mathit{h}}{argmin}{\mathcal{L}}_{{\lambda }_q}\left(\mathit{h};\mathit{y}\left(O^{\prime }\right),\mathit{Z}(:,O^{\prime })\right)\right\}\left[\mathrm{see}\left(10\right)\right]$$ $${\widehat{S}}_{|O^{\prime \prime }|}=\left\{i:\widehat{\mathit{h}}\left(i\right)\ne 0,\widehat{\mathit{h}}=\underset{\mathit{h}}{argmin}{\mathcal{L}}_{{\lambda }_q}\left(\mathit{h};\mathit{y}\left(O^{\prime \prime }\right),\mathit{Z}(:,O^{\prime \prime })\right)\right\}$$ $${\widehat{\Pi }}_B^{av}\left(k\right)={\widehat{\Pi }}_B^{av}\left(k\right)+{\displaystyle \frac{1}{2B}}·{\mathbb{1}}_{{\widehat{S}}_{|O^{\prime }|}}\left(k\right)+{\displaystyle \frac{1}{2B}}·{\mathbb{1}}_{{\widehat{S}}_{|O^{\prime \prime }|}}\left(k\right),\mathrm{for}\mathrm{all}k\in \{1,...,p\}$$ end for Output: ${\widehat{S}}_{\varphi }^{av}=\left\{k:{\widehat{\Pi }}_B^{av}\left(k\right)\ge \varphi \right\}$ [see (12)]

3. Experiments with Simulated Data

3.1. Artificial Data

We generate data sets according to the VARX model in (1). In our simulations, all VARX models are guaranteed to be stable (see again the stability condition in (3)). The most important attributes are:

• Parameters: (i) number of time series: $K_y=4$; (ii) autoregressive order: $p_y=3$; (iii) standard deviation for the non-zero entries of the matrices $\left\{{\mathit{A}}_i\right\}$ and for the entries of the vectors $\left\{{\mathit{v}}_t\right\}$: $\sigma =0.25$.
• Matrices $\left\{{\mathit{A}}_i\right\}$: (i) We generate the matrices ${\left\{{\stackrel{\sim }{\mathit{A}}}_i\right\}}_{i=1}^{p_y}$ whose entries are independent outcomes from the Gaussian distribution $\mathcal{N}(0,{\sigma }^2)$; (ii) let $\mathit{M}\in {\mathbb{R}}^{K_y\times K_y}$ be a matrix that has all entries equal to one. Some entries on the off-diagonal locations of $\mathit{M}$ are randomly selected and forced to be zero (the number of zero entries is restricted to be less than $0.6·K_y^2$); (iii) for $1\le i\le p_y$, the element-wise product of $\mathit{M}$ and ${\tilde{\mathit{A}}}_i$ gives the matrix ${\mathit{A}}_i$.
• Matrix $\mathit{C}$: (i) $c_{11}=c_{12}=c_{13}=1$; (ii) all other entries are equal to zero.
• Vectors $\left\{{\mathit{v}}_t\right\}$: (i) model for the first entry: zero-mean order-1 autoregressive process with autocorrelation function $\rho \left(\tau \right)={(-0.9)}^{\left|\tau \right|}$, for $\tau \in \mathbb{Z}$; model for the second entry: zero-mean order-1 autoregressive process with autocorrelation function $\rho \left(\tau \right)={(-0.5)}^{\left|\tau \right|}$, for $\tau \in \mathbb{Z}$; model for all other entries: independent outcomes from the Gaussian distribution $\mathcal{N}(0,{\sigma }^2)$.
• Vectors $\left\{{\mathit{w}}_t\right\}$: model: multivariate Gaussian distribution with zero-mean vector and covariance matrix ${\sigma }_{\nu }^2\mathit{I}$, where ${\sigma }_{\nu }^2={\left(\psi \sigma \right)}^2$ and $\psi \in \{0.1,1,10\}$.
• Signal-to-noise ratio (SNR): (i) formula: $10{log}_{10}{\displaystyle \frac{{\sigma }^2}{{\sigma }_{\nu }^2}}=10{log}_{10}{\displaystyle \frac{1}{{\psi }^2}}$; (ii) values: −20 dB, 0 dB and 20 dB.

With these settings, we simulate 100 data sets for each value of SNR mentioned above. For each data set, we have: (i) number of measurements: $T=10,000$; (ii) maximum order of autoregressions (used in estimation): $p_y^{max}=4$ (greater than the ‘true’ order $p_y$); (iii) number of endogenous predictors: $p_y^{max}·K_y=4·4=16$; (iv) number of exogenous predictors: $p_v=84$; total number of predictors: $p=16+84=100$.

The methodology utilized for producing the artificial data has a certain degree of similarity with the one used for simulating the data in an experiment reported in [8]. There are two differences between the approach in this work and the one in [8]. The first difference stems from the fact that, in [8], all the odd blocks have the same cardinality and all the even blocks have the same cardinality, but $|O_1|$ can be different from $|E_1|$. The second difference is that, in [8], after generating the blocks $O_1,\dots ,O_{{\mu }_T}$, only the data in $\mathit{y}\left(O_1\right)$ are used in order to select the best predictors by employing a greedy algorithm and an IT criterion. The same pair (greedy algorithm, IT criterion) is then used for selecting the predictors from the data in $\mathit{y}\left(O_2\right)$ and the procedure continues until the data from each odd block is utilized in the selection of the predictors. It is decided that a particular predictor can be included in the final model only if it was selected in at least $80\%$ of the data blocks $\mathit{y}\left(O_1\right),\dots ,\mathit{y}\left(O_{{\mu }_T}\right)$.

In the section below, we focus on the parameters for Algorithm 1. For the sake of simplicity, we sometimes use the acronym BPA for Algorithm 1. This acronym is inspired by [16] (Definition 1) and means Block Pair Average.

3.2. Settings for the Parameters of the Algorithm

3.2.1. Parameter $a_T$ [The Cardinality for Each Odd Block]

The value of $a_T$ is usually set to an integer multiple of the time series seasonality, and set to $\sqrt{T}$ or $logT$ in the absence of seasonality [16]. As ${\mu }_T=T/\left(2a_T\right)$, it is evident that $a_T$ determines the number of odd blocks ${\mu }_T$. When $a_T$ is larger, we have longer blocks and, therefore, fewer odd blocks. However, the number of data involved in estimation does not depend on $a_T$ because the sample size is given by $|O^{\prime }|=|O^{\prime \prime }|=({\mu }_T/2)a_T=T/4$. The possible influence of $a_T$ on the estimation results comes from the fact that the distance between any two consecutive odd blocks $O_j$ and $O_{j+1}$, $j\in \{1,\dots ,{\mu }_T-1\}$ is larger when $a_T$ increases and this potentially makes the blocks more independent. In our empirical evaluations of the algorithm, we allow $a_T$ to vary over an extensive range of values: $a_T\in \{5,10,50,100,250,500\}$.

3.2.2. Parameter $q/p$ [Average Selection Probability]

The magnitude of q determines the penalty parameter ${\lambda }_q$; the lower the q parameter, the harsher the Lasso penalty. The value $q/p=0.4$ was set as default in [16], with $q/p=0.2$ and $q/p=0.6$ tested in their real-life data experiment. When $q/p=0.2$, it might be too conservative, while $q/p=0.6$ is less conservative and easier to let predictors in for the second screening controlled by $\varphi$. Hence, we want to explore the different values of $q/p\in \{0.2,0.4,0.6\}$.

3.2.3. Parameter $\varphi$ [Threshold]

We will pay close attention to the relationship between q and $\varphi$. For a lower $\varphi$-threshold, it is easier for variables to be selected, while it will become harder when the $\varphi$-threshold is higher. For example, only the most prominent and useful variables will be selected for a threshold of 0.9. Therefore, it is expected that the final number of variables selected will be smaller when we raise the $\varphi$-threshold. The value for $\varphi$ is set to 0.8 in [16], while the parameter values tested in our experiments are $\varphi \in \{0.5,0.6,0.7,0.8,0.9\}$.

3.2.4. Parameter B [Number of Pairs $(O^{\prime },O^{\prime \prime })$]

The computational complexity of Algorithm 1 is mainly given by (i) the cost for running Lasso over all regularization parameters in the set $\Lambda$ when the dictionary size is $T\times p$ and (ii) the cost for executing Lasso $2B$ times for the fixed penalty parameter ${\lambda }_q$ when the dictionary size is $(T/4)\times p$. More information about the computational complexity for (i) and (ii) can be found in [2,12]. The number of iterations B is set to 50 in [16]. Since more iterations make the stability-based selection method more computationally intensive, we want to find out whether lowering B can retain the same performance of the algorithm. Hence, B is tested for the values $5,25,50$. When experimenting on B, we keep in mind that at each iteration we should have a different pair $(O^{\prime },O^{\prime \prime })$. Given that T is fixed, this condition is mainly related to the value of $a_T$. Note that even for the maximum value of $a_T$ employed in our experiments ($a_T=500$), the total number of ways in which one can sample without replacement ${\mu }_T/2$$O^{\prime }$-blocks from ${\mu }_T$ odd blocks is large enough. Elementary calculations show that ${\mu }_T=10$ when $a_T=500$ and the number of ways in which the $O^{\prime }$-blocks can be sampled from the O-blocks is $\left({\displaystyle \genfrac{}{}{0pt}{}{10}{5}}\right)=252$, which is greater than 50.

3.3. Empirical Evaluation of the Effect of Various Settings

3.3.1. SNR Level

In the first experiment, we use the 100 data sets that we have generated for each SNR-value and test how BPA performs under different noise levels. While the noise level changes, the parameters of the algorithm are fixed to the following default values: $a_T=100$, $q/p=0.4$, $\varphi =0.8$ and $B=50$. The true positive rate (TPR) and the false positive rate (FPR) are calculated and they are shown in Figure 1. In the figure, we see that TPR does not change much, and it stays at a pretty high level when SNR varies. However, there are more outliers and slightly lower TPR for higher SNR. The FPR drops as SNR increases, which is expected because the stability method is less likely to detect false predictors when the data are less noisy. The results suggest that BPA is quite robust to noise.

To explore further the cause of slightly lower TPR for higher SNR, we investigate the selection of ${\lambda }_q$ and the number of predictors selected ($p^{*}$) for each SNR level. In Figure 2, we notice that the ${\lambda }_q$-values are generally very similar, for all levels of SNR. It is interesting that for SNR = 20 dB there are more outliers and a broader range of ${\lambda }_q$-values than for SNR = −20 dB.

Next, we investigate the effect of the threshold $\varphi$ on the final number of predictors selected. The results, which are shown in Figure 3, indicate that the number of the predictors selected decreases significantly after applying the rule based on the threshold $\varphi =0.8$ at the last step of the algorithm. This confirms that the Lasso regularization controlled by q (via ${\lambda }_q$) only serves as an instrument for the preliminary selection of the predictors [16] and $\varphi$ has a critical role in identifying the high selection probability predictors that are included in the final model. We also notice in Figure 3 that there are fewer final predictors selected for SNR = 20 dB than for the smaller SNR values, which results in both TPR and FPR being lower for SNR = 20 dB in Figure 1.

We continue the analysis by considering various values for the parameters of the algorithm (see again Section 3.2). For conciseness, we do not report the results obtained for the parameter $a_T$ because our experimental findings show that $a_T$ does not have an important influence on TPR and FPR. The other parameters have a stronger impact on the outcome of the experiments.

3.3.2. Parameter $q/p$ [Average Selection Probability]

In Figure 4, we observe that as $q/p$ increases, the Lasso penalty becomes weaker and more variables can pass through the selection criteria. Note that the second selection criterion is based on the threshold $\varphi$, which is fixed at the value 0.8. Our results suggest that keeping $\varphi$ constant and loosening the first selection criterion by increasing $q/p$ leads to higher TPR and higher FPR. Interestingly, the graphs in the figure show a larger increase in FPR than in TPR. The much higher FPR for $q/p=0.6$ suggests that the loose Lasso regularization has a considerable impact on the correct final selections. The influence of $\varphi =0.8$ (the second condition being quite harsh) can only be effective at filtering out the less good variables and keeping the FPR relatively low when the variable cannot pass the first criterion easily.

3.3.3. Parameter $\varphi$ [Threshold]

In Figure 5, we see that as $\varphi$ increases, the FPR drops together with a slight drop in TPR. When $\varphi \in \{0.5,0.6,0.7\}$, the values of FPR are far too large. Therefore, $\varphi$ should not be less than 0.8. Setting $\varphi =0.9$ is acceptable, but it may be too harsh and lowers TPR. We conclude that $\varphi =0.8$ is a good choice.

3.3.4. Parameter B [Number of Pairs $(O^{\prime },O^{\prime \prime })$]

In Figure 6, we observe that the results for B suggest no significant difference in TPR/FPR between 25 and 50 iterations. This means that 25 iterations are sufficient to arrive at a good selection of the final predictors. Still, if we have the computing capacity at 50 iterations, it will help improve the FPR slightly. However, when we reduce the number of iterations to 5, the FPR is clearly higher than in the cases of 25 and 50 iterations. Thus, we should probably not use too few iterations.

3.3.5. A Variant of the BPA

We investigate the performance of a variant of BPA in which the Lasso selection is performed on each of the data blocks $\mathit{y}\left(O_1\right),\dots ,\mathit{y}\left(O_{{\mu }_T}\right)$. Let $N_k$ denote the number of times the kth predictor is selected, where $k\in \{1,\dots ,p\}$. It is obvious that $0\le N_k\le {\mu }_T$. We decide that the kth predictor should be included in the final model if ${\displaystyle \frac{N_k}{{\mu }_T}}\ge \varphi$, where $\varphi =0.8$. Remark that in comparison with Algorithm 1, there is no random selection of the $O^{\prime }$-blocks and the small data blocks are not aggregated. In order to distinguish this variant of the algorithm from the original one, we call it BPA-m. Another modification in comparison with Algorithm 1 is that in the selection of ${\lambda }_q$ only the data in $\mathit{y}\left(O_1\right)$ are utilized. The reason for this change is that the use of all $T=10,000$ data for the selection of ${\lambda }_q$ would give a small ${\lambda }_q$ that cannot perform an effective regularization on data blocks of length $a_T=100$. The original BPA does not have this issue because the small data blocks are aggregated, hence the size of $\mathit{y}\left(O^{\prime }\right)$ as well as the size of $\mathit{y}\left(O^{\prime \prime }\right)$ is equal to $T/4=2500$.

The comparison of the results produced by BPA and BPA-m is presented in Figure 7, which indicates that BPA performs much better than BPA-m in terms of TPR, but performs slightly worse in terms of FPR. Therefore, BPA-m is too conservative compared to BPA. We also notice that the TPR improvement from increasing SNR is very clear for BPA-m.

4. Experiments with Real-Life Data

4.1. Air Pollution Data

In this section, we apply the BPA stability method to the Auckland air pollution data set that was used in [5,8]. The problem that we address is the same as the one considered in [5,8] and concerns the selection of the predictors. The main difference between our approach and the methods used in the previous works comes from the fact that we do not use greedy algorithms and IT criteria/cross-validation for selecting the predictors.

The Auckland air pollution data set comprises daily measurements of the concentrations of particulate matter (PM), specifically ${\mathrm{PM}}_{2.5}$ and ${\mathrm{PM}}_{10}$ (in $\mathsf{\mu }\mathrm{g}/{\mathrm{m}}^3$) at four locations in Auckland, New Zealand. Note that ${\mathrm{PM}}_{2.5}$ includes particles less than 2.5 $\mathsf{\mu }\mathrm{m}$ in diameter; ${\mathrm{PM}}_{10}$ includes particles less than 10 $\mathsf{\mu }\mathrm{m}$ in diameter. Therefore, ${\mathrm{PM}}_{2.5}$ is a subset of ${\mathrm{PM}}_{10}$. The sites where the data were measured are Patumahoe (PA), Penrose (PE), Takapuna (TA), and Whangaparaoa (WH). In parentheses are written the acronyms that we use in this work for the sites. The measurements were collected from 30 April 2008 to 30 June 2014.

For a measurement site, let ${\theta }_{\mathrm{site}}\left(1\right),{\theta }_{\mathrm{site}}\left(2\right),\dots$ be the time series of log-transformed daily concentrations of ${\mathrm{PM}}_{2.5}$. For example, ${\theta }_{\mathrm{PA}}\left(1\right)$ is the log-transformed ${\mathrm{PM}}_{2.5}$ measurement for Patumahoe, on the first day. Similarly, ${\xi }_{\mathrm{site}}\left(1\right),{\xi }_{\mathrm{site}}\left(2\right),\dots$ are the log-transformed values for the concentrations of ${\mathrm{PM}}_{10}$ measured at the specific site, during day $1,2,\dots$

We want to estimate the concentration of ${\mathrm{PM}}_{2.5}$ at a specific site when this concentration cannot be measured because of a malfunction of the sensors. To this end, we find a linear model that describes the relationship between the log-transformed concentration of ${\mathrm{PM}}_{2.5}$ on the current day (at a specific site) and the following variables: (i) past and present log-transformed concentrations of ${\mathrm{PM}}_{10}$ for that specific site and (ii) past and present log-transformed concentrations of ${\mathrm{PM}}_{2.5}$ for the other three sites.

Let $n=365$ because there are 365 days during a normal year. For a certain site, we take the response vector to be

$$\begin{array}{c}\hfill {\mathit{y}}_{\mathrm{site}}\left(t\right)={[{\theta }_{\mathrm{site}}\left(t\right){\theta }_{\mathrm{site}}(t-1)\dots {\theta }_{\mathrm{site}}(t-n+1)]}^{\top },\end{array}$$

(13)

where $t\ge n$ denotes an arbitrary day. For the same site, we use the notation ${\mathit{X}}_{\mathrm{site}}\left(t\right)$ for the matrix of predictors. The columns of the matrix are arranged into four blocks. The representations for the response vector and the matrix of predictors for each site are shown in Figure 8a. Note that the response vector and the columns of the predictors matrix are centered and standardized.

We consider two scenarios for constructing the four blocks of the matrix of predictors.

• Scenario A—Full set of predictors (FullSet): The first block contains the measurements from last year of the log-transformed concentrations of ${\mathrm{PM}}_{10}$ for the site for which we wish to estimate the concentration of ${\mathrm{PM}}_{2.5}$. The next three blocks contain the log-transformed concentrations of ${\mathrm{PM}}_{2.5}$ for the other three sites that have been measured during the last year. The predictors are presented in Figure 8b. The total number of predictors is $p=4(n+1)=1464$, thus $p\gg n$ (high-dimensional case).
• Scenario B—Constrained set of predictors (ConSet): In this scenario, we reduce the total number of predictors by using empirical knowledge (see [5]). The predictors are presented in Figure 8c. Simple calculations show that the total number of predictors is $p=4·17=68$, thus $p<n$.

4.2. Performance Evaluation

The performance evaluation is performed by using the same methodology as in [5,8]. The procedure described below is applied for ${\mathrm{N}}_{TR}=100$ runs. In each run, we use a data segment of length $3n$ (three consecutive years of measurements). The first two years are used for training (i.e., for selecting the predictors by applying BPA) and the third year is used for testing. In the first run of this experiment, we take $t_0$ to be 30 April 2008 and let $t_1$ be the last day of the second year, so $t_1=t_0+2n-1$. It follows that the training response vector is ${\mathit{y}}_{\mathrm{site}}\left(t_1\right)$ and the training predictors matrix is ${\mathit{X}}_{\mathrm{site}}^{\mathrm{SC}}\left(t_1\right)$, where ${(·)}^{\mathrm{SC}}$ is used to distinguish between scenario A and scenario B. After the BPA is employed for selecting the predictors from the training predictors matrix, the corresponding linear coefficients are computed by minimizing the sum of the squared residuals (least squares). The vector of coefficients is further used together with the testing predictors matrix in the third year ${\mathit{X}}_{\mathrm{site}}^{\mathrm{SC}}(t_1+n)$ in order to produce the estimate ${\widehat{\mathit{y}}}_{\mathrm{site}}(t_1+n)$. The same procedure is used in the rth run, where $r\in \{2,\dots ,{\mathrm{N}}_{TR}\}$, by replacing $t_1$ with $t_r=t_1+(r-1)8$.

The normalized mean square error (NMSE) is computed by applying the following formula for each site and for each scenario:

$$\begin{array}{c}\hfill {\mathrm{NMSE}}_{\mathrm{site}}^{\mathrm{SC}}=\frac{{\sum }_{r=1}^{{\mathrm{N}}_{TR}}{∥\exp \left({\mathit{y}}_{\mathrm{site}}(t_r+n)\right)-\exp \left({\widehat{\mathit{y}}}_{\mathrm{site}}^{SC}(t_r+n)\right)∥}^2}{{\sum }_{r=1}^{N_{TR}}{∥\exp \left({\mathit{y}}_{\mathrm{site}}(t_r+n)\right)∥}^2},\end{array}$$

(14)

where applying $\exp (·)$ to the vector means that we exponentiate each entry of the vector in the argument. Exponentiation is used to back-transform the log-transformed data to the original scale.

4.3. Experimental Results

First we write down the values of the BPA parameters that are used in this experiment. For instance, it is natural to take $T=365$ because $n=365$. Furthermore, we take $a_T=3·7=21$, which is a multiple integer of the seasonality. The factor 3 is selected for the convenience of the implementation as we already know from the experiments with simulated data that the value of $a_T$ does not have an important effect on the performance of the algorithm. It follows that ${\mu }_T=\lfloor T/\left(2a_T\right)\rfloor =\lfloor 365/(2·21)\rfloor =8$. Therefore, there are eight odd blocks in total, and when running the algorithm we sample without replacement the blocks $O_1^{\prime },\dots ,O_4^{\prime }$. According to (8), $|O^{\prime }|=4·21=84$; it is also clear that $|O^{\prime \prime }|=|O^{\prime }|$. The other BPA parameters are: $q=\lfloor 0.4·p\rfloor$, $\varphi =0.8$ and $B=50$. Their selection is based on the results obtained with simulated data.

The values of NMSE computed by applying BPA are presented on the third row of

Table 1. The values of NMSE (in percentages), which are computed by applying the formula in (14), are shown on the third row of the table. For computing the deciles that are reported on the fourth row, for each site and for each scenario, we rank from lowest to highest 86 values of NMSE. One of these values is the NMSE reported on the third row of the table and the other 85 are the NMSE values obtained in [8] by employing various greedy algorithms and IT criteria/cross-validation. The best ranked methods are those that are assigned to the decile D1 because they produce the smallest $10\%$ NMSEs. Similarly, the methods assigned to D2 yield the smallest $20\%$ NMSEs, the methods assigned to D3 yield the smallest $30\%$ NMSEs, and so on.

Site	Patumahoe		Penrose		Takapuna		Whangaparaoa
Scenario	FullSet	ConSet	FullSet	ConSet	FullSet	ConSet	FullSet	ConSet
NMSE	$5.61\%$	$5.59\%$	$3.75\%$	$3.60\%$	$6.05\%$	$5.73\%$	$3.17\%$	$3.15\%$
Decile	D2	D7	D1	D6	D2	D10	D1	D8

. It is evident that the ConSet scenario always leads to a smaller NMSE than the NMSE for the FullSet scenario. This is expected because in ConSet we use prior knowledge to pre-select the useful predictors. To gain more insight, we comment briefly on the top three most frequently selected predictors in ${\mathrm{N}}_{TR}$ runs for the different sites, in each scenario. We note that ${\mathrm{PM}}_{10}$ from the same site on the present day is always the top voted predictor and is always selected in each of the ${\mathrm{N}}_{TR}$ runs. The selection of this predictor can be easily understood because ${\mathrm{PM}}_{2.5}$ is a subset of ${\mathrm{PM}}_{10}$, so the concentration of ${\mathrm{PM}}_{2.5}$ inevitably correlates with the ${\mathrm{PM}}_{10}$ measurement on the same day, at the same site. Then, the ${\mathrm{PM}}_{2.5}$ measurements on the same day for the other sites are usually the second and third most voted predictors. This is also expected because the ${\mathrm{PM}}_{2.5}$ concentration on the site that we want to predict most likely correlates with the ${\mathrm{PM}}_{2.5}$ measurements recorded for the other three sites on the same day. The only exception is for the ConSet scenario in Penrose, where the third most voted predictor is the ${\mathrm{PM}}_{2.5}$ measurement from Takapuna one day before the present day. This selection is reasonable because Penrose and Takapuna have some similarities (see [5,8]). This analysis is only descriptive and does not suggest any causal relationship between the predictors and the ${\mathrm{PM}}_{2.5}$ measurements.

On the fourth row of Table 1 are shown the results of comparison with the NMSE’s computed in [8], where the same problem for the Auckland air pollution data set was solved by using greedy algorithms and IT criteria/cross-validation. We mentioned in Section 1 that the greedy algorithms employed in [8] are MPA, OMP, RMP, FWA and CMP. In the same section, we pointed out that 22 IT criteria have been used in conjunction with MPA. As cross-validation was also used for MPA, it means that the total number of selection rules for MPA was 23. Similarly, the total number of selection rules for the other four greedy algorithms are: OMP–13, RMP–23, FWA–13 and CMP–13. Hence, the NMSE values produced by BPA for each site and for each scenario are compared with the results yielded by other 85 methods. Given the large number of NMSE’s that are considered, we compute the deciles for assigning different levels of performance to them.

We can see in Table 1 that the performance of BPA is very good for the FullSet scenario, where BPA is assigned either to D1 (top $10\%$ methods) or to D2 (top $20\%$ methods). This supports the argument that BPA is suitable for the high-dimensional FullSet scenario. This observation is perfectly in line with the theoretical grounds on which BPA was derived. However, the ranking of BPA is modest for the ConSet scenario, where the number of the available data is larger than the total number of the predictors.

The experimental results can be reproduced by using the Matlab code (accessed on 25 June 2023) available at https://www.stat.auckland.ac.nz/%7Ecgiu216/PUBLICATIONS.htm.

5. Conclusions, Limitations, and Future Research

In this work, we have analyzed the applicability of BPA to simulated and real-life time series data sets. During the testing of the algorithm on artificial data with various noise levels, we have found that BPA is quite robust. Based on the experiments with simulated data, where we have investigated various parameter settings for BPA, we concluded that the parameters $q/p$ and $\varphi$ are the most influential on BPA’s performance. Parameter $q/p$ should be set at a medium level to allow predictors to pass through the first layer, whereas $\varphi$ should be set at a relatively high level for having an effective selection of the predictors. We have used our findings from the simulated data experiment in the experiment with air pollution data. BPA performed very well in the FullSet scenario $(p\gg n)$ of this experiment on real-life data, where it was compared with 85 methods that have been previously evaluated in [8]. At the same time, the results for the ConSet scenario suggest that BPA should not be used when $n>p$.

An area where the performance of BPA can be further investigated is to evaluate the influence of the base selection procedure. In this work, we focused on Lasso because of the popularity of Lasso in the signal processing community. It might also be interesting to extend the application of BPA to other signal processing problems that are not related to the classical VARX model.