On Fast Converging Data-Selective Adaptive Filtering

Abstract

The amount of information currently generated in the world has been increasing exponentially, raising the question of whether all acquired data is relevant for the learning algorithm process. If a subset of the data does not bring enough innovation, data-selection strategies can be employed to reduce the computational complexity cost and, in many cases, improve the estimation accuracy. In this paper, we explore some adaptive filtering algorithms whose characteristic features are their fast convergence and data selection. These algorithms incorporate a prescribed data-selection strategy and are compared in distinct applications environments. The simulation results include both synthetic and real data.

1. Introduction

In many practical applications, the number of data-acquisition devices is growing at an exponential rate, such as in distributed networks, massive multiple-input multiple-output (MIMO) antennas, and for social networks. This trend calls for the parsimonious use of the acquired data when considering the overwhelming resources required such as storage capacity, device-to-device communications, power consumption, among others. In the era of Big Data, we face the challenge of efficiently utilizing a large amount of data to extract the critical information. In the context of adaptive filtering, the recently proposed strategy to perform data-selection approximating a prescribed update rate appears to be promising [1].

This new data-selection method prescribes a probability of updating utilizing a threshold based on the mean squared error (MSE) which determines if the acquired data sample contains enough information to justify a change in the parameter estimate. Utilizing a statistical model for the MSE, it is possible to prescribe the probability of updating inherent to the learning algorithm. Also, an additional threshold value can be utilized to verify if the data represents an outlier, i.e., abnormal data.

Previous work [1] addressed the classical Least Mean Square (LMS), the Affine Projection (AP), and the Recursive Least Squares (RLS) algorithms. Among these algorithms, the RLS has the fastest convergence in stationary environments, and the highest computational complexity, while facing numerical stability issues. In this work, we consider fast converging adaptive filtering algorithms with data selection and apply them to process real data in order to verify their effectiveness in addressing practical problems. The proposed algorithms are the LMS Newton (LMSN) [2], the LMS Quasi-Newton (LMSQN) [3] and the online conjugate gradient (CG) [4,5,6,7,8,9]. Like the RLS algorithm, the LMSN has fast convergence even when the input signal is highly correlated. By employing an estimate of the Hessian matrix, the LMSQN methods [10,11,12,13] have a similar computational complexity to the RLS and LMSN algorithms without sacrificing the performance. In fact, one version of the LMSQN algorithm exhibits improved robustness to quantization errors [3]. A low complexity algorithm originates from the Conjugate Gradient (CG) method [4] which does not require the computation of the Hessian matrix inverse as the LMSN, LMSQN, and RLS algorithms. The CG algorithm is attractive since it updates the adaptive filter coefficients based on conjugate directions leading to faster convergence than the gradient-based algorithms such as the LMS.

We explore the data-selective version of the LMSN, LMSQN and CG algorithms, showing their cost function in different applications, such as equalization and signal enhancement, which were not previously considered in [1,14,15]. For each configuration, we describe how to estimate the MSE to allow the prescription of the update rate.

The performance of the data selective algorithms is evaluated via simulations utilizing synthetic and real data in different adaptive filtering applications. The results indicate that the data selection strategy can indeed be applied to any type of applications of adaptive filters.

This paper is organized as follows. In Section 2, the data-selection strategy is discussed along with the calculation of the threshold parameter required to achieve a target probability of update. Section 3 describes the data-selective LMSN (DS-LMSN), LMSQN (DS-LMQSN), and the CG (DS-CG) algorithms. Section 4 compares the proposed algorithms. Some concluding remarks are provided in Section 5.

2. Problem Description

This section describes how to apply the data selection approach for distinct application set-ups. Regardless of the application, the filter output can be formulated as

$$y\left(k\right)={\mathbf{w}}^T\left(k\right)\mathbf{x}\left(k\right)$$

(1)

where $\mathbf{x}\left(k\right)={[x_0\left(k\right)x_1\left(k\right)\cdots x_{N-1}\left(k\right)]}^T$ is the input applied to the adaptive filter and $\mathbf{w}\left(k\right)={[w_0\left(k\right)w_1\left(k\right)\cdots w_{N-1}\left(k\right)]}^T$ represents the adaptive filter coefficients. We can compute the a priori error signal $e\left(k\right)$ as

$$e\left(k\right)=d\left(k\right)-{\mathbf{w}}^T\left(k\right)\mathbf{x}\left(k\right)$$

(2)

where $d\left(k\right)$ is the desired signal.

Basically the error signal $e\left(k\right)$ is used by the adaptive algorithm to determine the updating of the filter coefficients. Given the error distribution, it is possible to infer the degree of innovation the current data carries; an illustration is provided in Figure 1. Observe that the central interval (in light blue) is equivalent to lower error values, whereas the edge intervals in red correspond to higher error values, indicating the presence of possible outliers that damage the estimation. The data-selection strategy relies on updating the filter coefficients only when the current data are informative, i.e., generate an error that does not belong to any of those intervals. Hence, the overall computational cost is reduced, since the coefficients are no longer updated $100\%$ of the time.

The adaptive algorithms require an objective function of the error in order to perform the coefficient updating. A common objective function considered in the adaptive filtering theory is the instantaneous squared error,

$$J\left(\mathbf{w}\left(k\right)\right)=\frac{1}{2}{\left|e\left(k\right)\right|}^2$$

(3)

where $|·|$ denotes the absolute value.

If the error distribution is assumed to be Gaussian, then

$$\mathit{e}\sim \mathcal{N}(0,{\sigma }_e^2)$$

(4)

where ${\sigma }_e^2$ is the error variance. By normalizing the error distribution, we obtain

$$\frac{\mathit{e}}{{\sigma }_e}\sim \mathcal{N}(0,1).$$

(5)

The updating of the adaptive filter coefficients occurs if the normalized error $\frac{\left|e\right(k\left)\right|}{{\sigma }_e}$ is greater than a given threshold $\sqrt{\tau \left(k\right)}$. However, if $\frac{\left|e\right(k\left)\right|}{{\sigma }_e}$ is greater than another threshold $\sqrt{{\tau }_{\max }}$, an outlier is identified and thus, no update should be performed. These conditions can be incorporated to the minimization of the function

$$J^{\prime }\left(\mathbf{w}\left(k\right)\right)=\left\{\begin{array}{c}\frac{1}{2}{\left|e\left(k\right)\right|}^2,\mathrm{if}\sqrt{\tau \left(k\right)}\le \frac{\left|e\right(k\left)\right|}{{\sigma }_e}<\sqrt{{\tau }_{\max }}\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.$$

(6)

Hence, the coefficient updating follows the rule,

$$\begin{array}{c}\mathbf{w}(k+1)=\left\{\begin{array}{c}\mathbf{w}\left(k\right)+\mathbf{u}\left(k\right),\sqrt{\tau \left(k\right)}\le \frac{\left|e\right(k\left)\right|}{{\sigma }_e}<\sqrt{{\tau }_{\max }}\hfill \\ \mathbf{w}\left(k\right),\mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(7)

where the term $\mathbf{u}\left(k\right)$ depends on the adaptive algorithm employed. The desired probability of coefficient update $P_{\mathrm{up}}\left(k\right)$ represents how often the first statement in Equation (7) is performed and is modeled as

$$P_{\mathrm{up}}\left(k\right)=P\left\{\frac{\left|e\right(k\left)\right|}{{\sigma }_e}>\sqrt{\tau \left(k\right)}\right\}-P\left\{\frac{\left|e\right(k\left)\right|}{{\sigma }_e}>\sqrt{{\tau }_{\max }}\right\}.$$

(8)

By considering the distribution in (5), Equation (8) in steady-state becomes,

$$P_{\mathrm{up}}=2Q_e\left(\sqrt{\tau }\right)-2Q_e\left(\sqrt{{\tau }_{\max }}\right),$$

(9)

where $Q_e(·)$ is the complementary Gaussian cumulative distribution function, given by $Q_e\left(x\right)=1/\left(2\pi \right){\int }_x^{\infty }exp(-t^2/2)dt$ [16]. Even when outliers are present in the dataset, the probability $P\left\{\frac{\left|e\right(k\left)\right|}{{\sigma }_e}>\sqrt{{\tau }_{\max }}\right\}$ tends to be very small. Therefore, the parameter $\tau$ can be obtained from Equation (9) as

$$\sqrt{\tau }=Q_e^{-1}\left(\frac{P_{\mathrm{up}}}{2}\right),$$

(10)

where $Q_e^{-1}(·)$ is the inverse of the $Q_e(·)$ function. Basically, to apply the threshold $\tau$ in the coefficient updating (7), we need to calculate ${\sigma }_e^2$. At this point, it should be mentioned that in the system identification application with sufficient order, the minimum MSE in steady-state is ${\sigma }_n^2$, the variance of the measurement noise $n\left(k\right)$. Hence, it is convenient to express ${\sigma }_e^2$ as a function of the noise variance

$${\sigma }_e^2=(1+\rho ){\sigma }_n^2$$

(11)

in which the excess MSE is rewritten as $\rho {\sigma }_n^2$. The expression of $\rho$ depends on the adaptive algorithm and is key to establish the prescribed probability of update $P_{\mathrm{up}}$.

As a result, the coefficient updating is performed based on a scaled power noise, $\tau \left(k\right){\sigma }_n^2$ [1,14,15] so that an equivalent expression to Equation (8) can be rewritten as

$$P_{\mathrm{up}}\left(k\right)=P\left\{\frac{\left|e\right(k\left)\right|}{{\sigma }_n}>\sqrt{\tau \left(k\right)}\right\}-P\left\{\frac{\left|e\right(k\left)\right|}{{\sigma }_n}>\sqrt{{\tau }_{\max }}\right\}$$

(12)

resulting in the following modifications in Equations (9) and (10)

$$P_{\mathrm{up}}=2Q_e\left(\frac{{\sigma }_n\sqrt{\tau }}{{\sigma }_e}\right)-2Q_e\left(\frac{{\sigma }_n\sqrt{{\tau }_{\max }}}{{\sigma }_e}\right)\mathrm{and}\sqrt{\tau }=\sqrt{(1+\rho )}{Q}_e^{-1}\left(\frac{P_{\mathrm{up}}}{2}\right)\mathrm{for}{\tau }_{\max }\to \infty .$$

(13)

If the presence of outliers is known, a possible strategy to eliminate them consists of employing the first 20% of the data without taking into consideration the threshold ${\tau }_{\max }$, hence obtaining an estimate of the error behavior. For the remaining iterations, it is calculated by

$$\sqrt{{\tau }_{\max }}=\mathbb{E}[e\left(k\right)/{\sigma }_e]+3\mathrm{Var}[e\left(k\right)/{\sigma }_e].$$

(14)

Since the expression (5) represents a Gaussian distribution, we can use the empirical rule given by Equation (14) to identify the values that exceed the threshold as outliers.

Under the considered assumptions regarding the error distribution, $\mathbb{E}\left\{e\right(k\left)\right\}=0$ and thus,

$${\sigma }_e^2=\mathbb{E}[e^2\left(k\right)]=\xi \left(k\right)$$

(15)

where $\xi \left(k\right)$ for, $k\to \infty$ is the steady-state MSE obtained by employed algorithm. The expression of $\xi \left(k\right)$ depends on the filter application and the algorithm employed. In the following subsections, we compute the steady-state MSE for some adaptive filter applications.

Alternatively, the error variance can be estimated by

$${\sigma }_e^2=(1-b)e^2\left(k\right)+\left(b\right)e^2(k-1),$$

(16)

where b is a forgetting factor.

Although not discussed here, for the cases the error distribution is not Gaussian we can determine the threshold based on measured data through the evaluation of tail probabilities, see [17]. It is also worth mentioning that in particular applications, such as in medical data like ECG, the outlier threshold might affect the main feature to be observed since it resembles an outlier behavior.

2.1. Equalization

In the equalization application, the desired signal is a delayed version of the input

$$d\left(k\right)=s(k-l)$$

(17)

where l represents the delay. The adaptive filter output is written as

$$y\left(k\right)={\mathbf{w}}^T\left(k\right)\mathbf{x}\left(k\right)={\mathbf{w}}^T\left(k\right)(\mathbf{H}\mathbf{s}\left(k\right)+\mathbf{n}\left(k\right))$$

(18)

where $\mathbf{H}\in {\mathbb{R}}^{N\times L}$ is the finite impulse response (FIR) channel convolution matrix, $\mathbf{s}\left(k\right)=[s_0\left(k\right)s_1\left(k\right)\cdots s_{L-1}\left(k\right)]\in {\mathbb{R}}^L$ is the received signal and the channel noise, $\mathbf{n}\left(k\right)=[n_0\left(k\right)n_1\left(k\right)\cdots n_{N-1}\left(k\right)]\in {\mathbb{R}}^N$, is drawn from an independent Gaussian distribution with zero mean and variance ${\sigma }_n^2$. Therefore we can express the MSE as

$$\begin{array}{cc}\hfill \xi \left(k\right)& =\mathbb{E}[e^2\left(k\right)]=\mathbb{E}[{(s(k-l)-y\left(k\right))}^2]\hfill \\ & ={\sigma }_s^2-2\mathbb{E}[s(k-l)\left({\mathbf{w}}^T\left(k\right)(\mathbf{H}\mathbf{s}\left(k\right)+\mathbf{n}\left(k\right))\right)]+\mathbb{E}[{\left({\mathbf{w}}^T\left(k\right)(\mathbf{H}\mathbf{s}\left(k\right)+\mathbf{n}\left(k\right))\right)}^2]\hfill \\ & ={\sigma }_s^2-2{\mathbf{w}}^T\left(k\right)\mathbf{H}\mathbb{E}[s(k-l)\mathbf{s}\left(k\right)]+{\mathbf{w}}^T\left(k\right)\mathbf{H}\mathbb{E}[\mathbf{s}\left(k\right){\mathbf{s}}^T\left(k\right)]{\mathbf{H}}^T\mathbf{w}\left(k\right)+{\mathbf{w}}^T\left(k\right)\mathbb{E}[\mathbf{n}\left(k\right){\mathbf{n}}^T\left(k\right)]\mathbf{w}\left(k\right)\hfill \\ & ={\sigma }_s^2-2{\mathbf{w}}^T\left(k\right)\mathbf{H}{\mathbf{r}}_l+{\mathbf{w}}^T\left(k\right)(\mathbf{H}\mathbf{R}{\mathbf{H}}^T+{\mathbf{I}}_N{\sigma }_n^2)\mathbf{w}\left(k\right)\hfill \\ & \approx {\sigma }_s^2(1-2{\mathbf{w}}^T\left(k\right){\mathbf{h}}_l+{\mathbf{w}}^T\left(k\right)\mathbf{H}{\mathbf{H}}^T\mathbf{w}\left(k\right))+{\sigma }_n^2{\mathbf{w}}^T\left(k\right)\mathbf{w}\left(k\right)\hfill \end{array}$$

(19)

where $\mathbf{R}$ is the autocorrelation matrix of the input signal, ${\mathbf{r}}_l$ is the l-th column of the autocorrelation matrix, and ${\mathbf{h}}_l=\mathbf{H}{\mathbf{r}}_l$. We are assuming that the inputs and the additional noise are uncorrelated.

Assuming the channel model is unknown, the practical way to compute the data-selection threshold is to estimate the output error variance through (16).

2.2. Signal Enhancement

In the signal enhancement case, the desired signal is a signal of interest corrupted by noise,

$$d\left(k\right)=s\left(k\right)+n_1\left(k\right).$$

(20)

By using another noise correlated with the noise that impairs $s\left(k\right)$ as the adaptive filter input,

$$\mathbf{x}\left(k\right)={\mathbf{n}}_2\left(k\right),$$

(21)

the conventional error signal $e\left(k\right)$ will be an enhancement version of $d\left(k\right)$ and the adaptive filter output $y\left(k\right)$ will be the actual error. For this reason, in this signal enhancement case, the MSE is calculated based on the variance of $y\left(k\right)$ instead of $e\left(k\right)$. Hence, the MSE expression for signal enhancement is obtained as

$$\xi \left(k\right)={\sigma }_y^2=\mathbb{E}[y^2\left(k\right)]=\mathbb{E}[{\left({\mathbf{w}}^T\right(k\left){\mathbf{n}}_2\left(k\right)\right)}^2]={\sigma }_{n_2}^2\left(k\right)||\mathbf{w}{\left(k\right)||}_2^2.$$

(22)

2.3. Signal Prediction

In the signal prediction case, the desired signal is a delayed version $x(k+L)$ of the input signal $x\left(k\right)$. Therefore, the error signal is

$$e\left(k\right)=x(k+L)-{\mathbf{w}}^T\left(k\right)\mathbf{x}\left(k\right),$$

(23)

and the MSE expression

$$\xi \left(k\right)=\mathbb{E}[e^2\left(k\right)]=\mathbb{E}[{(x(k+L)-{\mathbf{w}}^T\left(k\right)\mathbf{x}\left(k\right))}^2]$$

(24)

give rise to a expression for the minimum MSE:

$${\xi }_{\min }\left(k\right)=r\left(0\right)-{\mathbf{w}}_{\mathrm{o}}^T\left[\begin{array}{c}r\left(L\right)\\ r(L+1)\\ ⋮\\ r(L+N)\end{array}\right]$$

(25)

where ${\mathbf{w}}_{\mathrm{o}}$ is the optimal coefficients of the predictor and $r\left(l\right)=\mathbb{E}[x(k\left)x\right(k-l)]$. Since in the prediction case $\xi \left(k\right)={\sigma }_e^2\approx {\xi }_{\min }$, Equation (25) can be used to obtain an estimate of ${\sigma }_e^2$ at iteration k by replacing ${\mathbf{w}}_{\mathrm{o}}^T$ by $\mathbf{w}\left(k\right)$ which are the coefficients of the adaptive filter at iteration k. We can estimate $r\left(l\right)$ through $r\left(l\right)=\zeta r(l-1)+(1-\zeta )x\left(k\right)x(k-l)$ in which $\zeta$ is a forgetting factor.

2.4. System Identification

In the system identification, the desired signal can be formulated as

$$d\left(k\right)={{\mathbf{w}}_{\mathrm{o}}}^T\mathbf{x}\left(k\right)+n\left(k\right)$$

(26)

where ${\mathbf{w}}_{\mathrm{o}}$ is the optimal coefficient, $\mathbf{x}\left(k\right)$ is the input vector and $n\left(k\right)$ is the noise drawn from AWGN with zero mean and variance ${\sigma }_n^2$. Therefore, the MSE can be expressed as:

$$\begin{array}{cc}\hfill \xi \left(k\right)& =\mathbb{E}[e^2\left(k\right)]=\mathbb{E}[n^2\left(k\right)]-2\mathbb{E}[n\left(k\right)\Delta {\mathbf{w}}^T\left(k\right)\mathbf{x}\left(k\right)]\hfill \\ & +\mathbb{E}[\Delta {\mathbf{w}}^T\left(k\right)\mathbf{x}\left(k\right){\mathbf{x}}^T\left(k\right)\Delta \mathbf{w}\left(k\right)]\hfill \end{array}$$

(27)

where we define $\Delta \mathbf{w}\left(k\right)=\mathbf{w}\left(k\right)-{\mathbf{w}}_{\mathrm{o}}$. Assuming that the noise and coefficients are uncorrelated, the second term in (27) is zero and we get the following expression

$$\xi \left(k\right)={\sigma }_n^2+{\xi }_{\mathrm{exc}}\left(k\right)$$

(28)

where ${\xi }_{\mathrm{exc}}\left(k\right)$ is the excess MSE and $\mathbb{E}[n^2\left(k\right)]={\sigma }_n^2$. As excess MSE tends to zero, the MSE expression for system identification, previously mentioned in (11), is rewritten as

$$\xi \left(k\right)={\sigma }_e^2=(1+\rho ){\sigma }_n^2.$$

(29)

3. Data-Selective Adaptive Filtering Algorithms

We consider the Newton-based methods LMSN and LMSQN as well as the online CG algorithm to solve the objective function

$$\xi =\frac{1}{2}\mathbb{E}[e^2\left(k\right)].$$

(30)

The Newton-based methods follow a second-order approximation of the objective function and hence perform the coefficient updating as

$$\mathbf{w}(k+1)=\mathbf{w}\left(k\right)-\mu \left(k\right){\widehat{\mathbf{R}}}^{-1}{\widehat{\mathbf{g}}}_{\mathbf{w}}\left(k\right),$$

(31)

where $\widehat{\mathbf{R}}$ and ${\widehat{\mathbf{g}}}_{\mathbf{w}}\left(k\right)$ are estimates of the Hessian and gradient of the objective function, respectively. In fact, the LMSN and LMSQN minimize the objective function in (3).

The CG method, on the other hand, falls in between steepest descent and Newton methods. In the CG algorithm, the search is performed along conjugate directions which produces generally faster convergence than steepest descent methods. The coefficient updating is performed as

$$\mathbf{w}(k+1)=\mathbf{w}\left(k\right)-\alpha \left(k\right)\mathbf{c}\left(k\right),$$

(32)

where the conjugate directions $\mathbf{c}\left(k\right)$ will be explained in more details in Section 3.2.

3.1. LMSN and LMSQN

Considering the same estimate of the gradient, ${\widehat{\mathbf{g}}}_{\mathbf{w}}\left(k\right)=-e\left(k\right)\mathbf{x}\left(k\right)$, used in the LMS algorithm and also a variable step-size, we end-up with the following recursive coefficient updating formula [2]

$$\mathbf{w}(k+1)=\mathbf{w}\left(k\right)+\frac{\nu }{{\mathbf{x}}^T\left(k\right){\widehat{\mathbf{R}}}^{-1}\left(k\right)\mathbf{x}\left(k\right)}{\widehat{\mathbf{R}}}^{-1}\left(k\right)\mathbf{x}\left(k\right)e\left(k\right),$$

(33)

where $\mu \left(k\right)=\nu /({\mathbf{x}}^T\left(k\right){\widehat{\mathbf{R}}}^{-1}\left(k\right)\mathbf{x}\left(k\right))$ is a step-size parameter in which $\nu$ is a positive constant and $e\left(k\right)=d\left(k\right)-{\mathbf{w}}^T\left(k\right)\mathbf{x}\left(k\right)$ is the a priori error.

The only difference between LMSN and the LMSQN algorithms is the way of matrix ${\widehat{\mathbf{R}}}^{-1}\left(k\right)$ is estimated. In the LMSN method, matrix $\widehat{\mathbf{R}}\left(k\right)$ is estimated via a Robbins-Monro procedure resulting in the following update of its inverse, given by [2]

$$\begin{array}{cc}\hfill {\widehat{\mathbf{R}}}^{-1}\left(k\right)=& \frac{1}{1-\theta }\left\{{\widehat{\mathbf{R}}}^{-1}(k-1)-\frac{{\widehat{\mathbf{R}}}^{-1}(k-1)\mathbf{x}\left(k\right){\mathbf{x}}^T\left(k\right){\widehat{\mathbf{R}}}^{-1}(k-1)}{\frac{1-\theta }{\theta }+{\mathbf{x}}^T\left(k\right){\widehat{\mathbf{R}}}^{-1}\left(k\right)\mathbf{x}\left(k\right)}\right\},\hfill \end{array}$$

(34)

where $\theta$ is a weight factor. The LMSQN algorithm updates matrix ${\widehat{\mathbf{R}}}^{-1}\left(k\right)$ by using the approach in [3] which ensures that ${\widehat{\mathbf{R}}}^{-1}\left(k\right)$ remains positive definite and bounded for a bounded input signal. As a result, the estimate ${\widehat{\mathbf{R}}}^{-1}\left(k\right)$ is obtained as

$$\begin{array}{cc}\hfill {\widehat{\mathbf{R}}}^{-1}\left(k\right)=& {\widehat{\mathbf{R}}}^{-1}(k-1)+\left(\frac{\nu }{2{\mathbf{x}}^T\left(k\right){\widehat{\mathbf{R}}}^{-1}(k-1)\mathbf{x}\left(k\right)}-1\right)\nu \frac{{\widehat{\mathbf{R}}}^{-1}(k-1)\mathbf{x}\left(k\right){\mathbf{x}}^T\left(k\right){\widehat{\mathbf{R}}}^{-1}(k-1)}{{\mathbf{x}}^T\left(k\right){\widehat{\mathbf{R}}}^{-1}(k-1)\mathbf{x}\left(k\right)}.\hfill \end{array}$$

(35)

These algorithms have been analyzed more deeply through the use of theory and simulations in [2,3]. Although the estimate obtained by LMSN is accurate enough, it is not free of possible instability behavior. On the other hand, the LMSQN guarantees stability but can lead to poor estimations of ${\mathbf{R}}^{-1}$.

Since the LMSN and LMSQN algorithms update the filter coefficients in the same manner, both utilizing estimates of ${\mathbf{R}}^{-1}$, the excess MSE is also the same [3]. Hence, as proposed in [14], the expression of the excess MSE at steady-state for both DS-LMSN and DS-LMSQN algorithms can be written as the following approximation

$${\xi }_{exc}\left(k\right)\approx \frac{\nu P_{\mathrm{up}}}{2-\nu P_{\mathrm{up}}}{\sigma }_n^2,k\to \infty .$$

(36)

As a result, we use a similar procedure to the one described in [18], in which the coefficient update of the DS-LMSN and DS-LMSQN algorithms are equivalent concerning their expected values. Hence, using this new update and by following the theoretical analysis demonstrated in [2], we obtain an approximation of the excess MSE at the steady-state in both algorithms. Thus, the value $\rho$ can be obtained from the expression (36).

The steps of both DS-LMSN and DS-LMSQN algorithms are summarized in Algorithm 1, where the quantities $\mathbf{t}\left(k\right)$ and $\psi \left(k\right)$ are included to simplify some steps in the computations.

Algorithm 1 Data-Selective LMSN and LMSQN algorithms

DS-LMSN and DS-LMSQN algorithms

Initialize

$0<\nu \le 1$, $0<\theta \le 1$ (for LMSN), $\gamma$ small positive constant,

$\mathbf{w}\left(0\right)=$ random vectors or zero vectors and ${\widehat{\mathbf{R}}}^{-1}\left(0\right)=\gamma {\mathbf{I}}_{L+1}$

Prescribe $P_{\mathrm{up}}$ and choose ${\tau }_{\max }$

$\sqrt{\tau }=\sqrt{(1+\rho )}{Q}^{-1}\left(\frac{P_{\mathrm{up}}}{2}\right)$

For prediction and equalizer use $\rho =0$

For system identification use $\rho =\frac{\nu P_{\mathrm{up}}}{2-\nu P_{\mathrm{up}}}$.

Do for $k>0$

acquire $\mathbf{x}\left(k\right)$ and $d\left(k\right)$

$e\left(k\right)=d\left(k\right)-{\mathbf{w}}^T\left(k\right)\mathbf{x}\left(k\right)$

$\delta \left(k\right)=$$\left\{\begin{array}{c}0,\mathrm{if}-\sqrt{\tau }\le \frac{e\left(k\right)}{{\sigma }_e}\le \sqrt{\tau }\hfill \\ 0,\mathrm{if}\frac{\left|e\right(k\left)\right|}{{\sigma }_e}\ge \sqrt{{\tau }_{\max }}\hfill \\ 1,\mathrm{otherwise}\hfill \end{array}\right.$

if $\delta \left(k\right)=0$

$\mathbf{w}(k+1)=\mathbf{w}\left(k\right)$

if $\frac{\left|e\right(k\left)\right|}{{\sigma }_e}\ge \sqrt{{\tau }_{\max }}$

$e\left(k\right)=0$

$d\left(k\right)=0$

end if

else

$\mathbf{t}\left(k\right)={\widehat{\mathbf{R}}}^{-1}(k-1)\mathbf{x}\left(k\right)$

$\psi \left(k\right)={\mathbf{x}}^T\left(k\right)\mathbf{t}\left(k\right)$

$\mathbf{w}(k+1)=\mathbf{w}\left(k\right)+\nu \frac{\mathbf{t}\left(k\right)e\left(k\right)}{\psi \left(k\right)}$

${\widehat{\mathbf{R}}}^{-1}\left(k\right)=\frac{1}{1-\theta }\left[{\widehat{\mathbf{R}}}^{-1}(k-1)-\frac{\mathbf{t}\left(k\right){\mathbf{t}}^T\left(k\right)}{\frac{1-\theta }{\theta }+\psi \left(k\right)}\right]$, for LMSN

${\widehat{\mathbf{R}}}^{-1}\left(k\right)={\widehat{\mathbf{R}}}^{-1}(k-1)+\nu \frac{\frac{\nu }{2\psi \left(k\right)}-1}{\psi \left(k\right)}\mathbf{t}\left(k\right){\mathbf{t}}^T\left(k\right)$, for LMSQN

end if

3.2. Online Conjugate Gradient

Minimizing the objective function in (30) is equivalent to

$$\underset{\mathbf{w}}{\min }\frac{1}{2}{\mathbf{w}}^T(k+1)\mathbf{R}\mathbf{w}(k+1)-{\mathbf{p}}^T\mathbf{w}(k+1)$$

(37)

in which $\mathbf{R}=\mathbb{E}[\mathbf{x}\left(k\right){\mathbf{x}}^T\left(k\right)]$ is the $N\times N$ autocorrelation matrix of the input signal and $\mathbf{p}=\mathbb{E}[d(k\left)\mathbf{x}\right(k)]$ is the cross-correlation vector between the input and reference signals. Similarly, our goal is to solve the linear equation

$$\mathbf{R}\mathbf{w}(k+1)=\mathbf{p}.$$

(38)

The CG method can solve this problem by expressing the solution

$${\mathbf{w}}_{\mathrm{o}}=\sum _{i=0}^{N-1}\alpha \left(i\right)\mathbf{c}\left(i\right)$$

(39)

in a basis formed by a set of vectors ${\mathbf{c}}_i,i\in \{0,\dots N-1\}$ that present $\mathbf{R}$-conjugacy, that is, ${\mathbf{c}}^T\left(i\right)\mathbf{R}\mathbf{c}\left(j\right)=0$ for all $i\ne j$. By premultiplying Equation (39) by ${\mathbf{c}}^T\left(k\right)\mathbf{R}$ and using conjugate definition:

$$\begin{array}{ccc}\hfill {\mathbf{c}}^T\left(k\right)\mathbf{R}{\mathbf{w}}_{\mathrm{o}}& =& {\mathbf{c}}^T\left(k\right)\mathbf{R}\left(\sum _{i=0}^{N-1}\alpha \left(i\right)\mathbf{c}\left(i\right)\right)\hfill \\ & =& \sum _{i=0}^{N-1}\alpha \left(i\right)\left({\mathbf{c}}^T\left(k\right)\mathbf{R}\mathbf{c}\left(i\right)\right)\hfill \\ & =& \alpha \left(k\right)\left({\mathbf{c}}^T\left(k\right)\mathbf{R}\mathbf{c}\left(k\right)\right).\hfill \end{array}$$

(40)

By replacing $\mathbf{R}{\mathbf{w}}_{\mathrm{o}}=\mathbf{p}$ in (40), we obtain an expression for the constant $\alpha$ at the kth iteration:

$$\alpha \left(k\right)=\frac{{\mathbf{c}}^T\left(k\right)\mathbf{p}}{{\mathbf{c}}^T\left(k\right)\mathbf{R}\mathbf{c}\left(k\right)}.$$

(41)

Equation (39) can be evaluated as an iterative process in which a portion $\alpha \left(k\right)\mathbf{c}\left(k\right)$ is added at the kth step:

$$\mathbf{w}(k+1)=\mathbf{w}\left(k\right)+\alpha \left(k\right)\mathbf{c}\left(k\right).$$

(42)

As observed in [19], the estimation of the matrix $\mathbf{R}$ and vector $\mathbf{p}$ can be both computed using the exponentially decaying window, giving rise to Equations (43) and (44), respectively.

$$\mathbf{R}\left(k\right)=\lambda \mathbf{R}(k-1)+\mathbf{x}\left(k\right){\mathbf{x}}^T\left(k\right)$$

(43)

$$\mathbf{p}\left(k\right)=\lambda \mathbf{p}(k-1)+d\left(k\right)\mathbf{x}\left(k\right)$$

(44)

Both estimations are also employed in RLS algorithm where $\lambda$ represents a forgetting factor.

By applying the line search method as done in [19], another expression for $\alpha \left(k\right)$ can be achieved:

$$\alpha \left(k\right)=\eta \frac{{\mathbf{c}}^T\left(k\right)\mathbf{g}(k-1)}{{\mathbf{c}}^T\left(k\right)\mathbf{R}\mathbf{c}\left(k\right)}$$

(45)

with $(\lambda -0.5)\le \eta \le \lambda$ to assure convergence. From Equations (42)–(44), we can obtain another expression for the negative gradient $\mathbf{g}\left(k\right)$:

$$\begin{array}{ccc}\hfill \mathbf{g}\left(k\right)& =& \mathbf{p}\left(k\right)-\mathbf{R}\left(k\right)\mathbf{w}(k+1)\hfill \\ & =& \lambda \mathbf{p}(k-1)+d\left(k\right)\mathbf{x}\left(k\right)\hfill \\ & & -[\lambda \mathbf{R}(k-1)+\mathbf{x}\left(k\right){\mathbf{x}}^T\left(k\right)][\mathbf{w}\left(k\right)+\alpha \left(k\right)\mathbf{c}\left(k\right)]\hfill \\ & =& \lambda \mathbf{g}(k-1)-\alpha \left(k\right)\mathbf{R}\left(k\right)\mathbf{c}\left(k\right)+\mathbf{x}\left(k\right)e\left(k\right)\hfill \end{array}$$

(46)

where $e\left(k\right)=d\left(k\right)-{\mathbf{x}}^T\left(k\right)\mathbf{w}\left(k\right)$.

The next conjugate direction $\mathbf{c}(k+1)$ can be obtained as the current negative gradient $\mathbf{g}\left(k\right)$ corrected by a term comprising a linear combination of the previous direction vectors:

$$\mathbf{c}(k+1)=\mathbf{g}\left(k\right)+\beta \left(k\right)\mathbf{c}\left(k\right)$$

(47)

in which

$$\beta \left(k\right)=\frac{{(\mathbf{g}\left(k\right)-\mathbf{g}(k-1))}^T\mathbf{g}\left(k\right)}{{\mathbf{g}}^T(k-1)\mathbf{g}(k-1)}$$

(48)

is a constant calculated to guarantee $\mathbf{R}$-conjugacy and improve performance as well.

As analyzed in [15], the CG and RLS algorithms are equivalent in steady-state and hence the excess MSE is also equivalent,

$${\xi }_{exc}\left(k\right)\approx (N+1)\frac{P_{\mathrm{up}}(1-\lambda )}{2-P_{\mathrm{up}}(1-\lambda )}{\sigma }_n^2,k\to \infty .$$

(49)

in which the derivation is detailed in [1]. Thus, we can obtain $\rho$ from Equation (36). As a result, we obtain the DS-CG summarized in Algorithm 2.

Algorithm 2 Data-Selective Conjugate Gradient algorithm

DS-CG algorithm

Initialize

$\lambda ,\eta \mathrm{with}(\lambda -0.5)\le \eta \le \lambda$, $\mathbf{w}\left(0\right)=$ random vectors or zero vectors

$R_0=\mathbf{I}$, $\mathbf{g}\left(0\right)=\mathbf{c}\left(1\right)=zeros(N+1,1)$, $\gamma =$ small constant for regularization

Prescribe $P_{\mathrm{up}}$, and choose ${\tau }_{\max }$

$\sqrt{\tau }=\sqrt{(1+\rho )}{Q}^{-1}\left(\frac{P_{\mathrm{up}}}{2}\right)$

For prediction and equalizer use $\rho =0$

For system identification use $\rho =(N+1)\frac{P_{\mathrm{up}}(1-\lambda )}{2-P_{\mathrm{up}}(1-\lambda )}$.

Do for $k>0$

acquire $\mathbf{x}\left(k\right)$ and $d\left(k\right)$

$e\left(k\right)=d\left(k\right)-{\mathbf{w}}^T\left(k\right)\mathbf{x}\left(k\right)$

$\delta \left(k\right)=$$\left\{\begin{array}{c}0,\mathrm{if}-\sqrt{\tau }\le \frac{e\left(k\right)}{{\sigma }_e}\le \sqrt{\tau }\hfill \\ 0,\mathrm{if}\frac{\left|e\right(k\left)\right|}{{\sigma }_e}\ge \sqrt{{\tau }_{\max }}\hfill \\ 1,\mathrm{otherwise}\hfill \end{array}\right.$

if $\delta \left(k\right)=0$

$\mathbf{w}(k+1)=\mathbf{w}\left(k\right)$

if $\frac{\left|e\right(k\left)\right|}{{\sigma }_e}\ge \sqrt{{\tau }_{\max }}$

$e\left(k\right)=0$

$d\left(k\right)=0$

end if

else

$\mathbf{R}\left(k\right)=\lambda \mathbf{R}(k-1)+\mathbf{x}\left(k\right){\mathbf{x}}^T\left(k\right)$

$\alpha \left(k\right)=\eta \frac{{\mathbf{c}}^T\left(k\right)\mathbf{g}(k-1)}{[{\mathbf{c}}^T\left(k\right)\mathbf{R}\left(k\right)\mathbf{c}\left(k\right)+\gamma ]}$

$\mathbf{w}(k+1)=\mathbf{w}\left(k\right)+\alpha \left(k\right)\mathbf{c}\left(k\right)$

$\mathbf{g}\left(k\right)=\lambda \mathbf{g}(k-1)-\alpha \left(k\right)\mathbf{R}\left(k\right)\mathbf{c}\left(k\right)+\mathbf{x}\left(k\right)e\left(k\right)$

$\beta \left(k\right)=\frac{{[\mathbf{g}\left(k\right)-\mathbf{g}(k-1)]}^T\mathbf{g}\left(k\right)}{[{\mathbf{g}}^T(k-1)\mathbf{g}(k-1)+\gamma ]}$

$\mathbf{c}(k+1)=\mathbf{g}\left(k\right)+\beta \left(k\right)\mathbf{c}\left(k\right)$

end if

4. Simulation Results

In this section, we present simulations utilizing both synthetic and real-world data for the algorithms explained in the previous section in order to verify the impact on the performance when the data selection method is applied. Moreover, the desired probability of updating $P_{\mathrm{up}}$ is varied between 0% and 100%, and it is compared to the measured probability of update ${\widehat{P}}_{\mathrm{up}}$.

4.1. Simulation 1: Equalizer

In this subsection, the channel we want to equalize is one of the FIR channel impulse responses provided by The Signal Processing Information Base repository [20]. The complex channel taps were obtained from digital microwave radio systems measurements, and the FIR model frequency response is illustrated in black in Figure 2a. The transmitted signal $\mathbf{s}\left(k\right)$, modeled as realizations of a Gaussian random variable with ${\sigma }_s^2=1$, transverses through the channel and it is corrupted by additive Gaussian noise with ${\sigma }_n^2={10}^{-3}$. The adaptive filter performs the equalization and its output is an equalized version of $\mathbf{s}\left(k\right)$. Each complex version of the data-selective algorithm is applied and their frequency responses try to invert the channel behavior as illustrated in Figure 2a, for $P_{\mathrm{up}}=1$ and $P_{\mathrm{up}}=0.45$. We used $\theta =9\times {10}^{-4}$, $\gamma =1$ and $\nu =0.05$ for both LMSN and LMSQN. For CG, we used $\lambda =0.9995$ and $\eta =0.48$. The filter order is $N=100$. The error variance was estimated as in Equation (16) for $b=0.9999$. Since the channel coefficients are complex values, the threshold is computed as

$$\tau =2(1+\rho )\ln \left(\frac{1}{P_{\mathrm{up}}}\right)$$

(50)

where $\rho =0$ [1]. The estimated probability of updating obtained by each algorithm is quite close the prescribed $P_{\mathrm{up}}$, as depicted in Figure 2b. As can be seen in Figure 3, it is possible to obtain the transmitted signal with only $45\%$ of the input data with almost the same accuracy obtained when $100\%$ of the input data are used.

4.2. Simulation 2: Prediction

The dataset used in this subsection is taken from anemometer readings provided by GoogleâAZs RE < C Initiative [21]. The data consists of the wind’s speed recorded by five sensors on 25 May 2011. The dataset is split into 40 sets of size 8192 in order to use the Monte Carlo method.

The performance of the MSE is verified in Figure 4a using $P_{\mathrm{up}}=0.4$. The parameters for the DS-CG algorithm is $\lambda =0.98$ and $\eta =0.48$. Both the DS-LMSN and DS-LMSQN algorithms employed $\nu =0.1$ and LMSN utilized $\theta =0.1$. All algorithms obtain a similar convergence with 40 independent runs, but the DS-LMSQN algorithm achieves better performance due to its faster convergence to the steady state. The employed adaptive filter order is $N=7$.

In Figure 4b, the prescribed and observed probabilities of update are compared. We can observe that all DS algorithms obtained an observed probability of update close to the prescribed one.

The output of the prediction is illustrated in Figure 5a for $P_{\mathrm{up}}=0.4$ and Figure 5b for $P_{\mathrm{up}}=0.7$ between iterations 8000 and 8150. In both the cases, it was observed an acceptable performance in the prediction, leading us to conclude that even if we perform a reduced number of updates, data selection algorithms achieve an accurate prediction.

4.3. Simulation 3: System Identification

In this simulation, our problem is to identify an unknown channel impulse response, described as:

$$\mathbf{h}={\left[0.10100.30300-0.2020-0.4040-0.7071-0.4040-0.2020\right]}^T.$$

(51)

The unknown system output is written as $d\left(k\right)={\mathbf{H}}^T\mathbf{x}\left(k\right)+n\left(k\right)$, where $n\left(k\right)$ is a Gaussian noise with zero mean and variance ${\sigma }_n^2={10}^{-3}$. We consider two cases of input signals: a first-order and a fourth-order AR process, given by

$$\begin{array}{cc}\hfill x\left(k\right)=& 0.88x(k-1)+n_1\left(k\right),\hfill \\ \hfill x\left(k\right)=& -0.55x(k-1)-1.221x(k-2)-0.49955x(k-3)\hfill \\ & -0.4536x(k-1)+n_2\left(k\right),\hfill \end{array}$$

where $n_1\left(k\right)$ and $n_2\left(k\right)$ are samples from a Gaussian noise uncorrelated with the additional noise $n\left(k\right)$. The variances ${\sigma }_{n_1}^2$ and ${\sigma }_{n_2}^2$ are set such as the input signal is of unit variance. The parameters employed in the system identification problem for conjugate gradient are $\lambda =0.98$ and $\eta =0.48$, for both DS-LMSN and DS-LMSQN we set $\nu =0.1$ and for DS-LMSN $\theta =0.1$. The filter order is $N=7$ with the purpose of ensuring the convergence of the filter coefficients to the optimal coefficients due to channel size.

The learning curves of the algorithms are compared in Figure 6a for a prescribed probability of update $P_{\mathrm{up}}=0.4$ and first-order AR input signal. It can be noted that all algorithms achieve good performance even with a smaller amount of update. The DS-CG algorithm attains better performance than the DS-LMSN and DS-LMSQN since it converges faster to the steady-state. The observed ${\widehat{P}}_{\mathrm{up}}$ and the prescribed $P_{\mathrm{up}}$ probabilities of update are depicted in Figure 6b. In all cases, these values are close, except at low values of $P_{\mathrm{up}}$ where we obtained a bit more updates than the ones prescribed. By using a fourth-order AR process as input signal in Figure 7, the results are similar to the first-order AR, confirming the expected robustness with respect to the statistical properties of the input signal, as long as the rank of its autocorrelation matrix does not become too small.

In another example utilizing a fourth-order AR input signal, we included an outlier signal affecting the reference signal 1% of the time with an amplitude equal to five. The desired $P_{\mathrm{up}}=0.3$ was set, and we measured the misalignment in the adaptive filter coefficients, defined as $\frac{\parallel \mathbf{w}\left(k\right)-{\mathbf{w}}_o\parallel }{\parallel {\mathbf{w}}_o\parallel }$ where ${\mathbf{w}}_o$ represents the optimal vector of coefficients, for the algorithms discussed in the paper. As observed in

Table 1. Misalignment with outliers, in dBs.

	Outlier	Yes	Yes	Yes	No
	${\tau }_{\max }$ on	yes	no	yes	no
	$P_{\mathrm{up}}$	0.3	0.3	0.1	1
	DS-CG	−33.29	−15.25	−30.47	−33.37
Average	DS-LMSQN	−32.75	−15.42	−32.45	−32.80
Misalignment (dB)	DS-LMSN	−31.17	−13.81	−30.39	−31.91

, the misalignment is higher when ignoring the outliers and that the level of misalignment achieved by considering outliers for $P_{\mathrm{up}}=0.1$ matches the one for $P_{\mathrm{up}}=0.3$ addressing the outliers. It is also possible to verify that the proposed solutions approach the solution when the algorithms are updated all the time.

5. Conclusions

In this work, the data-selective versions of the LMSN, LMSQN and CG algorithms were explored in different applications. The key idea is providing a systematic form to prescribe the probability of update through a simple statistical model, where the environment data is classified as innovative, non-innovative, and outlier. The data-selection can not only reduce the computational complexity but also enhance the estimation accuracy when outliers are present. Simulation results on both real and synthetic data show that the data selection strategy works for all types of applications of adaptive filtering. Future work will address the extension of the data-selection approach to a broader class of learning algorithms as well as its effectiveness in distributed adaptive networks.