A Soft Parameter Function Penalized Normalized Maximum Correntropy Criterion Algorithm for Sparse System Identification

Abstract

A soft parameter function penalized normalized maximum correntropy criterion (SPF-NMCC) algorithm is proposed for sparse system identification. The proposed SPF-NMCC algorithm is derived on the basis of the normalized adaptive filter theory, the maximum correntropy criterion (MCC) algorithm and zero-attracting techniques. A soft parameter function is incorporated into the cost function of the traditional normalized MCC (NMCC) algorithm to exploit the sparsity properties of the sparse signals. The proposed SPF-NMCC algorithm is mathematically derived in detail. As a result, the proposed SPF-NMCC algorithm can provide an efficient zero attractor term to effectively attract the zero taps and near-zero coefficients to zero, and, hence, it can speed up the convergence. Furthermore, the estimation behaviors are obtained by estimating a sparse system and a sparse acoustic echo channel. Computer simulation results indicate that the proposed SPF-NMCC algorithm can achieve a better performance in comparison with the MCC, NMCC, LMS (least mean square) algorithms and their zero attraction forms in terms of both convergence speed and steady-state performance.

1. Introduction

In nature, the impulse response of most unknown systems can be regarded as sparse, which consists of only a few dominant coefficients [1,2,3,4]. The prior known sparse information can be used for improving the estimation performance in signal processing. Thus, sparse signal processing has been garnering significant attention in recent decades [1,2,3,4,5]. In particular, the developed sparse signal processing techniques have been used in wireless communications, speech signal processing and imaging processing, which include compressed sensing (CS) [6,7,8] and sparse adaptive filtering [9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]. Although compressed sensing algorithms are useful for dealing with sparse signals, they need to construct measurement matrices with the limitation of restricted isometry property (RIP) [31]. Thus, the computational complexity might be higher in comparison with sparse adaptive filtering algorithms [21,22,23].

Fueled by the demand of sparse signal processing, sparse adaptive filter algorithms have been widely studied for sparse system identification, echo cancellation and channel estimation [9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]. These proposed sparse adaptive filtering algorithms can be mainly categorized into two groups: proportionate-type adaptive filters [17,18,19,20] and zero-attracting adaptive filters [9,10,11,12,13,14,15,16,21,22,23,24,25,26,27,28,29,30]. As we know, the proportionate-type adaptive filter algorithms assign proportional step sizes to different coefficients on the basis of their magnitudes, such as proportionate normalized least mean square (PNLMS) [17,18,19], proportionate affine projection algorithms (PAPA) [32] and their variants [19]. Other effective sparse adaptive filters have been reported as motivated by the least absolute shrinkage and selection operator (LASSO) [6] and compressed sensing [7,8], which are known as zero attracting adaptive filters. Firstly, an $l_1$-norm penalty has been incorporated into the cost function of the traditional least mean square (LMS) to form the zero-attracting LMS (ZA-LMS) algorithm [9]. As a result, the ZA-LMS algorithm can produce an additional term which acts as a zero attractor to make the ZA-LMS algorithm converge quickly. However, the shrinkage of the ZA-LMS algorithm cannot distinguish between the zero coefficients and non-zero coefficients [9]. Furthermore, the ZA-LMS algorithm gives a penalty to all the coefficients uniformly, and, hence, its performance may be deteriorated for less sparse systems. Then, a reweighted ZA-LMS (RZA-LMS) algorithm is presented by introducing a log-sum function into the cost function of the conventional LMS algorithm [9]. By using this log-sum function, the zero attractor in the RZA-LMS algorithm can provide a selective shrink according to the magnitudes of the coefficients [9]. However, these sparse LMS algorithms are sensitive to the scaling of the input training signal because of the drawback of the LMS algorithm. Recently, the proposed zero attracting techniques have been expanded to high-order-moment (HOM) algorithms to reduce the effects of the scaling of input signal, which include least mean fourth (LMF) algorithm [12,13,23,33], least mean square/fourth (LMS/F) algorithm [24,25,26,34], least-mean mixture-norm (LMMN) algorithm [35,36,37] and affine projection (AP) algorithms [14,15,16,38,39]. However, some of these sparse algorithms have high computational complexity. New sparse adaptive filtering algorithms should provide a balance parameter to give a trade-off between the convergence and steady-state performance. Another effective method to reduce the effect of the scaling of the input training signal is to utilize a normalized factor to form sparse normalized LMS (NLMS) algorithms [40,41]. In [42], the sparse NLMS algorithm has been used for sparse channel estimation to achieve a better performance. In essence, these adaptive filter algorithms estimate the unknown system by minimizing the instantaneous error which is the difference between the desired signal and the output of the filters.

Recently, information theoretic quantities have been used as cost function to construct adaptive systems. An entropy estimator has been proposed to make the computation of the quantities reliable. After that, a great number of adaptive filtering algorithms have been proposed on the basis of the minimum error entropy (MEE) [43]. By using entropy as a cost function, the signal structure can be well justified. The MEE cost function can provide robustness when dealing with non-Gaussian and impulsive noise signals, but its complexity is higher in comparison with the LMS-like algorithms. High complexity ($O(NL)$), where L is the length of a batch of error samples, is not suitable for practical engineering application such as noise cancellation. Then, a maximum correntropy criterion (MCC) algorithm has been proposed by using localized similarity as a cost function [2,44,45]. As a result, the computational complexity ($O(N)$) of the MCC is comparable to the LMS algorithm, while its robustness is similar to the MEE algorithm in terms of convergence and estimation behavior in impulsive noise environment [2,44,45,46,47,48]. Thus, the MCC is more suitable for practical engineering applications such as noise cancellation in speech data [2,44,45]. However, the MCC algorithm cannot exploit the sparse structure of in-nature systems such as multi-path channels and network echo channels because it indiscriminately handles all the filter coefficients. Inspired by the ZA-LMS and RZA-LMS algorithms, the zero-attracting techniques have been introduced into the traditional MCC algorithm to construct zero-attracting MCC (ZA-MCC) and reweighted ZA-MCC (RZA-MCC) algorithms to exploit the sparsity of the in-nature systems [49].

In this paper, a soft parameter function penalized normalized maximum correntropy criterion (SPF-NMCC) algorithm is proposed based on the normalized adaptive filtering theory, MCC algorithm and zero attracting techniques. The proposed SPF-NMCC algorithm is realized by normalizing the MCC algorithm to form a normalized MCC (NMCC) algorithm that is similar to the NLMS algorithm. Then, a soft parameter function is included into the cost function of the NMCC algorithm to give a desired zero attractor term in the update equation of the SPF-NMCC algorithm. The proposed SPF-NMCC algorithm is derived in detail based on the Lagrange multiplier method, and its performance is investigated for estimating a sparse system and an echo channel. The results obtained from the computer simulations show that the proposed SPF-NMCC algorithm is better than the traditional MCC, ZA-MCC, and RZA-MCC algorithms in terms of the convergence speed rate and the steady-state performance.

The rest of this paper is structured as follows. In Section 2, the traditional MCC algorithm and its sparse forms are reviewed and discussed within the system identification framework. In Section 3, the proposed SPF-NMCC algorithm is described mathematically in the framework of sparse system identification, which is based on the normalized adaptive filtering theory, MCC algorithm and zero attracting theories. In Section 4, the estimation behaviors of the proposed SPF-NMCC algorithm are well investigated in the context of sparse system identifications. Finally, a conclusion of this work is given in Section 5.

2. Traditional MCC Algorithm and ZA Techniques

2.1. Traditional MCC Algorithm

We review the traditional MCC algorithm in the context of sparse system identification. Herein, we are interested in the adaptive estimation vector $\widehat{\mathbf{w}}(n)\in {\mathbb{R}}^N$, which should be similar to the unknown system ${\mathbf{w}}_{\mathbf{o}}(n)\in {\mathbb{R}}^N$, by reducing the difference between the desired signal $d(n)$ and the estimation output $y(n)$. Here,

$$y(n)={\widehat{\mathbf{w}}}^T(n)\mathbf{x}(n),$$

(1)

and

$$d(n)={\mathbf{w}}_o^T(n)\mathbf{x}(n)+v(n),$$

(2)

where $\mathbf{x}(n)={[x(n),x(n-1),\cdots ,x(n-N-1)]}^T$ is the training input signal, ${\mathbf{w}}_{\mathbf{o}}(n)={[w_0,w_1,\cdots ,w_{N-1}]}^T$, and $v(n)$ is an additional Gaussian white noise signal. In general, the instantaneous estimation error can be written as

$$e(n)=d(n)-y(n),$$

(3)

where $e(n)$ is the difference between the reference signal and the adaptive filter output. In practical engineering, ${\mathbf{w}}_{\mathbf{o}}(n)$ may be a sparse multi-path channel, an acoustic channel of a room, an underwater communication channel or a network echo channel, which should be identified by obtaining the echo cancellation. Similarly to the NLMS algorithm [40], the minimization of the difference of the least Euclidian norm is used to get the update equation of the MCC algorithm [2,44,45] under a constraint. Here, the least Euclidian norm of the difference is

$$\widehat{\mathbf{w}}(n+1)-\widehat{\mathbf{w}}(n)=\alpha \widehat{\mathbf{w}}(n+1),$$

(4)

where α is a very small constant. Then, we minimize

$${∥\alpha \widehat{\mathbf{w}}(n+1)∥}^2=\sum _{k=0}^{N-1}(w_k(n+1)-w_k{(n))}^2={∥\widehat{\mathbf{w}}(n+1)-\widehat{\mathbf{w}}(n)∥}^2$$

(5)

under a constraint

$$d(n)-{\widehat{\mathbf{w}}}^T(n+1)\mathbf{x}(n)=0.$$

(6)

According to the previous studies, we consider the update Equations in [2,44,45]. Then, we have

$$\begin{array}{cc}\overline{e}(n)\hfill & =d(n)-{\widehat{\mathbf{w}}}^T(n+1)\mathbf{x}(n)\hfill \\ & =d(n)-{\widehat{\mathbf{w}}}^T(n)\mathbf{x}(n)-\eta exp(-\frac{e^2(n)}{2{\sigma }^2})e(n){\mathbf{x}}^T(n)\mathbf{x}(n)\hfill \\ & =e(n)-\xi exp(-\frac{e^2(n)}{2{\sigma }^2})e(n)\hfill \\ & =\left[1-\xi exp(-\frac{e^2(n)}{2{\sigma }^2})\right]e(n).\hfill \end{array}$$

(7)

Thus, the MCC algorithm is to solve the following problem [2,44,45]

$$\begin{array}{c}min\frac{1}{2}{∥\widehat{\mathbf{w}}(n+1)-\widehat{\mathbf{w}}(n)∥}^2,\hfill \\ \mathrm{subject}\mathrm{to}\overline{e}(n)=\left[1-\xi exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)\right]e(n),\hfill \end{array}$$

(8)

where $\overline{e}(n)=d(n)-{\widehat{\mathbf{w}}}^T(n+1)\mathbf{x}(n)$, ${∥·∥}^2$ is the Euclidean norm of a vector, and $\xi =\chi {∥\mathbf{x}(n)∥}^2$. Here, χ acts as a step-size.

In this paper, we use the Lagrange multiplier method to find the solution of Equation (8). Thus, the cost function of the MCC algorithm can be written as [44]

$$J_{\mathrm{MCC}}(n)=\frac{1}{2}{∥\widehat{\mathbf{w}}(n+1)-\widehat{\mathbf{w}}(n)∥}^2+\lambda \left(\overline{e}(n)-\left[1-\xi exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)\right]e(n)\right),$$

(9)

where λ denotes the Lagrange multiplier. In order to minimize Equation (9), we calculate its gradient with respect to $\widehat{\mathbf{w}}(n+1)$. Then, we have

$$\frac{\partial J_{\mathrm{MCC}}(n)}{\partial \tilde{\mathbf{w}}(n+1)}=0\mathrm{and}\frac{\partial J_{\mathrm{MCC}}(n)}{\partial \lambda }=0.$$

(10)

From Equation (10), we have

$$\widehat{\mathbf{w}}(n+1)=\widehat{\mathbf{w}}(n)+\lambda \mathbf{x}(n),$$

(11)

and we can get Lagrange multiplier, which is written as

$$\lambda =\xi \frac{exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)e(n)}{{∥\mathbf{x}(n)∥}^2}.$$

(12)

By substituting Equation (12) into Equation (11), we have

$$\widehat{\mathbf{w}}(n+1)=\widehat{\mathbf{w}}(n)+\chi exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)e(n)\mathbf{x}(n).$$

(13)

We can see that the MCC algorithm employs an exponential term to eliminate the large errors, rendering it robust against impulsive-like noises.

2.2. Zero Attracting Techniques

As we know, the zero attracting technique has been used to make use of the sparse property of in-nature signals such as wireless multi-path channels and echo channels [9]. Recently, zero attracting methods have been used for exploiting the sparse MCC algorithms [49], resulting in two sparse MCC algorithms named as zero attracting MCC (ZA-MCC) and reweighted ZA-MCC (RZA-MCC) algorithms. The ZA-MCC algorithm is realized by using an $l_1$-norm to modify the cost function of the traditional MCC algorithm, and, hence, the ZA-MCC algorithm is to solve the following problem [49]

$$\begin{array}{c}min\frac{1}{2}{∥\widehat{\mathbf{w}}(n+1)-\widehat{\mathbf{w}}(n)∥}^2+{\gamma }_{\mathrm{ZA}}{∥\widehat{\mathbf{w}}(n+1)∥}_1,\hfill \\ \mathrm{subject}\mathrm{to}\overline{e}(n)=\left[1-\xi exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)\right]e(n).\hfill \end{array}$$

(14)

By using the Lagrange multiplier method to find the solution of Equation (14), we write the cost function as [49]

$$J_{\mathrm{ZA}}(n)=\frac{1}{2}{∥\widehat{\mathbf{w}}(n+1)-\widehat{\mathbf{w}}(n)∥}^2+{\gamma }_{\mathrm{ZA}}{∥\widehat{\mathbf{w}}(n+1)∥}_1+\lambda \left(\overline{e}(n)-\left[1-\xi exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)\right]e(n)\right),$$

(15)

where ${\gamma }_{\mathrm{ZA}}$ is a regularization parameter which is greater than zero, and ${∥·∥}_1$ represents the $l_1$-norm. Then, the updated equation of the ZA-MCC algorithm can be expressed as

$$\widehat{\mathbf{w}}(n+1)=\widehat{\mathbf{w}}(n)+\chi exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)e(n)\mathbf{x}(n)-{\rho }_{\mathrm{ZA}}\mathrm{sgn}\left[\widehat{\mathbf{w}}(n)\right],$$

(16)

where ${\rho }_{\mathrm{ZA}}$ is a zero attracting controlling parameter that is used for balancing the zero attraction ability, and $\mathrm{sgn}\left[x\right]$ is an element-wise sign operator, which is defined as

$$\mathrm{sgn}[x]=\left\{\begin{array}{c}\frac{x}{\left|x\right|},x\ne 0,\hfill \\ 0,x=0.\hfill \end{array}\right.$$

(17)

Comparing with the MCC algorithm, it is found that there is an additional term $-{\rho }_{\mathrm{ZA}}\mathrm{sgn}\left[\widehat{\mathbf{w}}(n)\right]$ which is used to provide a zero attracting effect to make the ZA-MCC algorithm converge more quickly [9,49]. Here, $-{\rho }_{\mathrm{ZA}}\mathrm{sgn}\left[\widehat{\mathbf{w}}(n)\right]$ is also defined as a zero attractor in the ZA algorithms [9,22,49]. Similar to the previously proposed zero-attracting-based sparse LMS algorithms, the ZA-MCC algorithm can be used for dealing with sparse system identification. After that, the RZA-MCC algorithm has been proposed by using a sum-log function instead of the $l_1$-norm to further enhance the ZA-MCC algorithm for handling less sparse systems [49]. The RZA-MCC provides a reweighting step-size in its zero attractor to selectively give penalties on the zero taps rather than uniformly exert penalties on all the taps.

3. Proposed Sparse SPF-NMCC Algorithm

Although the ZA-MCC and RZA-MCC algorithms can well utilize the sparse property of the in-nature systems, they might be affected by scaling of the input signals. As is known to us, normalized adaptive filtering can give resistance to the effects on this scaling, and the zero attracting technique is an effective method to estimate sparse systems. Motivated by the advantages of the normalized adaptive filters and zero attracting techniques, we propose a SPF-NMCC algorithm to further exploit sparseness characteristics for sparse system identification.

Similar to the sparse MCC algorithms, a soft parameter function is used to exploit the sparsity characteristics of these systems, which is defined as

$$S_{\beta }(\widehat{\mathbf{w}}(n))=(1+{\beta }^{-1})(1-e^{-\beta \left|\widehat{\mathbf{w}}(n)\right|}),$$

(18)

where $\beta >0$. The performance of the soft parameter function is shown in Figure 1. It can be seen that the soft parameter function is close to $l_0$-norm when β is very large, while it can be regarded as a $l_1$-norm for small β. We can choose a large β to approximately implement the $l_0$-norm penalty, and we can use small β to construct $l_1$-norm penalty to exploit the sparsity of existing sparse systems.

Then, we integrate this soft parameter function into the cost function of an NMCC algorithm to exploit the sparse characteristics of in-nature systems such as echo channels. Thus, the proposed SPF-NMCC algorithm is to find out a solution of the following problem

$$\begin{array}{c}{\displaystyle min\frac{1}{2}{∥\widehat{\mathbf{w}}(n+1)-\widehat{\mathbf{w}}(n)∥}^2+{\gamma }_{\mathrm{SPF}}{S}_{\beta }(\widehat{\mathbf{w}}(n+1)),}\hfill \\ {\displaystyle \mathrm{subject}\mathrm{to}\overline{e}(n)=\left[1-\xi exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)\right]e(n),}\hfill \end{array}$$

(19)

where ${\gamma }_{\mathrm{SPF}}$ is a regularization parameter that can be tuned to give a tradeoff between sparsity and estimation performance. Then, the cost function of the SPF-NMCC algorithm can be written as

$${\displaystyle J_{\mathrm{SPF}}(n)=\frac{1}{2}{∥\widehat{\mathbf{w}}(n+1)-\widehat{\mathbf{w}}(n)∥}^2+{\gamma }_{\mathrm{SPF}}{S}_{\beta }(\widehat{\mathbf{w}}(n+1))+\lambda \left(\overline{e}(n)-\left[1-\xi exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)\right]e(n)\right).}$$

(20)

By using Lagrange multiplier method on Equation (20), we have

$$\frac{\partial J_{\mathrm{SPF}}(n)}{\partial \widehat{\mathbf{w}}(n+1)}=0,$$

(21)

and

$$\frac{\partial J_{\mathrm{SPF}}(n)}{\partial \lambda }=0.$$

(22)

Then, we have

$${\displaystyle \widehat{\mathbf{w}}(n+1)-\widehat{\mathbf{w}}(n)+{\gamma }_{\mathrm{SPF}}{S}_{\beta }^{\prime }(\widehat{\mathbf{w}}(n+1))-\lambda \mathbf{x}(n)=0,}$$

(23)

and

$$\overline{e}(n)-\left[1-\xi exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)\right]e(n)=0,$$

(24)

where

$${\displaystyle s_{i,\beta }^{\prime }({\widehat{w}}_i(n+1))=(\beta +1)e^{(-\beta \left|{\widehat{w}}_i(n+1)\right|)}\mathrm{sgn}(w_i(n+1)),}$$

(25)

and its vector form is

$${\displaystyle S_{\beta }^{\prime }(\widehat{\mathbf{w}}(n+1))=(\beta +1)e^{(-\beta \left|\widehat{\mathbf{w}}(n+1)\right|)}\mathrm{sgn}(\widehat{\mathbf{w}}(n+1)).}$$

(26)

Left multiplying by ${\mathbf{x}}^T\left(n\right)$ on the Equation (23) and moving $\widehat{\mathbf{w}}(n+1)$ to the left side, we can obtain

$${\mathbf{x}}^T\left(n\right)\widehat{\mathbf{w}}\left(n+1\right)={\mathbf{x}}^T\left(n\right)\widehat{\mathbf{w}}\left(n\right)-{\gamma }_{\mathrm{SPF}}{\mathbf{x}}^T\left(n\right){S^{\prime }}_{\beta }\left(\widehat{\mathbf{w}}\left(n+1\right)\right)+\lambda {\mathbf{x}}^T\left(n\right)\mathbf{x}\left(n\right).$$

(27)

Combining with Equations (24) and (27),

$$\lambda =\xi \frac{e^{\left(-\frac{e^2\left(n\right)}{2{\sigma }^2}\right)}e\left(n\right)}{{∥\mathbf{x}\left(n\right)∥}^2}+{\gamma }_{\mathrm{SPF}}\frac{{\mathbf{x}}^T\left(n\right){S^{\prime }}_{\beta }\left(\widehat{\mathbf{w}}\left(n+1\right)\right)}{{∥\mathbf{x}\left(n\right)∥}^2}.$$

(28)

If we take Equations (23) and (28) into consideration and consider $\widehat{\mathbf{w}}(n+1)=\widehat{\mathbf{w}}(n)$ when the iteration is stable, we can obtain

$$\begin{array}{c}{\displaystyle \widehat{\mathbf{w}}(n+1)=\widehat{\mathbf{w}}(n)+\xi \frac{e^{\left(-\frac{e^2(n)}{2{\sigma }^2}\right)}e(n)\mathbf{x}(n)}{{∥\mathbf{x}(n)∥}^2}-{\gamma }_{\mathrm{SPF}}{S}_{\beta }^{\prime }(\widehat{\mathbf{w}}(n))+{\gamma }_{\mathrm{SPF}}\frac{{\mathbf{x}}^T(n)S_{\beta }^{\prime }(\widehat{\mathbf{w}}(n))\mathbf{x}(n)}{{∥\mathbf{x}(n)∥}^2}.}\hfill \end{array}$$

(29)

Discarding the term ${\gamma }_{\mathrm{SPF}}\frac{{\mathbf{x}}^T(n)S_{\beta }^{\prime }(\widehat{\mathbf{w}}(n))\mathbf{x}(n)}{{∥\mathbf{x}(n)∥}^2}$ since it is very small, we can get the update equation of the SPF-NMCC algorithm, which is given by

$$\begin{array}{c}{\displaystyle \widehat{\mathbf{w}}(n+1)=\underset{\mathrm{SPF}-\mathrm{NMCC}\text{algorithm}}{\underbrace{\underset{\mathrm{NMCC}\text{algorithm}}{\underbrace{\widehat{\mathbf{w}}(n)+\stackrel{\mathrm{A}1}{\overbrace{{\mu }_{\mathrm{SPF}}\frac{e^{\left(-\frac{e^2(n)}{2{\sigma }^2}\right)}e(n)\mathbf{x}(n)}{{∥\mathbf{x}(n)∥}^2}}}}}\stackrel{\mathrm{A}2}{\overbrace{-{\gamma }_{\mathrm{SPF}}{S}_{\beta }^{\prime }(\widehat{\mathbf{w}}(n))}}}},}\hfill \end{array}$$

(30)

where ${\mu }_{\mathrm{SPF}}=\xi$ acts as a step-size. It is observed that Equation (30) represents a unified form of the sparse MCC algorithm. The previously reported sparse MCC can be regarded as special cases of the proposed SPF-NMCC algorithm. If ${\gamma }_{\mathrm{SPF}}\ne 0$ and $\xi =\chi {∥\mathbf{x}(n)∥}^2$, the proposed SPF-NMCC algorithm is a soft function parameter penalized MCC algorithm. In addition, we can choose different β to control the SPF-NMCC algorithm. From the derivation of the SPF-NMCC algorithm, we can see that the proposed SPF-NMCC algorithm is a zero attracting algorithm, whose zero attracting ability can be controlled by the regularization parameter ${\gamma }_{\mathrm{SPF}}$. In the update Equation (30), $-{\gamma }_{\mathrm{SPF}}{S}_{\beta }^{\prime }(\widehat{\mathbf{w}}(n))$ is a zero attractor, which is used to speed up the convergence of the SPF-NMCC algorithm.

To better understand the optimization problem given in Equation (19), we present an intuitive geometric explanation of the updated equation in Equation (30), which is illustrated in Figure 2. It is found that $\widehat{\mathbf{w}}(n)$ converges to the hyperplane II that is defined by $\overline{e}(n)=0$ through paths A1 and A2. Additionally, path A1 is orthogonal to the hyperplane. Path A2 is the zero-attracting path, which forces the zero or near-zero coefficients of $\widehat{\mathbf{w}}(n)$ in the direction to zero. We can also see that the $exp\left(-\frac{e^2(n)}{2{\sigma }^2}\right)$ approximates to $\widehat{\mathbf{w}}(n+1)$ for large $e(n)$, which helps to reduce the update magnitude to render the proposed algorithm robust in dealing with impulsive noise.

4. Performance of the SPF-NMCC Algorithm

In this section, we will investigate the estimation performance of the proposed SPF-NMCC algorithm in the context of sparse system identification. The key parameters of the proposed SPF-NMCC algorithm are discussed and the performance of the SPF-NMCC algorithm is evaluated with different sparsity levels. Furthermore, the convergence of the proposed SPF-NMCC algorithm is given and is compared with LMS, MCC, ZA-MCC and RZA-MCC algorithms. In all the simulations, 500 Monte Carlo runs are used to get each point for all the mentioned adaptive filtering algorithms. Herein, the sparsity level K is defined as the number of non-zero coefficients and these K dominant coefficients are randomly distributed within the length of the sparse system. In addition, the FIR system with a length of $N=16$ and the sparsity levels of $K=2$, $K=4$ and $K=8$ are considered to investigate the performance of the proposed SPF-NMCC algorithm. Since the MCC algorithms perform well in an impulsive noise environment, we employ an impulsive noise $v(n)$ model, which is given as follows

$$(1-\theta )N({\iota }_1,{\nu }_1^2)+\theta N({\iota }_2,{\nu }_2^2),$$

(31)

where $N({\iota }_i,{\nu }_i^2)(i=1,2)$ are the Gaussian distributions whose means are ${\iota }_i$ and variances are ${\nu }_i^2$, and θ is a mixture parameter used for controlling the mixture of the two noises. In addition, we assume the mixed Gaussian noise is independent of $\mathbf{x}(n)$. In all of the simulations, the mixed impulsive noise $({\iota }_i,{\nu }_i^2,{\iota }_2,{\nu }_2^2,\theta )$ is set to be $(0,0.05,0,20,0.05)$. The sparse system estimation performance is given by mean square deviation (MSD) whose definition is

$$\mathrm{MSD}(\widehat{\mathbf{w}}(n))=\mathrm{E}[{∥{\mathbf{w}}_o(n)-\widehat{w}(n)∥}^2].$$

(32)

Firstly, the effects of parameters β, ${\mu }_{\mathrm{SPF}}$, and ${\gamma }_{\mathrm{SPF}}$ are investigated in detail. The effects of β on the proposed SPF-NMCC algorithm are shown in Figure 3. It is found that the steady-state error floor is reduced with an increment of β because large β can well approximate $l_0$-norm. The steady-state behavior of the proposed SPF-NMCC algorithm with various ${\mu }_{\mathrm{SPF}}$ is demonstrated in Figure 4. It is noted that the convergence of the proposed SPF-NMCC algorithm becomes faster when ${\mu }_{\mathrm{SPF}}$ increases from 0.2 to 0.9, which is similar to the MCC algorithm. Thus, parameters ${\gamma }_{\mathrm{SPF}}$, β and ${\mu }_{\mathrm{SPF}}$ of the proposed SPF-NMCC algorithm should be properly selected to achieve a good performance. The effects of the regularization parameter ${\gamma }_{\mathrm{SPF}}$ is investigated to evaluate the steady-state performance of the proposed SPF-NMCC algorithm. In this experiment, the simulation parameters are $\sigma =1000$, $\beta =6$ and $\xi =0.2$. The simulated results of the parameter ${\gamma }_{\mathrm{SPF}}$ are shown in Figure 5. It can be seen that the regularization parameter ${\gamma }_{\mathrm{SPF}}$ has an important effect on the steady-state performance of the proposed SPF-NMCC algorithm. When ${\gamma }_{\mathrm{SPF}}$ decreases from $8\times {10}^{-4}$ to $5\times {10}^{-5}$, the steady-state error floor is reduced. If we continue to decrease the value of ${\gamma }_{\mathrm{SPF}}$, the steady-state error floor is deteriorated and is rebounded to an opposite direction. This is because the large ${\gamma }_{\mathrm{SPF}}$ can provide a strong zero attraction to attract the zero or near zero coefficients to zero quickly, while the small ${\gamma }_{\mathrm{SPF}}$ exerts a weak zero attraction on the zero or near zero coefficients [26,28]. Thus, the regularization parameter ${\gamma }_{\mathrm{SPF}}$ can well control the zero attracting ability to provide an excellent performance for sparse system identification.

Secondly, we perform an experiment to investigate the convergence speed of the proposed SPF-NMCC algorithm. The convergence of the SPF-NMCC algorithm is illustrated in Figure 6 and its convergence is also compared with the existing LMS, MCC, NLMS, NMCC, ZA-LMS, ZA-MCC, RZA-LMS, and RZA-MCC algorithms. Here, the simulation parameters are ${\mu }_{\mathrm{LMS}}=0.005$, $\chi =0.0052$, ${\mu }_{\mathrm{NLMS}}=0.082$, ${\chi }_{\mathrm{NMCC}}=0.087$, ${\mu }_{\mathrm{ZALMS}}=0.008$, ${\rho }_{\mathrm{ZALMS}}=9\times {10}^{-6}$, ${\xi }_{\mathrm{ZA}}=0.008$, ${\rho }_{\mathrm{ZA}}=3\times {10}^{-5}$, ${\mu }_{\mathrm{RZALMS}}=0.013$, ${\rho }_{\mathrm{RZALMS}}=5\times {10}^{-5}$, ${\xi }_{\mathrm{RZA}}=0.018$, ${\rho }_{\mathrm{RZA}}=7\times {10}^{-5}$, ${\mu }_{\mathrm{SPF}}=0.3$, and ${\gamma }_{\mathrm{SPF}}=2.5\times {10}^{-5}$, which result in nearly the same steady-state error floor. Here, ${\mu }_{\mathrm{LMS}}$, ${\mu }_{\mathrm{NLMS}}$, ${\mu }_{\mathrm{ZALMS}}$ and ${\mu }_{\mathrm{RZALMS}}$ represent the step-sizes of the LMS, NLMS, ZA-LMS and RZA-LMS algorithms, while ${\rho }_{\mathrm{ZALMS}}$ and ${\rho }_{\mathrm{RZALMS}}$ are the regularization parameters of the ZA-LMS and RZA-LMS algorithms. It is observed that our proposed SPF-NMCC algorithm achieves the fastest convergence speed rate. It is worth noting that the NMCC algorithm also converges faster than the MCC algorithm. Another interesting result of this figure is that zero attracting algorithms increase convergence speed, and further increments on it can be obtained by reweighing the coefficients. It is also worth noting that algorithms using MCC converge faster than those using LMS criterion. Furthermore, the proposed SPF scheme can provide a flexible zero attraction to switch the norm penalties between $l_0$-norm and $l_1$-norm, resulting in the fastest convergence.

Thirdly, our proposed SPF-NMCC algorithm is analyzed under different sparsity level K, namely, $K=2$, $K=4$ and $K=8$. In this experiment, the related parameters are set as ${\mu }_{\mathrm{LMS}}={\mu }_{\mathrm{ZALMS}}={\mu }_{\mathrm{RZALMS}}=0.06$, $\chi =0.05$, ${\mu }_{\mathrm{NLMS}}=0.6$, ${\chi }_{\mathrm{NMCC}}=0.5$, ${\rho }_{\mathrm{ZALMS}}=8\times {10}^{-5}$, ${\xi }_{\mathrm{ZA}}={\xi }_{\mathrm{RZA}}=0.05$, ${\rho }_{\mathrm{ZA}}=5\times {10}^{-5}$, ${\rho }_{\mathrm{RZALMS}}={\rho }_{\mathrm{RZA}}=8\times {10}^{-4}$, ${\mu }_{\mathrm{SPF}}=0.45$, and ${\gamma }_{\mathrm{SPF}}=8\times {10}^{-5}$. The system estimation behaviors of the SPF-NMCC algorithm for $K=2$, $K=4$ and $K=8$ are demonstrated in Figure 7, Figure 8 and Figure 9, respectively. It is found that our proposed SPF-NMCC algorithm has the lowest steady-state error floor for $K=2$. In addition, the proposed NMCC algorithm with the same initial convergence is better than the MCC algorithm in terms of the steady-state behavior. When K increases from 2 to 8, the steady-state error floor increases. However, our proposed SPF-NMCC algorithm outperforms the previously presented algorithms with respect to the steady-state behavior. From Figure 7, Figure 8 and Figure 9, the convergence speed of the proposed SPF-NMCC algorithm is a bit larger than that of the other mentioned algorithms, which is to obtain a lower steady-state behavior.

As we know, adaptive filtering algorithms have been widely used for echo cancellation. Therefore, we also construct an experiment to discuss the steady-state behavior of the proposed algorithm over an acoustic echo channel. Here, we define the measurement of the sparseness of the used acoustic echo channel as ${\zeta }_{12}({\mathbf{w}}_{\mathbf{o}})=\frac{N}{N-\sqrt{N}}\left(1-\frac{{∥{\mathbf{w}}_{\mathbf{o}}∥}_1}{\sqrt{N}{∥{\mathbf{w}}_{\mathbf{o}}∥}_2}\right)$. Figure 10 gives a typical acoustic echo channel whose length is 256, and there are 16 dominant coefficients. In order to better understand the effects of the sparsity, ${\zeta }_{12}({\mathbf{w}}_{\mathbf{o}})=0.8222$ and ${\zeta }_{12}({\mathbf{w}}_{\mathbf{o}})=0.7362$ are employed to evaluate the tracking behavior of the proposed SPF-NMCC algorithm. The parameters are $\chi =0.0045$, ${\mu }_{\mathrm{NLMS}}=1$, ${\chi }_{\mathrm{NMCC}}=0.9$, ${\mu }_{\mathrm{ZALMS}}={\mu }_{\mathrm{RZALMS}}=0.005$, ${\rho }_{\mathrm{ZALMS}}={\rho }_{\mathrm{RZALMS}}=2\times {10}^{-5}$, ${\xi }_{\mathrm{ZA}}=0.0035$, ${\rho }_{\mathrm{ZA}}={\rho }_{\mathrm{RZA}}=5\times {10}^{-6}$, ${\xi }_{\mathrm{RZA}}=0.003$, ${\mu }_{\mathrm{SPF}}=0.9$, and ${\gamma }_{\mathrm{SPF}}=3\times {10}^{-6}$. The simulation results for tracking the acoustic echo channel are shown in Figure 11. It can be seen that our proposed SPF-NMCC algorithm outperforms the conventional LMS and MCC algorithms and their related sparse forms in terms of both the convergence speed rate and steady-state behavior. Even for a reduced sparsity of ${\zeta }_{12}({\mathbf{w}}_{\mathbf{o}})=0.7362$, our proposed SPF-NMCC algorithm still has the best steady-state performance. Thus, we can say that the proposed SPF-NMCC algorithm shows little dependence on the sparsity of the unknown system ${\mathbf{w}}_{\mathbf{o}}$ in comparison with the mentioned previously presented adaptive filtering algorithms. Thus, the proposed SPF-NMCC algorithm is more effective for dealing with sparse system identification.

At last, we give the complexity of the SPF-NMCC algorithm and compare it with the mentioned algorithms above in

Table 1. Computational complexity.

Algorithms	Additions	Multiplications	Divisions	Exponential Calculation
LMS	$2N+1$	$2N+1$	-	-
MCC	$2N+1$	$2N+2$	-	1
NLMS	$3N$	$3N+1$	1	-
NMCC	$3N$	$3N+2$	1	1
ZA-LMS	N+ 3K	N+ 3K + 1	-	-
ZA-MCC	N+ 3K	N+ 3K + 2	-	1
RZA-LMS	N+ 4K	N+ 4K + 1	N	-
RZA-MCC	N+ 4K	N+ 4K + 2	N	1
SPF-NMCC	N+ 4K + 1	N+ 5K	N	N+ 1

. We use the additions, multiplications, divisions and exponential calculation to give a comparison of the computational complexity. The comparisons of the related computational complexity are given in Table 1. We can see that our SPF-NMCC algorithm has ($N+4K+1$) additions, ($N+5K$) multiplications, (N) divisions and ($N+1$) exponential calculations, which provides a moderate computational complexity for sparse system identification.

On the basis of the discussions and analysis aforementioned, we can give a short summary about our proposed SPF-NMCC algorithm for handling sparse system identification. It is found that our proposed SPF-NMCC algorithm has the fastest convergence speed rate for achieving the same steady-state error floor. Moreover, it can also provide the smallest system estimation misalignment in comparison with the LMS and MCC algorithm and their related zero attracting forms. This is because the proposed SPF-NMCC algorithm employs the SPF function to form a desired zero attractor to exploit the sparseness of the systems, and it uses the normalization to give a resistance to the scaling of the input. In addition, the zero attracting abilities of the proposed SPF-NMCC algorithm can be controlled by ${\gamma }_{\mathrm{SPF}}$. Our SPF-NMCC algorithm introduced two extra parameters to exploit the sparsity of the traditional MCC algorithm by introducing a zero attractor. Although two extra parameters have been added, which slightly increases the computational complexity, the performance of the SPF-NMCC algorithm is better than the MCC and NMCC algorithms. Even compared with the previously reported ZA-MCC and RZA-MCC algorithms, the SPF-NMCC algorithm has the fastest convergence and lowest estimation error in terms of MSD. However, the proposed SPF-NMCC algorithm has a cost of selecting the proper hyper-parameter values in practical engineering applications because it requires more time and data to select the parameters β and ${\rho }_{\mathrm{SPF}}$ in comparison with the MCC algorithm. Compared with the ZA-MCC and RZA-MCC algorithms, the proposed SPF-NMCC algorithm needs to find a proper β.

5. Conclusions

A sparse SPF-NMCC algorithm has been proposed for sparse system identification applications. The proposed SPF-NMCC algorithm was realized by using a soft parameter function as a penalty to construct the desired zero attractor in its iterations. Typical sparse systems were used to evaluate the performance of the SPF-NMCC algorithm in terms of both convergence speed and steady-state behaviors. In addition, the key parameters were also investigated to give a clear explanation of the SPF-NMCC algorithm. Simulation results show that the proposed SPF-NMCC algorithm can achieve the fastest convergence speed and smallest steady-state misalignment. However, the proposed algorithm requires more time and data to select the hyper parameters. In the future, we will develop sparse NMCC algorithms with high performance and fewer hyper parameters.