A Novel NLMS Algorithm for System Identification

Abstract

In this paper, we propose a novel normalized least mean squares (NLMS) algorithm for system identification applications. Our approach involves analyzing the mean squared deviation performance of the NLMS algorithm using a random walk model to select two optimal parameters, the step size and regularization parameters, for the rapid convergence of the colored input signals. We verified that the proposed algorithm exhibited faster convergence than existing algorithms, even in scenarios of sudden system changes.

1. Introduction

The applications of system identification techniques in robotics are wide-ranging and diverse. They encompass areas such as robot localization and mapping, sensor calibration, motion planning, and control system design [1,2,3,4]. System identification enables robots to learn and adapt to their environment, allowing them to adjust their behaviors and responses autonomously based on observed data.

Normalized least mean squares (NLMS) algorithms are widely employed in system identification applications because of their robustness, computational simplicity, and ease of implementation [5,6]. For NLMS algorithms, the convergence speed and steady-state error performance are most important. Faster convergence enables adaptive filters to adapt quickly to changes in the input signals or system conditions. Additionally, smaller steady-state errors ensure the accurate estimation or overall reconstruction of the desired signals. The performance of NLMS algorithms is directly affected by two key parameters: the step size and regularization parameters. The appropriate adjustment of these parameters is crucial for optimizing the performance of an algorithm. Several algorithms have been proposed to address this challenge, such as the variable step size (VSS) and variable regularization (VR) algorithms. These algorithms aim to enhance the convergence behaviors and steady-state misalignment [7,8,9,10,11,12,13,14]. Because many of these algorithms were developed under the assumption of a time-invariant system to simplify the algorithm design, they cannot perform optimally when the system undergoes significant changes.

To address this limitation, researchers developed the joint-optimization NLMS (JO-NLMS) algorithm using a time-varying system model [15]. This algorithm controlled the step size and regularization parameter to minimize system misalignment using a joint-optimization scheme. However, the JO-NLMS algorithm only considered white Gaussian input signals, which could lead to performance degradation when faced with colored input signals.

This paper proposes a novel NLMS algorithm for the fast estimation of unknown system coefficients with colored input signals. In the application of system identification, the primary aim is to estimate unknown system (or plant) coefficients rather than solely minimizing the errors. The mean square deviation (MSD) performance of the NLMS algorithm could provide guidelines for selecting the optimal parameters, such as the step size and regularization parameters. Therefore, we analyzed the MSD of the conventional NLMS algorithm with a fixed step size and regularization parameter. Additionally, the optimal parameters were derived by minimizing the MSD of the NLMS at each iteration.

The remainder of this paper is organized as follows. Section 2 reviews the conventional NLMS algorithm. Section 3 presents an MSD analysis of the NLMS algorithm using the random walk model. Section 4 presents the optimal parameters derived by the MSD analysis. Section 5 presents the computer simulation results to verify the convergence and misalignment performance of the proposed algorithm for white and colored inputs. Lastly, Section 6 concludes the paper.

2. Review of the Conventional NLMS Algorithm

We focused on the adaptive identification of a linear adaptive filter that was assumed to be a finite impulse response filter for the system identification problem based on given input–output data. Figure 1 shows the structure of an adaptive filter for system identification.

In the system identificatioin problem, the unknown system coefficients and the input vector at the discrete-time index n are given by

$${\mathbf{w}}_o={[w_0,w_1,\cdots ,w_{M-1}]}^T,$$

(1)

$$\mathbf{u}\left(n\right)={[u\left(n\right),u(n-1),\cdots ,u(n-M+1)]}^T,$$

(2)

respectively, where M is the filter length and ${(·)}^T$ is the transpose operation. $u\left(n\right)$ is a random input sequence and the vector $\mathbf{u}\left(n\right)$ consists of a total of M input samples, from the current input sample to the past $M-1$ input samples. The desired signal $d\left(n\right)$ is

$$d\left(n\right)=\mathbf{u}{\left(n\right)}^T{\mathbf{w}}_o+v\left(n\right),$$

(3)

where $v\left(n\right)$ is the measurement noise, which follows a zero-mean white Gaussian distribution with variance ${\sigma }_v^2$. The standard NLMS algorithm estimates ${\mathbf{w}}_o$ at index n, defined by $\widehat{\mathbf{w}}\left(n\right)$, using the updated equation

$$\widehat{\mathbf{w}}\left(n\right)=\widehat{\mathbf{w}}(n-1)+\frac{\mu \mathbf{u}\left(n\right)e\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta },$$

(4)

where $\mu$ is the step size, $\delta$ is the regularization parameter, and $e\left(n\right)=d\left(n\right)-{\mathbf{u}}^T\left(n\right)\widehat{\mathbf{w}}(n-1)$ is the output error signal.

3. MSD Analysis of the NLMS Algorithm Using the Random Walk Model

We considered that the unknown system vector ${\mathbf{w}}_o\left(n\right)$ at index n had the following model [15,16,17]:

$${\mathbf{w}}_o\left(n\right)={\mathbf{w}}_o(n-1)+\mathbf{q}\left(n\right),$$

(5)

where $\mathbf{q}\left(n\right)={[q\left(n\right),q(n-1),\cdots ,q(n-M+1)]}^T$ is a zero-mean white Gaussian noise vector with variance ${\sigma }_q^2$. The covariance matrix of $\mathbf{q}\left(n\right)$ was assumed to be $E\left(\mathbf{q}\left(n\right){\mathbf{q}}^T\left(n\right)\right)={\sigma }_q^2{\mathbf{I}}_M$, where $E(·)$ is the mathematical expectation, and ${\mathbf{I}}_M$ is the $M\times M$ identity matrix. By subtracting the random walk model, the error signal could be obtained as

$$\begin{array}{cc}\hfill e\left(n\right)& =d\left(n\right)-{\mathbf{u}}^T\left(n\right)\widehat{\mathbf{w}}(n-1)\hfill \\ \hfill & ={\mathbf{u}}^T\left(n\right){\mathbf{w}}_o\left(n\right)+v\left(n\right)-{\mathbf{u}}^T\left(n\right)\widehat{\mathbf{w}}(n-1)\hfill \\ \hfill & ={\mathbf{u}}^T\left(n\right){\mathbf{w}}_o(n-1)+{\mathbf{u}}^T\left(n\right)\mathbf{q}\left(n\right)+v\left(n\right)-{\mathbf{u}}^T\left(n\right)\widehat{\mathbf{w}}(n-1)\hfill \\ \hfill & ={\mathbf{u}}^T\left(n\right)\tilde{\mathbf{w}}(n-1)+{\mathbf{u}}^T\left(n\right)\mathbf{q}\left(n\right)+v\left(n\right),\hfill \end{array}$$

(6)

where $\tilde{\mathbf{w}}\left(n\right)\triangleq {\mathbf{w}}_o\left(n\right)-\widehat{\mathbf{w}}\left(n\right)$. Therefore, we could obtain the recursion of $\tilde{\mathbf{w}}\left(n\right)$ by applying (4) and (6) as follows:

$$\begin{array}{cc}\hfill \tilde{\mathbf{w}}\left(n\right)& =\tilde{\mathbf{w}}(n-1)-\frac{\mu \mathbf{u}\left(n\right)e\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }+\mathbf{q}\left(n\right)\hfill \\ \hfill & =\left({\mathbf{I}}_M-\frac{\mu \mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }\right)\tilde{\mathbf{w}}(n-1)-\frac{\mu \mathbf{u}\left(n\right)v\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }\hfill \\ \hfill & -\frac{\mu \mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)\mathbf{q}\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }+\mathbf{q}\left(n\right).\hfill \end{array}$$

(7)

We can define the covariance matrix of $\tilde{\mathbf{w}}\left(n\right)$ using a conditional covariance matrix based on the given set of input signals ${\mathcal{U}}_n\triangleq \left\{\mathbf{u}\left(j\right)\right|0\le j\le n\}$ as follows [18]:

$$\mathbf{P}\left(n\right)\triangleq E\left(\tilde{\mathbf{w}}\left(n\right){\tilde{\mathbf{w}}}^T\left(n\right)\right)\equiv E\left(E\left(\tilde{\mathbf{w}}\left(n\right){\tilde{\mathbf{w}}}^T\left(n\right)|{\mathcal{U}}_n\right)\right),$$

(8)

$$\overline{\mathbf{P}}\left(n\right)\triangleq E\left(\tilde{\mathbf{w}}\left(n\right){\tilde{\mathbf{w}}}^T\left(n\right)|{\mathcal{U}}_n\right).$$

(9)

Hence, the MSD is defined as

$$\mathrm{MSD}(n)\triangleq E\left({\tilde{\mathbf{w}}}^T\left(n\right)\tilde{\mathbf{w}}\left(n\right)\right)\equiv Tr\left(\mathbf{P}\left(n\right)\right).$$

(10)

To obtain the MSD of the NLMS algorithm at index n, we considered the following three assumptions [17,19,20,21,22,23]:

Assumption 1.

The signals $v\left(n\right)$, $\mathbf{u}\left(n\right)$, $\mathbf{q}\left(n\right)$, and $\tilde{\mathbf{w}}(n-1)$ are statistically independent.

Assumption 2.

The input signal vector $\mathbf{u}\left(n\right)$ has a zero mean and an independent and identically distributed and covariance matrix

$$\mathbf{R}=E\left(\mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)\right)=\mathbf{V}\mathbf{\Lambda }{\mathbf{V}}^T,$$

(11)

where $\mathbf{\Lambda }=diag({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_M)$ are the eigenvalues of covariance matrix $\mathbf{R}$, and $\mathbf{V}=\left[{\mathit{\nu }}_1{\mathit{\nu }}_2\cdots {\mathit{\nu }}_M\right]$ are the corresponding orthonormal eigenvectors $({\mathbf{V}}^T\mathbf{V}={\mathbf{I}}_M)$.

Assumption 3.

The input signal vector $\mathbf{u}\left(n\right)$ comprises three independent variables $s,r,\mathcal{V}$ that are independent and identically distributed, such that

$$\begin{array}{cc}\hfill & \mathbf{u}\left(n\right)=sr\mathcal{V},\hfill \\ \hfill & \mathit{where}\left\{\begin{array}{c}P(s=\pm 1)=\frac{1}{2}\hfill \\ r\sim \parallel \mathbf{u}\left(n\right)\parallel \hfill \\ P(\mathcal{V}={\mathit{\nu }}_j)=p_j=\frac{{\lambda }_j}{Tr\left(\mathbf{R}\right)},j=1,\dots ,M\hfill \end{array}\right.\hfill \end{array}$$

(12)

Here, $r\sim \parallel \mathbf{u}\left(n\right)\parallel$ indicates that r and the norm of $\mathbf{u}\left(n\right)$ have the same distribution. We note that ${\sum }_{j=1}^M{p}_j=1$.

Under Assumption 1, the conditional covariance matrix of $\tilde{\mathbf{w}}\left(n\right)$ was propagated as follows:

$$\begin{array}{cc}\hfill \overline{\mathbf{P}}\left(n\right)& =\left({\mathbf{I}}_M-\frac{\mu \mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }\right)\overline{\mathbf{P}}(n-1){\left({\mathbf{I}}_M-\frac{\mu \mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }\right)}^T\hfill \\ \hfill & +\frac{{\mu }^2\mathbf{u}\left(n\right)E\left(v{\left(n\right)}^2\right){\mathbf{u}}^T\left(n\right)}{{\left({\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta \right)}^2}+\frac{{\mu }^2\mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)E\left(\mathbf{q}\left(n\right){\mathbf{q}}^T\left(n\right)\right)\mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)}{{\left({\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta \right)}^2}\hfill \\ \hfill & -\frac{\mu \mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)E\left(\mathbf{q}\left(n\right){\mathbf{q}}^T\left(n\right)\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }-\frac{\mu E\left(\mathbf{q}\left(n\right){\mathbf{q}}^T\left(n\right)\right)\mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }\hfill \\ \hfill & +E\left(\mathbf{q}\left(n\right){\mathbf{q}}^T\left(n\right)\right)\hfill \\ \hfill & =\left({\mathbf{I}}_M-\frac{\mu \mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }\right)\overline{\mathbf{P}}(n-1){\left({\mathbf{I}}_M-\frac{\mu \mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }\right)}^T\hfill \\ \hfill & +\frac{{\mu }^2{\sigma }_v^2\mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)}{{\left({\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta \right)}^2}+\frac{{\mu }^2{\sigma }_q^2\mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)}{{\left({\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta \right)}^2}\hfill \\ \hfill & -\frac{2\mu {\sigma }_q^2\mathbf{u}\left(n\right){\mathbf{u}}^T\left(n\right)}{{\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)+\delta }+{\sigma }_q^2{\mathbf{I}}_M.\hfill \end{array}$$

(13)

Therefore, applying Assumptions 2 and 3, the propagation of $\mathbf{P}\left(n\right)$ could be obtained as

$$\begin{array}{cc}\hfill \mathbf{P}\left(n\right)& =E\left\{\left({\mathbf{I}}_M-\frac{\mu r^2\mathcal{V}{\mathcal{V}}^T}{r^2+\delta }\right)\mathbf{P}(n-1){\left({\mathbf{I}}_M-\frac{\mu r^2\mathcal{V}{\mathcal{V}}^T}{r^2+\delta }\right)}^T\right\}\hfill \\ \hfill & +E\left\{\frac{{\mu }^2{\sigma }_v^2{r}^2\mathcal{V}{\mathcal{V}}^T}{{(r^2+\delta )}^2}\right\}+E\left\{\frac{{\mu }^2{\sigma }_q^2{r}^4\mathcal{V}{\mathcal{V}}^T\mathcal{V}{\mathcal{V}}^T}{{(r^2+\delta )}^2}\right\}\hfill \\ \hfill & -E\left\{\frac{2\mu {\sigma }_q^2{r}^4\mathcal{V}{\mathcal{V}}^T}{r^2+\delta }\right\}+{\sigma }_q^2{\mathbf{I}}_M.\hfill \end{array}$$

(14)

We defined ${\pi }_k\left(n\right)\triangleq {\mathit{\nu }}_k^T\mathbf{P}\left(n\right){\mathit{\nu }}_k$ by multiplying ${\mathit{\nu }}_k^T$ and ${\mathit{\nu }}_k$ for both sides of $\mathbf{P}\left(n\right)$, where $k=1,2,\dots ,M$. From (14), we obtained the recursion of ${\pi }_k\left(n\right)$ as follows:

$$\begin{array}{cc}\hfill {\pi }_k\left(n\right)& =E\left\{{\mathit{\nu }}_k^T\left({\mathbf{I}}_M-\frac{\mu r^2\mathcal{V}{\mathcal{V}}^T}{r^2+\delta }\right)\mathbf{P}(n-1){\left({\mathbf{I}}_M-\frac{\mu r^2\mathcal{V}{\mathcal{V}}^T}{r^2+\delta }\right)}^T{\mathit{\nu }}_k\right\}\hfill \\ \hfill & +E\left\{\frac{{\mu }^2{\sigma }_v^2{r}^2{\mathit{\nu }}_k^T\mathcal{V}{\mathcal{V}}^T{\mathit{\nu }}_k}{{(r^2+\delta )}^2}\right\}+E\left\{\frac{{\mu }^2{\sigma }_q^2{r}^4{\mathit{\nu }}_k^T\mathcal{V}{\mathcal{V}}^T\mathcal{V}{\mathcal{V}}^T{\mathit{\nu }}_k}{{(r^2+\delta )}^2}\right\}\hfill \\ \hfill & -E\left\{\frac{2\mu {\sigma }_q^2{r}^4{\mathit{\nu }}_k^T\mathcal{V}{\mathcal{V}}^T{\mathit{\nu }}_k}{r^2+\delta }\right\}+{\sigma }_q^2{\mathit{\nu }}_k^T{\mathit{\nu }}_k\hfill \\ \hfill & =\sum _{j=1}^M{p}_{j}E\left\{{\mathit{\nu }}_k^T\left({\mathbf{I}}_M-\frac{\mu r^2{\mathit{\nu }}_j{\mathit{\nu }}_j^T}{r^2+\delta }\right)\mathbf{P}(n-1){\left({\mathbf{I}}_M-\frac{\mu r^2{\mathit{\nu }}_j{\mathit{\nu }}_j^T}{r^2+\delta }\right)}^T{\mathit{\nu }}_k\right\}\hfill \\ \hfill & +\sum _{j=1}^M{p}_{j}E\left\{\frac{{\mu }^2{\sigma }_v^2{r}^2{\mathit{\nu }}_k^T{\mathit{\nu }}_j{\mathit{\nu }}_j^T{\mathit{\nu }}_k}{{(r^2+\delta )}^2}\right\}+\sum _{j=1}^M{p}_{j}E\left\{\frac{{\mu }^2{\sigma }_q^2{r}^4{\mathit{\nu }}_k^T{\mathit{\nu }}_j{\mathit{\nu }}_j^T{\mathit{\nu }}_j{\mathit{\nu }}_j^T{\mathit{\nu }}_k}{{(r^2+\delta )}^2}\right\}\hfill \\ \hfill & -\sum _{j=1}^M{p}_{j}E\left\{\frac{2\mu {\sigma }_q^2{r}^4{\mathit{\nu }}_k^T{\mathit{\nu }}_j{\mathit{\nu }}_j^T{\mathit{\nu }}_k}{r^2+\delta }\right\}+\sum _{j=1}^M{p}_j{\sigma }_q^2\hfill \\ \hfill & =\left\{1+p_k\alpha r^2\left(\alpha r^2-2\right)\right\}\left({\pi }_k(n-1)+{\sigma }_q^2\right)+p_k{\sigma }_v^2{\alpha }^2{r}^2,\hfill \end{array}$$

(15)

where $\alpha \triangleq \mu /(r^2+\delta )$. To calculate the MSD from ${\pi }_k\left(n\right)$, we could evaluate the following equation:

$$Tr\left(\mathbf{P}\left(n\right)\right)=Tr\left({\mathbf{V}}^T\mathbf{P}\left(n\right)\mathbf{V}\right)=\sum _{k=1}^M{\pi }_k\left(n\right).$$

(16)

If Equation (15) was accumulated diagonally, we obtained

$$\mathbf{\Pi }\left(n\right)=\left\{{\mathbf{I}}_M+\frac{\alpha r^2\left(\alpha r^2-2\right)}{Tr\left(\mathbf{R}\right)}\mathbf{\Lambda }\right\}\left(\mathbf{\Pi }(n-1)+{\sigma }_q^2{\mathbf{I}}_M\right)+\frac{{\sigma }_v^2{\alpha }^2{r}^2}{Tr\left(\mathbf{R}\right)}\mathbf{\Lambda },$$

(17)

where

$$\mathbf{\Pi }\left(n\right)=\left[\begin{array}{ccc}{\pi }_1\left(n\right)& & \\ & \ddots & \\ & & {\pi }_M\left(n\right)\end{array}\right].$$

(18)

Finally, we obtained

$$\begin{array}{cc}\hfill Tr\left(\mathbf{P}\left(n\right)\right)& =Tr\left(\mathbf{P}(n-1)\right)+\frac{\alpha r^2\left(\alpha r^2-2\right)}{Tr\left(\mathbf{R}\right)}Tr\left(\mathbf{\Lambda }\mathbf{\Pi }(n-1)\right)\hfill \\ \hfill & +\frac{{\sigma }_q^2\alpha r^2\left(\alpha r^2-2\right)}{Tr\left(\mathbf{R}\right)}Tr\left(\mathbf{\Lambda }\right)+M{\sigma }_q^2+\frac{{\sigma }_v^2{\alpha }^2{r}^2}{Tr\left(\mathbf{R}\right)}Tr\left(\mathbf{\Lambda }\right)\hfill \\ \hfill & =\left\{1+\frac{\alpha r^2\left(\alpha r^2-2\right)}{\overline{M}}\right\}Tr\left(\mathbf{P}(n-1)\right)+\left\{1+\frac{\alpha r^2\left(\alpha r^2-2\right)}{M}\right\}M{\sigma }_q^2+{\sigma }_v^2{\alpha }^2{r}^2,\hfill \end{array}$$

(19)

where $\overline{M}\ge M$ is the virtual filter length that considers colored input signals [11,12]. When the input signal is a white Gaussian input, the matrix $\frac{\mathbf{\Lambda }}{Tr\left(\mathbf{R}\right)}$ is approximately equal to $\frac{1}{M}{\mathbf{I}}_M$. However, for colored input signals, $\frac{\mathbf{\Lambda }}{Tr\left(\mathbf{R}\right)}\ne \frac{1}{M}{\mathbf{I}}_M$.

4. Proposed NLMS Algorithm

The optimal parameters of the NLMS algorithm that most rapidly minimized the MSD could be obtained using

$$\frac{\partial \mathrm{MSD}\left(n\right)}{\partial \mu \left(n\right)}=\frac{\partial \mathrm{MSD}\left(n\right)}{\partial \alpha \left(n\right)}\times \frac{\partial \alpha \left(n\right)}{\partial \mu \left(n\right)}=0,$$

(20)

$$\frac{\partial \mathrm{MSD}\left(n\right)}{\partial \delta \left(n\right)}=\frac{\partial \mathrm{MSD}\left(n\right)}{\partial \alpha \left(n\right)}\times \frac{\partial \alpha \left(n\right)}{\partial \delta \left(n\right)}=0,$$

(21)

which is known as the joint-optimization approach [15,17,24]. Both Equations (20) and (21) produced the same result, as follows:

$${\alpha }^{*}\left(n\right)\triangleq \frac{{\mu }^{*}\left(n\right)}{r^2+{\delta }^{*}\left(n\right)}=\frac{\mathrm{MSD}(n-1)+\overline{M}{\sigma }_q^2}{r^2\left\{\mathrm{MSD}(n-1)+\overline{M}{\sigma }_q^2\right\}+\overline{M}{\sigma }_v^2},$$

(22)

where ${\mu }^{*}\left(n\right)$ and ${\delta }^{*}\left(n\right)$ represent the optimal values for the step size and regularization parameter, respectively. From the two optimal parameters, ${\mu }^{*}\left(n\right)$ and ${\delta }^{*}\left(n\right)$, the novel NLMS algorithm was derived as follows:

$$\widehat{\mathbf{w}}\left(n\right)=\widehat{\mathbf{w}}(n-1)+{\alpha }^{*}\left(n\right)\mathbf{u}\left(n\right)e\left(n\right),$$

(23)

where

$${\alpha }^{*}\left(n\right)\triangleq \frac{g\left(n\right)}{r^{2}g\left(n\right)+\overline{M}{\sigma }_v^2},$$

(24)

$$g\left(n\right)\triangleq \mathrm{MSD}(n-1)+\overline{M}{\sigma }_q^2,$$

(25)

$$\mathrm{MSD}(n)=\left\{1-\frac{{\alpha }^{*}\left(n\right)r^2}{\overline{M}}\right\}g\left(n\right)+{\sigma }_q^2\left(M-\overline{M}\right).$$

(26)

4.1. Steady-State Value of the Proposed NLMS Algorithms

Substituting (24) and (25) into (26), the recursion of MSD was obtained as

$$\begin{array}{cc}\hfill \mathrm{MSD}\left(n\right)& =\left\{1-\frac{r^2\mathrm{MSD}(n-1)+r^2\overline{M}{\sigma }_q^2}{r^2\overline{M}\mathrm{MSD}(n-1)+r^2{\overline{M}}^2{\sigma }_q^2+{\overline{M}}^2{\sigma }_v^2}\right\}\left\{\mathrm{MSD}(n-1)+\overline{M}{\sigma }_q^2\right\}\hfill \\ \hfill & +{\sigma }_q^2\left(M-\overline{M}\right).\hfill \end{array}$$

(27)

when the recursion of MSD was stable, ${lim}_{n\to \infty }\mathrm{MSD}\left(n\right)={lim}_{n\to \infty }\mathrm{MSD}(n-1)=\mathrm{MSD}(\infty )$. From (27), therefore, we obtained the quadratic equation on $\mathrm{MSD}(\infty )$ as follows:

$$r^2{\left\{\mathrm{MSD}(\infty )+\overline{M}{\sigma }_q^2\right\}}^2-r^{2}M\overline{M}{\sigma }_q^2\left\{\mathrm{MSD}(\infty )+\overline{M}{\sigma }_q^2\right\}-r^{2}M{\overline{M}}^2{\sigma }_q^2{\sigma }_v^2=0.$$

(28)

Through the solution of the quadratic equation, the steady-state MSD of the proposed NLMS algorithm could be obtained as

$$\mathrm{MSD}(\infty )=\frac{\overline{M}{\sigma }_q^2}{2}\left\{(M-2)+M\sqrt{1+\frac{4{\sigma }_v^2}{r^{2}M{\sigma }_q^2}}\right\}.$$

(29)

As can be seen from Equation (29), it was evident that both ${\sigma }_q^2$ and ${\sigma }_v^2$ had a significant impact on the steady state. These findings were consistent with the results obtained from the JO-NLMS algorithm, indicating a strong similarity between the two algorithms.

4.2. Practical Consideration

The proposed NLMS algorithm requires three primary parameters: $r^2$, ${\sigma }_v^2$, and ${\sigma }_q^2$. To obtain their values, we had to either predetermine or estimate these parameters. The first parameter, $r^2$, could be easily estimated utilizing the variance of the input signal when $M\gg 1$ as follows:

$$r^2\simeq M{\widehat{\sigma }}_u^2\left(n\right)={\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right).$$

(30)

The second parameter is the variance in the measurement noise ${\sigma }_v^2$, which is a crucial parameter that significantly impacted the performance of the proposed algorithm. This parameter could be readily measured or estimated. For example, in the case of speech signals, the measurement noise variance could be estimated during an initial silence period, or real-time methods for estimating the noise variance have also been proposed [7,9,25,26]. While various methods exist for estimating this parameter, a discussion of specific estimation algorithms exceeds the scope of our study. Therefore, we made the assumption that the value of the measurement noise variance was available a priori for the purpose of our algorithm analysis.

The third parameter ${\sigma }_q^2$ could be estimated as follows [15,24]:

$${\widehat{\sigma }}_q^2\left(n\right)\triangleq \frac{\Vert \widehat{\mathbf{w}}\left(n\right)-\widehat{\mathbf{w}}{(n-1)\Vert }^2}{M}.$$

(31)

The proposed NLMS algorithm is summarized in Algorithm 1.

Algorithm 1 Proposesd NLMS algorithm summary.

Initialization:

$\widehat{\mathbf{w}}\left(0\right)=\mathbf{0}$

$\mathrm{MSD}\left(0\right)=1$

${\widehat{\sigma }}_q^2\left(0\right)=0$

Parameters:

${\sigma }_v^2$, known or estimated

$\overline{M}=\kappa \times M$, $\kappa =1$ (for white inputs) or $\kappa >1$ (for colored inputs)

For each index n:

$e\left(n\right)=d\left(n\right)-{\mathbf{u}}^T\left(n\right)\widehat{\mathbf{w}}(n-1)$

$g\left(n\right)=\mathrm{MSD}(n-1)+\overline{M}{\sigma }_q^2(n-1)$

$M{\widehat{\sigma }}_u^2\left(n\right)={\mathbf{u}}^T\left(n\right)\mathbf{u}\left(n\right)$

${\alpha }^{*}\left(n\right)=\frac{g\left(n\right)}{r^{2}g\left(n\right)+\overline{M}{\sigma }_v^2}$

$\mathbf{x}\left(n\right)={\alpha }^{*}\left(n\right)\mathbf{u}\left(n\right)e\left(n\right)$

$\widehat{\mathbf{w}}\left(n\right)=\widehat{\mathbf{w}}(n-1)+\mathbf{x}\left(n\right)$

$\mathrm{MSD}\left(n\right)=\left\{1-\frac{{\alpha }^{*}\left(n\right)M{\widehat{\sigma }}_u^2\left(n\right)}{\overline{M}}\right\}g\left(n\right)+{\widehat{\sigma }}_q^2\left(M-\overline{M}\right)$

${\widehat{\sigma }}_q^2\left(n\right)=\frac{\mathbf{x}{\left(n\right)}^T\mathbf{x}\left(n\right)}{M}$

5. Simulation Results

We conducted computer simulations of a system identification problem to assess the MSD behaviors of various algorithms. In our experiments, we employed two types of unknown system coefficients ${\mathbf{w}}_o$. To assess the general system identification performance of the white and colored inputs, we used randomly generated unknown system coefficients with unit variance. Additionally, we used the acoustic impulse response of a room to compare the performance of the acoustic echo cancellation for speech inputs. The adaptive filter and unknown system were assumed to have the same filter length, which was set to $M=64$ or 512 during the simulations. To generate the colored input signals, we filtered white Gaussian noise using two transfer functions:

$$G_1\left(z\right)=\frac{1}{1-0.95z^{-1}},$$

(32)

$$G_2\left(z\right)=\frac{1}{1-1.6z^{-1}+0.81z^{-2}},$$

(33)

referred to as ${\mathrm{AR}}_1$ and ${\mathrm{AR}}_2$, respectively. The signal-to-noise ratio (SNR) was set to 20 or 30 dB, and measurement noise was added to the output signal $y\left(n\right)$. The SNR was defined as

$$\mathrm{SNR}\triangleq 10{log}_{10}\frac{E\left\{{\left(\mathbf{u}{\left(n\right)}^T{\mathbf{w}}_o\right)}^2\right\}}{E\left\{v{\left(n\right)}^2\right\}}.$$

(34)

Furthermore, we assumed knowledge of the noise variance for all algorithms. The performance metric used was the normalized mean squared deviation (NMSD), which is defined as

$$\mathrm{NMSD}\left(n\right)\triangleq 10{log}_{10}E\left(\frac{{\tilde{\mathbf{w}}}^T\left(n\right)\tilde{\mathbf{w}}\left(n\right)}{{\mathbf{w}}_o^T\left(n\right){\mathbf{w}}_o\left(n\right)}\right).$$

(35)

The simulation results were averaged over 30 independent trials.

5.1. Performance Comparison with White and Colored Input Signals

Figure 2 shows the NMSD learning curves of standard NLMS algorithms with various step sizes (the joint-optimizatioin step size and regularization parameter nomalized sub-band adaptive filter (JOSR-NSAF) algorithm [24], the JO-NLMS algorithm, and the proposed algorithm) for white input signals, which were generated in a zero-mean Gaussian random sequence.

The standard NLMS algorithm was used as a baseline for comparison with the proposed algorithm. The step size of the NLMS algorithm was set to 1 and 0.01, respectively. This was carried out because the NLMS algorithm converged the fastest when the step size was 1, while the steady-state error was similar to that of other algorithms when the step size was 0.01 in this simulation environment. The virtual filter length $\overline{M}=\kappa M$ was set to $\kappa =1$ for the proposed algorithm.

The experimental results indicated that the NMSD performances of the proposed NLMS algorithm, JOSR-NSAF, and JO-NLMS were very similar. The proposed NLMS algorithm for white inputs was slightly slower than the standard NLMS algorithm with a step size of 0.01. However, it was confirmed that it had a sufficiently small error within a short period of time.

Although JOSR-NSAF, JO-NLMS, and the proposed algorithm presented similar performance, the computational complexity of each algorithm was different. In particular, JOSR-NSAF required additional operations for N filter banks with filter length L to use N sub-bands in order to reduce the dynamic range of the input signal. Therefore, it required more computation than the NLMS-based algorithms.

Table 1. The comparison of computational complexity.

Algorithm	Multiplications	Divisions
JO-NLMS	$4M+7$	3
JOSR-NSAF	$3M+(M+1)/N+(N+1)L+5$	2
Proposed NLMS algorithm	$4M+7$	3

shows the computational complexity of JO-NLMS, JOSR-NSAF, and the proposed algorithm per iteration.

Figure 3 shows the NMSD learning curves of the standard NLMS algorithms with various step sizes ($\mu =1$ and $\mu =0.01$), JOSR-NSAF, JO-NLMS, and the proposed NLMS for the colored input signals generated by ${\mathrm{AR}}_1$. The virtual filter length $\overline{M}=\kappa M$ was set to $\kappa =2$ for the proposed algorithm. Compared with the JO-NLMS algorithm, the proposed algorithm exhibited a slightly higher steady-state error performance but had a faster convergence in terms of MSD performance for ${\mathrm{AR}}_1$ input signals.

Figure 4 shows the NMSD learning curves of standard NLMS algorithms with various step sizes ($\mu =1$ and $\mu =0.04$)—JOSR-NSAF, JO-NLMS, and the proposed NLMS—for the colored input signals generated by ${\mathrm{AR}}_2$. The virtual filter length $\overline{M}=\kappa M$ was set to $\kappa =20$ for the proposed algorithm. Similar to the results for ${\mathrm{AR}}_1$ input signals, the proposed algorithm showed fast convergence in terms of MSD performance for ${\mathrm{AR}}_2$ input signals.

Figure 5 shows the optimal parameter values selected for JO-NLMS and the proposed NLMS algorithm for ${\mathrm{AR}}_2$ inputs. The optimal parameter values for each algorithm showed that the parameter value of JO-NLMS decreased rapidly, even though the MSD value was not sufficiently small. However, the optimal parameter value of the proposed algorithm did not decrease as rapidly as that of JO-NLMS due to the value of $\overline{M}$, and therefore it could maintain a fast convergence speed.

5.2. Performance Comparison with Speech Input Signals

We conducted experiments using real speech signals and acoustic echo paths to verify the system identification performance of the algorithms for speech signals. The acoustic echo paths and speech signals used are shown in Figure 6. The acoustic echo path was of length $M=512$.

Figure 7 shows the NMSD learning curves of standard NLMS algorithms with various step sizes ($\mu =1$ and $\mu =0.01$), JOSR-NSAF, JO-NLMS, and the proposed NLMS for speech signals. The virtual filter length $\overline{M}=\kappa M$ was set to $\kappa =20$ for the proposed algorithm. The experimental results confirmed that the proposed algorithm outperformed the JO-NLMS algorithm in terms of convergence speed and steady-state error for speech inputs and acoustic echo paths.

6. Conclusions

In this paper, we proposed a new NLMS algorithm for system identification that exhibited a high convergence speed and low steady-state error, even for colored inputs. To achieve this, we utilized an method for interpreting the MSD performance of the NLMS algorithm using a random walk model to select the optimal step size and regularization parameters. Specifically, we employed a virtual filter length to achieve fast convergence for colored inputs. Additionally, we experimentally demonstrated that our algorithm performed well even without requiring the additional reset algorithms commonly used in existing VSS and VR algorithms to adapt to significant system changes. The simulations revealed that the proposed algorithm for system identification achieved fast convergence even in scenarios of sudden system changes.