Digital Self-Interference Canceler with Joint Channel Estimator for Simultaneous Transmit and Receive System

Abstract

Simultaneous transmit and receive wireless communications have been highlighted for their potential to double the spectral efficiency. However, it is necessary to mitigate self-interference (SI). Considering both the SI channel and remote transmission (RT) channel need to be estimated before equalizing the received signal, we propose two adaptive algorithms for linear and nonlinear self-interference cancellation (SIC), based on a multi-layered joint channel estimator structure. The proposed algorithms estimate the RT channel while performing SIC, and the multi-layered structure ensures improved performance across various interference-to-signal ratios. The M-estimate function enhances the robustness of the algorithm, allowing it to converge even when affected by impulsive noise. For nonlinear SIC, this paper introduces an adaptive algorithm based on generalized Hammerstein polynomial basis functions. The simulation results indicate that this approach achieves a better convergence speed and normalized mean squared difference compared to existing SIC methods, leading to a lower system bit error rate.

1. Introduction

Full duplex (FD) relay technology enables a communication system to simultaneously transmit and receive signals on the same frequency through a relay node [1]. Inspired by this technology, simultaneous transmit and receive (STAR) technology, also referred to as in-band full-duplex (IBFD), has emerged as a novel innovation within the field. It aims to allow communication users to directly transmit and receive signals at the same carrier frequency, facilitating an even more efficient use of spectral resources. Moreover, STAR technology can be synergistically combined with other advanced technologies that enhance spectral utilization, such as reconfigurable intelligent surfaces [2], millimeter-wave technology [3], and multiple access techniques [4]. As a result, STAR technology has arisen as a promising approach to increase data throughput and solve the problem of scarce spectrum resources, which can be applied in the field of wireless communications, radar, satellites, and unmanned aerial vehicles [5,6].

However, the self-interference (SI) signal, which is generated from the near end, can be 100–120 dB stronger than the weak received signal, thus making the receiver inoperable [7]. Therefore, self-interference cancellation (SIC) technology is key to the implementation of STAR communications. At present, the SI signal can be eliminated in several stages and various domains, as shown in Figure 1. Typically, SIC is achieved through the cascaded antenna domain [8], analog domain [9], and digital domain [10,11,12,13,14,15]. The antenna domain SIC is passive in nature, and mainly uses antenna isolation or related beamforming algorithms to prevent RF reception path blockage. In the analog domain, directly coupled interference suppression and digital-assisted analog interference-suppression methods are commonly used at the receiver low-noise amplifier (LNA) input before the analog-to-digital conversion. The last stage is digital SIC, which aims to eliminate the remaining SI from the received signal by reconstructing the SI signal in the receiving link and reducing its power below the noise floor. These steps have been applied in some prototypes and have demonstrated the potential of STAR communications [7,16]. The remainder of this paper focuses on digital SIC algorithm design and both the linear and nonlinear SIC algorithms are evaluated through extensive simulations, demonstrating their effectiveness in improving the performance of STAR communication systems.

Digital SIC has attracted significant attention in recent years to address the SI problem in STAR communication systems. Nonetheless, the practical application of digital SIC still faces many challenges. First of all, the observed SI signal exhibits significant frequency and time selectivity due to reflections within the device and its surrounding environment, which demonstrate temporal variations. This can be regarded as a classic system identification problem, which is typically solved through adaptive finite impulse response (FIR) filters. Secondly, in practical communication environments, there are often additional sources of noise, such as electromagnetic interference, lightning noise, radar signals, and other human-made disturbances. These types of noise typically exhibit strong pulse-like characteristics within very short time durations, which can lead to performance degradation of various filtering algorithms based on the assumption of Gaussian noise. Furthermore, the significantly higher power of the SI signal compared to the desired signal at the receiver implies that even minor distortions can significantly degrade the signal of interest. These hardware impairments, such as phase noise, in-phase/quadrature (I/Q) imbalance, and power amplifier (PA) and baseband nonlinearity, among others, with PA nonlinearity are especially harmful to the digital canceler, making digital SIC insufficient.

Digital cancellation algorithms accounting for impulsive noise have been proposed in earlier research [10,17]. The work in [11,12] focused on linear SIC by employing the least mean square (LMS) adaptive filter. In addition, different linear SIC methods were compared in [13]. To compensate for nonlinear PA distortion, the authors in [18] proposed novel predistorters and their parameter extraction algorithms. The proposed self-adaptive nonlinear digital cancelers in [14,15] both utilize a novel orthogonalization procedure for nonlinear basis functions, together with low-cost LMS-based parameter learning. Both PA nonlinearity and I/Q imbalances were considered in [19,20], and more impairments, including local oscillator leakage and baseband nonlinearity, were analyzed and measured on the Universal Software Radio Peripheral in [7]. Some studies have also proposed various frequency domain SIC methods [21,22]. In [21], a frequency domain SI canceler based on the parallel Hammerstein (PH) model for an IBFD orthogonal frequency division multiplexing System was proposed, while Ref. [22] utilized a successive cancellation cascaded structure as a replacement for the adaptive filter, effectively reducing the computational load of SIC processing and simplifying the procedure. Most published works are based on the scenario where only the own transmitter (TX) signal is known, without considering the remote transmission (RT) signal. When the RT sends a signal for channel estimation, it can be assumed to be known, and additional prior information will further enhance the performance of SIC [23].

In this paper, a robust multi-layered M-estimate total least mean square (m-MTLS) canceler is proposed to enhance the performance in scenarios where both the data matrix and the observation signal are also affected by impulsive noise. The method exhibits significant robustness against impulse noise, managing to maintain a low convergence error even in the presence of impulse noise. When both the PA nonlinearities of the local transmitter and the training signal from RT are taken into account, we introduce a novel algorithm for nonlinear digital SIC based on a set of generalized Hammerstein polynomials (HPs) [15], by estimating the SI channel and the RT channel at the same time. This method of joint estimation can more accurately estimate the remote channel while reducing the residual SI signal, compared with existing SIC methods. The simulation results show that this method exhibits a faster convergence rate and demonstrably enhances system performance as the signal-to-noise ratio (SNR) increases.

The rest of this paper is organized as follows. In Section 2, the basic STAR system and linear SI signal modeling are first described. Then, considering the nonlinearity introduced by the PA, the SI model is extended to a nonlinear model, followed by the introduction of a joint estimator that can simultaneously estimate both SI and RT channels. Section 3 proposes two new algorithms for linear and nonlinear SIC based on adaptive FIR, respectively. The linear SIC also enhances the robustness of the algorithm under impulsive noise interference. The simulation results are reported in Section 4. Finally, Section 5 concludes the paper.

In the following, continuous and discrete time signals are expressed in italic lowercase, while vectors are denoted by bold lowercase symbols. The acronyms used throughout the paper are also summarized in

Table 1. Table of acronyms.

Acronym	Definition
STAR	simultaneous transmit and receive
IBFD	in-band full-duplex
SI	self-interference
SIC	self-interference cancellation
RF	Radio Frequency
LNA	low-noise amplifier
PA	power amplifier
LMS	least mean square
PH	parallel Hammerstein
TX	transmitter
RT	remote transmitter
HP	Hammerstein polynomials
SNR	signal-to-noise ratio
ISR	interference-to-signal ratio
m-MTLS	multi-layered M-estimate total least squares
NMSD	normalized mean squared difference
MSE	mean squared error
BER	bit error rate
DFE	decision feedback equalizer

.

2. System Model

The structure of the analyzed STAR system is presented in Figure 2, with signals propagating at the different stages. We denote the baseband transmission at time slot n as $x\left[n\right]$, which is perfectly known. In this analysis, the RF components are assumed to be ideal and the nonlinear model will be discussed later. The received signal, after being processed by the LNA and and subjected to SIC in the analog domain, is converted into a baseband signal by the ADC at time slot n and can be written as

$$\begin{array}{cc}\hfill y\left[n\right]& =r\left[n\right]+f\left(x\right[n],x[n-1],\dots x[n-L+1\left]\right)+v\left[n\right]\hfill \\ \hfill & =r\left[n\right]+s\left[n\right]+v\left[n\right],\hfill \end{array}$$

(1)

where $r\left[n\right]$ and $v\left[n\right]$ denote the desired signal from the remote source and noise at time instant n, respectively. The received signal is assumed to be generated by a function $f(·)$ with L previous transmitted signals. Furthermore, the SI signal from the local TX, $s\left[n\right]$, can be represented by a linear model as

$$\begin{array}{cc}\hfill s\left[n\right]& =f\left(x\right[n],x[n-1],\dots ,x[n-L+1\left]\right)\hfill \\ \hfill & ={\mathit{w}}^T\left[n\right]{\mathit{x}}_L\left[n\right],\hfill \end{array}$$

(2)

where $\mathit{w}\in {\mathbb{R}}^{N\times 1}$ denotes the SI channel and ${\mathit{x}}_L\left[n\right]=[x\left[n\right],x[n-1],\dots ,x[n+L-1]]$.

When the nonlinear hardware impairments are taken into account, one particularly important imperfection is PA nonlinearity; thus, we assume that the transmitted signal is only distorted by high-order harmonics of the PA, excluding the effects of I/Q imbalance or phase noise. The PAs exhibit nonlinearity when power-efficient operation is sought, and it is commonly modeled with the well-known PH model [14,19,22,24]. With an input vector ${\mathit{x}}_L\left[n\right]$, the output of a P-th order model can be expressed as

$$\begin{array}{cc}\hfill x_{\mathrm{PA}}\left(t\right)& =\sum _{\begin{array}{c}p=1\\ p\mathrm{odd}\end{array}}^{\frac{P+1}{2}}{h}_{p,\mathrm{PA}}\left(t\right)\ast {\varphi }_p\left(x\left(t\right)\right),\hfill \end{array}$$

(3)

where P is the highest nonlinearity order of the model, $h_{p,PA}\left(t\right)$ represents the response, and ${\varphi }_p\left(x\left(t\right)\right)={\left|x\right(t\left)\right|}^{2p}x\left(t\right)$ is the pth-order basis function. The symbol ∗ represents the convolution operation. We assume that the PA memory length is $L_1$ and the length of the SI channel is $L_2$. Then, the observed SI signal passes through the SI channel and can be rewritten as

$$\begin{array}{cc}\hfill s\left[n\right]& =\sum _{\begin{array}{c}p=1\\ p\mathrm{odd}\end{array}}^{\frac{P+1}{2}}\sum _{l=0}^{L-1}{w}_p\left[l\right]{\varphi }_p\left(x[n-l]\right),\hfill \end{array}$$

(4)

where ${\mathit{w}}_p$ is the response weight of the entire channel, with a length of $L=L_1+L_2-1$.

Building on the analysis presented earlier, for the purpose of mitigating residual SI within the digital domain, it is imperative to acquire the optimal estimate of the SI channel response, denoted as $\widehat{\mathit{w}}\left[n\right]$, corresponding to $\mathit{w}\left[n\right]$. This involves subtracting the reconstructed SI signal from $y\left[n\right]$. Followed by the SIC process, the remainder of the signal is presumed to embody the RT signal. Subsequent stages may include RT channel estimation and equalization, facilitating the detection of data symbols embedded within the RT signal.

From the SI model mentioned in (2) and (4), it is evident that the performance of digital SIC is fundamentally contingent upon the accuracy of the channel response coefficient estimation. Moreover, the power of the residual SI signal remains high after RF cancellation. when the interference-to-signal ratio (ISR) is high, even minimal residuals can severely compromise the separation of the RT signal. Additionally, at higher SNRs, noise ceases to be the predominant factor impacting the estimation of the SI signal. Under such circumstances, the primary source of error in SIC emanates from the RT signal itself. Consequently, in scenarios where the RT signal is known, it becomes essential to employ a joint channel estimator to enhance the precision of both SI and RT signal separations, thereby mitigating the aforementioned challenges. We assume that the original RT signal, $i\left[n\right]$, is known; then, (1) and (2) can be rewritten in [23] as

$$\begin{array}{c}\hfill y\left[n\right]={\mathit{w}}^T\left[n\right]\mathit{x}\left[n\right]+{\mathit{h}}^T\left[n\right]\mathit{i}\left[n\right]+v\left[n\right],\end{array}$$

(5)

where $h\left[n\right]$ denotes the RT channel coefficient, and the interest signal, $r\left[n\right]$ is represented as ${\mathit{h}}^T\left[n\right]\mathit{i}\left[n\right]$.

By merging two channels into a joint channel, according to (5), we can obtain the following expression:

$$\begin{array}{c}\hfill y\left[n\right]={\mathit{c}}^T\left[n\right]\mathit{u}\left[n\right]+v\left[n\right],\end{array}$$

(6)

where ${\mathit{c}}^T\left[n\right]=[{\mathit{w}}^T\left[n\right],{\mathit{h}}^T\left[n\right]]$ combines the coefficients of both the SI and RT channels, and $\mathit{u}\left[n\right]={[{\mathit{x}}^T\left[n\right],{\mathit{i}}^T\left[n\right]]}^T$ is the new input signal.

In (5), the estimation of SI is influenced by both $r\left[n\right]={\mathit{h}}^T\left[n\right]\mathit{i}\left[n\right]$ and $v\left[n\right]$. However, by combining the SI and RT channel as a new estimation parameter, only the noise $v\left[n\right]$ impacts this process. The benefits of this joint channel estimation approach will be detailed in the simulations presented in Section 4.

3. Proposed Digital Canceler

When considering the SI and RT channels, it is important to acknowledge that they will vary over time, as the environmental reflection paths are also time-variant. Therefore, we opted for adaptive filtering algorithms and incorporated a multi-layered structure, as shown in Figure 3, to enhance the RT estimation performance of the joint estimator.

As depicted in Figure 3, the estimation of joint channel coefficients $\widehat{\mathit{c}}\left[n\right]$ in the first layer is given by ${\widehat{\mathit{c}}}_{\left(1\right)}^T\left[n\right]=[{\widehat{\mathit{w}}}_{\left(1\right)}^T\left[n\right],{\widehat{\mathit{h}}}_{\left(1\right)}^T\left[n\right]]$; then, the residual signal $y_{\left(2\right)}\left[n\right]$, obtained by subtracting the reconstructed SI signal from $y\left[n\right]$, serves as the input for the second layer. Furthermore, we can obtain

$$\begin{array}{cc}\hfill y_{\left(2\right)}\left[n\right]& ={\mathit{c}}_{\left(1\right)}^T\left[n\right]\mathit{u}\left[n\right]-{\widehat{\mathit{w}}}_{\left(1\right)}^T\left[n\right]\mathit{x}\left[n\right]+v\left[n\right]\hfill \\ \hfill & ={\mathit{c}}_{\left(2\right)}^T\left[n\right]\mathit{u}\left[n\right]+v\left[n\right].\hfill \end{array}$$

(7)

This procedure is consistently implemented across successive layers of the filter. In the m-th layer, this process is articulated as

$$\begin{array}{cc}\hfill y_{(m+1)}\left[n\right]& =y_{\left(m\right)}\left[n\right]-{\widehat{\mathit{w}}}_{\left(m\right)}^T\left[n\right]\mathit{x}\left[n\right]+v\left[n\right]\hfill \\ \hfill & ={\mathit{c}}_{(m+1)}^T\left[n\right]\mathit{u}\left[n\right]+v\left[n\right]\hfill \end{array},$$

(8)

where $y_{\left(m\right)}\left[n\right]$ is the input of the m-th layer and ${\widehat{\mathit{w}}}_{\left(m\right)}\left[n\right]$ is the estimation of the SI channel, and we have ${\widehat{\mathit{c}}}_{\left(m\right)}^T\left[n\right]=[{\widehat{\mathit{w}}}_{\left(m\right)}^T\left[n\right],{\widehat{\mathit{h}}}_{\left(m\right)}^T\left[n\right]]$ as the estimation of the joint channel.

For each layer, while subtracting the residual from the received signal, an estimation of the RT channel ${\widehat{\mathit{h}}}_{\left(m\right)}\left[n\right]$ is performed and the output of the last layer $y_{(M+1)}\left[n\right]$ is regarded as the interest signal $\widehat{r}\left[n\right]$.

3.1. Robust m-MTLS Adaptive Algorithm

We consider a linear model where both the independent and dependent variables are subject to measurement errors, which can be illustrated by

$$(y_n+v_n)={({\mathit{x}}_n+{\mathit{u}}_n)}^T\mathit{w},$$

(9)

where $\mathit{w}\in {\mathbb{R}}^{L\times 1}$ denotes the system vector to be estimated, with ${\mathit{x}}_n\in {\mathbb{R}}^{L\times 1}$ as the input vector and $y_n\in \mathbb{R}$ as the output signal at time n. The noise vectors ${\mathit{u}}_n$, distributed as $N(\mathbf{0},{\sigma }_i^2\mathbf{I})\in {\mathbb{R}}^{L\times 1}$, represent the input noise, and $v_n$, distributed as $N(0,{\sigma }_o^2)\in \mathbb{R}$, represent the output noise, where ${\sigma }_i^2$ and ${\sigma }_o^2$ are the variances in the input and output noise, respectively. Here, $\mathbf{0}$ indicates the zero vector and $\mathbf{I}$ the identity matrix.

Total least squares (TLS) considers errors in all variables, making it particularly useful for more accurate modeling and estimation when errors are present in both predictors and outcomes [25]. Its cost function can be formulated as

$$\underset{\mathit{w}}{min}J\left(\mathit{w}\right)=\frac{1}{N}\sum _{n=1}^N\frac{{\left({\tilde{y}}_n-{\mathit{w}}^T{\tilde{\mathit{x}}}_n\right)}^2}{{\parallel \mathit{w}\parallel }^2+\gamma },$$

(10)

where $\gamma \stackrel{\mathrm{def}}{=}{\sigma }_o^2/{\sigma }_i^2$ is a parameter to normalize the noise variances, ${\tilde{y}}_n=y_n+v_n$ and ${\tilde{\mathit{x}}}_n={\mathit{x}}_n+{\mathit{u}}_n$.

Similar to other adaptive algorithms, the iterative formula for TLS based on stochastic gradient descent can be expressed as

$$\begin{array}{cc}\hfill {\widehat{\mathit{w}}}_{n+1}& ={\widehat{\mathit{w}}}_n-\mu \widehat{\mathit{g}}\left({\widehat{\mathit{w}}}_n\right)\hfill \\ \hfill & ={\widehat{\mathit{w}}}_n+\mu {\alpha }_n\left({\tilde{\mathit{x}}}_n+{\alpha }_n{\widehat{\mathit{w}}}_n\right),\hfill \end{array}$$

(11)

where $\mu$ is the step-size parameter, and $\widehat{\mathit{g}}$ is the instantaneous gradient of $J\left(\mathit{w}\right)$, which can be represented as

$$\begin{array}{cc}\hfill & \widehat{\mathit{g}}\left(\mathit{w}\right)={\alpha }_n\left({\tilde{\mathit{x}}}_n+{\alpha }_n{\mathit{w}}_n\right),\hfill \\ \hfill & {\alpha }_n\stackrel{\mathrm{def}}{=}\frac{{\tilde{y}}_n-{\tilde{\mathit{x}}}_n^T{\mathit{w}}_n}{{∥{\mathit{w}}_n∥}^2+\gamma }.\hfill \end{array}$$

(12)

To enhance the robustness of the algorithm when the received signal is subjected to impulsive noise, an M-estimate function can be utilized to improve TLS. This improved algorithm, referred to as the M-estimate total least mean square (MTLS) adaptive algorithm, has its cost function defined as

$$\underset{\mathit{w}}{min}J\left(\mathit{w}\right)=E\left[\frac{\rho \left(e_n^2\right)}{{\parallel \mathit{w}\parallel }^2+\gamma }\right],$$

(13)

where $\rho (·)$ is the M-estimate function, which is a real-valued even function, given by

$$\rho \left(e_n\right)=\left\{\begin{array}{cc}{e}_n^2/2,\hfill & |e_n|<\xi \hfill \\ {\xi }^2/2,\hfill & |e_n|\ge \xi ,\hfill \end{array}\right.$$

(14)

where $\xi =c_1\widehat{{\sigma }_e}$ represents the threshold parameter, with $c_1$ set to 2.576, and $\widehat{{\sigma }_e}$ can be calculated by

$${\widehat{\sigma }}_{e_n}^2={\lambda }_{\sigma }{\widehat{\sigma }}_{e_{n-1}}^2+c_2\left(1-{\lambda }_{\sigma }\right)\mathrm{med}\left(A_{e_n}\right),$$

(15)

where ${\mathit{A}}_e\left[n\right]=\left[e^2\left[n\right],e^2[n-1],\dots ,e^2\left[n-N_w+1\right]\right]$, $\mathrm{med}(·)$ denotes the median operation, and the parameter ${\lambda }_{\sigma }$, indicative of the weighting factor, is typically selected within the range of 0.98 to 0.99. The constant $c_2=1.483\left(1+5/\left(N_w-1\right)\right)$, and $N_w$ represents the length of the window over which the estimation is performed.

The iterative Formula (11) can be rewritten as

$$\widehat{\mathit{g}}\left(\mathit{w}\right)=\left\{\begin{array}{cc}-\frac{\left({\parallel \mathit{w}\parallel }^2+\gamma \right)e_n{\tilde{\mathit{x}}}_n+e_n^2\mathit{w}}{{\left({\parallel \mathit{w}\parallel }^2+\gamma \right)}^2},\hfill & |e_n|<\xi \hfill \\ 0,\hfill & |e_n|\ge \xi ,\hfill \end{array}\right.$$

(16)

and

$$\begin{array}{cc}\hfill {\widehat{\mathit{w}}}_{n+1}& ={\widehat{\mathit{w}}}_n-\mu \widehat{\mathit{g}}\left({\widehat{\mathit{w}}}_n\right).\hfill \end{array}$$

(17)

By integrating the MTLS adaptive algorithm with the multi-layered joint channel estimator discussed in Section 2, a robust linear SI canceler can be achieved [10].

3.2. Multi-Layered Generalized HP-Based Adaptive Algorithm

Nonlinear SIC can be regarded as a problem of nonlinear system identification. Before engaging in adaptive filtering, it is necessary to linearize the SI signal using the basis functions described in (4).

The nonlinear PA model first proposed in [15] employs a set of generalized HPs for its representation. The p-th order nonlinear basis function in Equation (3) is defined as

$$\begin{array}{c}\hfill {\varphi }_p\left(x\left(t\right);{\mathit{c}}_p\right)=\sum _{k=0}^{p-1}{c}_{p,k}{\left|x\left(t\right)\right|}^{2k}x\left(t\right)\end{array},$$

(18)

which can be rewritten in the digital domain as

$$\begin{array}{c}\hfill {\varphi }_p\left(x\left[n\right];{\mathit{c}}_p\right)=\sum _{k=0}^{p-1}{c}_{p,k}{\left|x\left[n\right]\right|}^{2k}x\left[n\right]\end{array},$$

(19)

where ${\mathit{c}}_p={\left[c_{p,0},\dots c_{p,p-1}\right]}^T$ represents the polynomial coefficients of the p-th basis function.

When the polynomial coefficients satisfy $c_{p,p-1}=1$ and $c_{p,k}=0,\forall k\ne p-1$, the basis function simplifies to the standard HP basis function, i.e., ${\varphi }_p\left(x\left[n\right]\right)={\left|x\left[n\right]\right|}^{p-1}x\left[n\right]$. Then, the SI signal $s\left[n\right]$ can be represented by the aforementioned generalized HP as

$$\begin{array}{cc}\hfill s\left[n\right]& =\sum _{p=1}^{\frac{P+1}{2}}\sum _{l=0}^{L-1}{w}_p\left[l\right]{\varphi }_p\left(x\left[n-l\right];{\mathit{c}}_p\right)\hfill \\ \hfill & =\sum _{p=1}^{\frac{P+1}{2}}\sum _{l=0}^{L-1}{w}_p\left[l\right]\left(\sum _{k=0}^{p-1}{c}_{p,k}{\left|x\left[n-l\right]\right|}^{2k}x\left[n-l\right]\right).\hfill \end{array}$$

(20)

To enhance the convergence speed of the adaptive filtering algorithm, it is also necessary to determine the coefficients ${\mathit{c}}_p={\left[c_{p,0},\dots c_{p,p-1}\right]}^T$ of the generalized HP basis functions to ensure their orthogonality. A set of orthonormal basis functions is defined as $\left\{{\varphi }_1\left(x_L\left[n\right];{\mathit{c}}_1\right),\dots ,{\varphi }_{\frac{P+1}{2}}\left(x_L\left[n\right];{\mathit{c}}_{\frac{P+1}{2}}\right)\right\}$, which satisfies the following orthogonality condition:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}E\left[{\varphi }_p\left(x\left[n\right];{\mathit{c}}_p\right){\varphi }_j\left(x\left[n\right];{\mathit{c}}_j\right)\right]=1,p=j\hfill \\ E\left[{\varphi }_p\left(x\left[n\right];{\mathit{c}}_p\right){\varphi }_j\left(x\left[n\right];{\mathit{c}}_j\right)\right]=0,p\ne j\hfill \end{array}\right..\end{array}$$

(21)

For polynomial basis functions with the highest order term of $2p-1$$(p>1)$, assuming orthogonality with any basis function of order less than $2p-1$, it is possible to derive $p-1$ orthogonal equations as

$$\begin{array}{c}\hfill E\left[{\varphi }_p\left(x\left[n\right];{\mathit{c}}_p\right){\varphi }_k\left(x\left[n\right];{\mathit{c}}_k\right)\right]=0,k=1,2,\dots ,p-1\end{array}.$$

(22)

Furthermore, we can obtain

$$\begin{array}{c}\hfill \left\{\begin{array}{c}E\left[{\varphi }_p\left(x\left[n\right];{\mathit{c}}_p\right)c_{1,0}x\left[n\right]\right]=0,\left(k=1\right)\hfill \\ E\left[{\varphi }_p\left(x\left[n\right];{\mathit{c}}_p\right)\left(c_{2,0}x\left[n\right]+c_{2,1}{\left|x\left[n\right]\right|}^{2}x\left[n\right]\right)\right]=0,\left(k=2\right)\hfill \\ ⋮\hfill \\ E\left[{\varphi }_p\left(x\left[n\right];{\mathit{c}}_p\right)\left(c_{p-1,0}x\left[n\right]+\cdots +c_{p-1,p-2}{\left|x\left[n\right]\right|}^{2p-2}x\left[n\right]\right)\right]=0,\left(k=p-1\right).\hfill \end{array}\right.\end{array}$$

(23)

Assume $c_{p,p-1}=1,\forall p$. When $k=1$, we define ${\mu }_p=E\left[{\left(x\left[n\right]\right)}^p\right]$; then, (23) can be rewritten as

$$\begin{array}{c}E\left[\left(c_{p,0}x\left[n\right]+c_{p,1}{\left|x\left[n\right]\right|}^{2}x\left[n\right]+\cdots +{\left|x\left[n\right]\right|}^{2p-2}x\left[n\right]\right)x\left[n\right]\right]\hfill \\ =E\left[\left(c_{p,0}{\left|x\left[n\right]\right|}^2+c_{p,1}{\left|x\left[n\right]\right|}^4+\cdots +{\left|x\left[n\right]\right|}^{2p}\right)\right]\hfill \\ =c_{p,0}{\mu }_2+c_{p,1}{\mu }_4+\cdots +{\mu }_{2p}\hfill \\ =0\hfill \end{array},\left(k=1\right).$$

(24)

Let ${\mathit{\mu }}_{2a}^{2b}=\left\{{\mu }_{2a},{\mu }_{2a+2},\dots ,{\mu }_{2b}\right\},1⩽a⩽b$. Equation (23) can be written in matrix form as

$${\mathbf{C}}_p{\mathbf{M}}_p{\overline{\mathit{c}}}_p+{\mathbf{C}}_p{\mathit{\mu }}_{2p}^{4p-4}=0,$$

(25)

where ${\overline{\mathit{c}}}_p$ is a subvector of the ${\mathit{c}}_p={\left[{{\overline{\mathit{c}}}_{\mathit{p}}}^{T}1\right]}^T$ and ${\mathbf{C}}_p$ is an invertible lower triangular matrix. The matrix ${\mathbf{M}}_p$ is defined as

$${\mathbf{M}}_p=\left[\begin{array}{cccc}{\mu }_2& {\mu }_4& \dots & {\mu }_{2p-2}\\ {\mu }_4& {\mu }_6& \dots & {\mu }_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ {\mu }_{2p-2}& {\mu }_{2p}& \dots & {\mu }_{4p-6}\end{array}\right]\mathrm{and}det\left({\mathbf{M}}_p\right)\ne 0$$

(26)

Thus, we have

$${\overline{\mathit{c}}}_p=-{{\mathbf{M}}_p}^{-1}{\mathit{\mu }}_{2p}^{4p-4}.$$

(27)

The instantaneous basis function vector is defined as

$$\Phi \left[n\right]={\left[{\mathit{\varphi }}_1\left({\mathit{x}}_L\left[n\right];{\mathit{c}}_1\right),\dots ,{\mathit{\varphi }}_{\frac{P+1}{2}}\left({\mathit{x}}_L\left[n\right];{\mathit{c}}_{\frac{P+1}{2}}\right)\right]}^T.$$

(28)

Integrating the nonlinear PA model with the multi-layered joint channel estimator employing the LMS algorithm results in

$$\begin{array}{cc}\hfill y\left[n\right]& ={\mathit{w}}^T\left[n\right]\Phi \left[n\right]+{\mathit{h}}^T\left[n\right]\mathit{i}\left[n\right]+v\left[n\right]\hfill \\ \hfill & =\left[{\mathit{w}}^T\left[n\right]{\mathit{h}}^T\left[n\right]\right]{\left[{\Phi }^T\left[n\right]{\mathit{i}}^T\left[n\right]\right]}^T+v\left[n\right]\hfill \\ \hfill & ={{\mathit{h}}^T}_{joint}\left[n\right]\mathit{u}\left[n\right]+v\left[n\right],\hfill \end{array}$$

(29)

where ${\mathit{h}}_{joint}^T\left[n\right]=\left[{\mathit{w}}^T\left[n\right]{\mathit{h}}^T\left[n\right]\right]$ and $\mathit{u}\left[n\right]={\left[{\Phi }^T\left[n\right]{\mathit{i}}^T\left[n\right]\right]}^T$.

Furthermore, the iterative formula can be expressed as

$${\widehat{\mathit{h}}}_{joint}\left[n+1\right]={\widehat{\mathit{h}}}_{joint}\left[n\right]+\mu \mathit{u}\left[n\right]e\left[n\right],$$

(30)

where $\mu$ is the step size.

At the m-th layer, the received signal after digital SIC is

$$\begin{array}{c}\hfill y_{(m+1)}\left[n\right]=y_{\left(m\right)}\left[n\right]-{\widehat{\mathit{h}}}_{joint\left(m\right)}^T\left[n\right]\Phi \left[n\right]+v\left[n\right].\end{array}$$

(31)

The M-layered adaptive nonlinear SIC algorithm is summarized in Algorithm 1 and

Table 2. Table of notations.

Parameter	Symbol
p-th order basis functions	${\varphi }_p\left(x\left[n\right];{\mathit{c}}_p\right)$
Coefficients of the p-th basis function	${\mathit{c}}_p$
Nonlinear order of PA	P
p-th SI channel response	${\mathit{w}}_p$
p-th RT channel response	${\mathit{h}}_p$
Joint channel response	${\mathit{h}}_{joint}$
p-th even-order partial moments	${\mathbf{M}}_p$
Number of layers	M
Input of m-th layers	$y_{\left(m\right)}$

.

Algorithm 1 Adaptive algorithm for nonlinear SIC

Input:  received signal $y\left[n\right]$, known signals $x\left[n\right]$ and $i\left[n\right]$, basis order P, moment collect length $N_m$ and basis function generation interval $N_{int}$ Output:  signal after SIC $\widehat{r}\left[n\right]$ and RT channel estimation ${\widehat{\mathit{h}}}_{\left(m\right)}^T\left[n\right]$      Initialization:  $n_s=1$ and ${\mathit{h}}_{joint\left(m\right)}^T\left[n\right]=\mathbf{0}$ 1: for  $n=1,2,\dots ,N$  do 2:       if $n=n_s+N_m-1$ then 3:            ${\mu }_p=\frac{1}{N}{\sum }_{n_s}^{n_s+N_m-1}{\left(x\left[n\right]\right)}^p$ 4:            for $p=2,3,\dots ,\frac{P+1}{2}$ do 5:                  ${\overline{\mathit{c}}}_p=-{{\mathbf{M}}_p}^{-1}{\mathit{\mu }}_{2p}^{4p-4}$, basis function ${\varphi }_p\left(x\left[n\right];{\mathit{c}}_p\right)$ generation 6:            end for 7:       end if 8:       if $n=n_s+N_{int}-1$ then 9:            $n_s=n$ 10:     end if 11:     $y_{\left(1\right)}\left[n\right]=y\left[n\right]={\mathit{h}}_{joint}^T\left[n\right]\mathit{u}\left[n\right]+v\left[n\right]$ 12:     for $m=1,2,\dots ,M$ do 13:          $e_{\left(m\right)}\left[n\right]=y_{\left(m\right)}\left[n\right]-{\widehat{\mathit{h}}}_{joint\left(m\right)}^T\left[n\right]\mathit{u}\left[n\right]$ 14:          ${\widehat{\mathit{h}}}_{joint\left(m\right)}^T[n+1]={\widehat{\mathit{h}}}_{joint\left(m\right)}^T\left[n\right]+\mu \mathit{u}\left[n\right]e_{\left(m\right)}\left[n\right]$ 15:          ${\widehat{\mathit{w}}}_{\left(m\right)}^T\left[n+1\right]={\left({\widehat{\mathit{h}}}_{joint\left(m\right)}^T[n+1]\right)}_{1:N_1}$ 16:          ${\widehat{\mathit{h}}}_{\left(m\right)}^T\left[n+1\right]={\left({\widehat{\mathit{h}}}_{joint\left(m\right)}^T\left[n\right]\right)}_{N_1+1:N_1+N_2}$ 17:          $y_{(m+1)}\left[n\right]=y_{\left(m\right)}\left[n\right]-{\widehat{\mathit{w}}}_{\left(m\right)}^T\left[n\right]\mathit{x}\left[n\right]$ 18:     end for 19:     $\widehat{r}\left[n\right]=y_{(m+1)}\left[n\right]$ 20: end for

4. Simulation Results

4.1. Linear SI Canceler

In the scenario described by Equation (9), the unknown vector $\mathit{w}$, with a dimensionality of $L=14$, conforms to the condition ${\parallel \mathit{w}\parallel }^2=1$. Additionally, the elements of $\mathit{w}$ are distributed according to a Gaussian distribution with a mean of zero. The input signal is independently generated from zero-mean Gaussian with unit variance. The input and output noises are denoted as $v_{in}=v_b$ and $v_{out}=v_a+v_i$, respectively, where $v_a$ and $v_b$ are Gaussian-distributed with equal variances of ${\sigma }_a^2={\sigma }_b^2=0.1$. The Bernoulli–Gaussian (BG) process is utilized as an impulsive noise model. The impulsive noise $v_i\left[n\right]=b\left[n\right]p\left[n\right]$, where $b\left[n\right]$ is a Bernoulli process with the probability density function $P(b\left[n\right]=1)=P_i,P(b\left[n\right]=0)=1-P_i$ and $p\left[n\right]\sim N(0,10)$ [26]. Their performance is displayed in Figure 4, quantitatively evaluated via the normalized mean squared difference (NMSD), defined as $=10log\left(\parallel {\widehat{\mathit{w}}}_n{-\mathit{w}\parallel }^2/{\parallel \mathit{w}\parallel }^2\right)$. It can be observed that the LMS and TLS algorithms decline due to noise interference, whereas the MTLS algorithm remains largely unaffected by such disturbances.

We consider a baseband STAR system characterized by a 5 MHz signal bandwidth. The system operates at a sampling rate of 10 MHz, with both the local and remote transmitters employing binary phase shift keying modulation for signal transmission. The SI and RT channels have lengths specified as $N_1=4$ and $N_2=10$, respectively. Furthermore, their average energy follows the conditions ISR $=E(\parallel w{\parallel }^2/\parallel h{\parallel }^2)$ and SNR $=E(\parallel s\left(n\right){\parallel }^2/{\sigma }_a^2)$. The received signal $y\left(n\right)$ is subject to corruption from two sources: Gaussian white noise $v_a$ and random impulsive noise $v_i$. The occurrence probability of the impulsive noise is denoted as $P_i$. Additionally, the local reference signal $i\left(n\right)$ is influenced by Gaussian noise $v_b$, where the condition ${\sigma }_a^2={\sigma }_b^2$ holds true. In the context of this simulation, we employ the m-MTLS algorithm with a total of number of $M=2$ layers, which is mainly a compromise considering the additional computational cost brought by multi-layer filters. As M increases, the effectiveness of SIC initially improves and then tends to stabilize or even decline. This is due to the propagation of estimation errors across different layers. To assess the efficacy of the proposed multi-layered joint estimator, a comparative analysis is conducted against the minimum mean square error (MMSE) estimator and the single-layer MTLS estimator, focusing on their performance in estimating the RT channel. The physical meaning of the parameters involved in this simulation is summarized in

Table 3. Table of parameters.

Parameter	Symbol	Value
Length of SI channel	$N_1$	4
Length of RT channel	$N_2$	10
Layers	M	2
Input noise	$v_{in}$	$v_b$
Output noise	$v_{out}$	$v_a+v_i$
Gaussian noise	$v_a$, $v_b$	${\sigma }_a^2={\sigma }_b^2$
Impulsive noise	$v_i$	${\sigma }_i^2=10$
ISR	-	20 dB

.

Figure 5 illustrates the performance curves of several algorithms in terms of NMSD as the SNR varies across different noise environments, with ISR $=20$ dB. Compared to the MMSE estimator, the proposed method exhibits higher robustness. The MMSE estimator shows a noticeable decline in performance at certain sampling points when affected by impulsive noise pollution. However, thanks to the M-estimate function, the m-MTLS estimator remains virtually unaffected. Furthermore, as the SNR increases, the NMSD of the m-MTLS estimator gradually decreases, whereas the MMSE estimator experiences limited gain due to convergence being disrupted by interference from the interest signal.

4.2. Nonlinear SI Canceler

In this section, We use the Saleh’s PA model [27] to simulate the nonlinear distortion, and the SI signal received after ADC can be described as follows:

$$f\left(x\left[n\right]\right)=\sum _{l=0}^{L-1}h\left[l\right]\frac{\gamma x\left[n-l\right]}{1+\beta {\left|x\left[n-l\right]\right|}^2},$$

(32)

where $\gamma =3$ and $\beta =0.03$ in this simulation, and $h\left[n\right]$ is the overall memory length, set to $L=5$.

We assess the performance in terms of the mean squared error (MSE), defined as follows:

$$MSE=\mathrm{E}\left[{\left(f\left(\mathit{x}\left[n\right]\right)-\widehat{y}\left[n\right]\right)}^2\right].$$

(33)

The performance of the proposed algorithm and the commonly used LMS algorithms based on HPs are evaluated and the results are shown in Figure 6. In this simulation, we assume the transmitted data follow a uniform distribution in the range $[-2.5,2.5]$ and $v\left[n\right]$ is zero-mean Gaussian noise with variance 1.

As shown in Figure 6, compared to existing algorithms based on standard HPs, the proposed algorithm achieves faster convergence while maintaining equivalent MSE performance.

In the nonlinear SIC scenario, we simulate data transmission using a uniform distribution within the range of $[-2.5,2.5]$, and the remote signal $i\left[n\right]\in \{-1,1\}$. The nonlinear SI canceler is demonstrated in Section 3.2. Considering both computational complexity and algorithm performance, the joint SIC algorithm has a two-layer filtering structure with $M=2$. The basis functions for all algorithms are of order up to five.

Figure 7 and Figure 8 compare the power spectrum of different signals at the receiver for ISR $=20$ dB and ISR $=40$ dB, respectively. As seen, in both cases, the signal after SIC more closely approximates the remote signal when using the proposed SIC algorithm compared to the method based on HP-LMS.

Figure 9 shows the NMSD and bit error rate (BER) performance of the proposed algorithm and other SIC algorithms. For symbol detection at the receiver, a decision feedback equalizer (DFE) is employed, with both the feedforward and feedback filters having lengths of 15. The DFE employs a truncated version of the transmitted signal as the training signal, with tap weights being trained exclusively during the first block. In this result, as expected, the joint SIC algorithm with a generalized HP basis function with orthogonality exhibits superior performance.

5. Conclusions

In this paper, we proposed two adaptive algorithms for linear and nonlinear SIC, based on a multi-layered joint channel estimator structure. In the case of linear SI, given that both the independent and dependent variables are subject to measurement errors, and considering the impact of impulsive noise present in wireless environments on the convergence process of adaptive algorithms, the m-MTLS algorithm is proposed to enhance robustness. For the nonlinear distortion generated by the PA, we combine the joint channel estimator and generalized HP basis function to propose a joint SIC method that outperforms current state-of-the-art SIC algorithms in scenarios of both high and low ISR, as well as high SNR. However, the complexity of this method is higher, mainly due to the increase in the number of basis function coefficients and the length of the filter during joint estimation, as well as the number of layers M, which also contributes to the increased complexity.