Maximum Likelihood Sequence Estimation Optimum Receiver Design with Channel Identification Based on Zero Distribution

Abstract

In this paper, a novel maximum likelihood sequence estimation (MLSE) optimum receiver design is proposed to reduce the complexity of the conventional optimum receiver. We first propose a multipath channel classification method according to the zero distribution of the channel impulse response; that is, the multipath channel is divided into a minimum phase channel and a non-minimum phase channel. Then, an MLSE optimum receiver design with channel identification based on zero distribution is also proposed. In this design, the distribution of zero can be obtained directly by calculating the Z transform of the channel. Thus, the complexity is reduced. The number of the multiplication reduced is $N(2N+1)$. In addition, $2N+1$ memory units storing the autocorrelation sequence are reduced. The simulation results indicate that this channel classification method effectively represents the characteristics of the channel. Moreover, the MLSE optimum receiver proposed performs similarly and is less complex than the conventional one. The performance is greatly improved compared with the minimum mean square error (MMSE) equalizer.

1. Introduction

Multipath transmission in wireless communication systems has been one of the challenges widely studied by scholars in the past decades [1,2]. The fading caused by multipath transmission causes serious distortion of the received signal, which brings great difficulty to the correct detection of the signal at the receiving end [3,4,5]. When deep fading occurs, the linear equalization based on the zero forcing criterion (ZF) [6] and the minimum mean square error criterion (MMSE) [7] cannot reach the error target. Therefore, only nonlinear equalization can be adopted to improve the performance [8,9,10]. Among them, nonlinear equalization based on the maximum likelihood sequence estimation (MLSE) criterion has been proven to be the optimum receiver criterion. In [11], the optimum receiver framework is proposed. Moreover, the MLSE detection is implemented using the Viterbi algorithm (VA).

However, the MLSE optimum receiver is usually highly complex. This fact has prompted many researchers to work on reducing the complexity of it. In [12], a reduced complexity adaptive MLSE receiver is proposed. A tentative decision (Per-survivor Processing) is considered. The novelty of the proposed algorithm is the reduced computational complexity, which is independent of the channel impulse response length. In [13], an MLSE combined equalizer and decoder with soft-decision decoding performance, which has a computational complexity superior to that of conventional MLSE equalization, is proposed. The proposed algorithm’s performance matches conventional soft-decision decoding while eliminating using a decoder in the receiver for frequency-selective Rayleigh fading channels. In [14], an iterative MLSE equalizer with hard outputs with a computational complexity quadratic in the data block and the channel length is proposed. The proposed iterative MLSE detector can detect Binary Phaseshift Keying (BPSK) signals in systems with significantly larger channel lengths than what is possible with the VA for frequency-selective Rayleigh fading channels. In [15], a near-optimum hard output neural network-based iterative MLSE equalizer is proposed to equalize single carrier 4-QAM signals in underwater acoustic channels with extremely long delay spreads. The results show unparalleled performance at a fraction of the computational cost of a conventional MLSE equalizer. In [16], a novel linear equalization and progressive grouping-assisted constrained Viterbi algorithm as a near-maximum likelihood sequence estimation (MLSE) approach for the downlink is proposed. The simulation results show that the reduced-complexity algorithm is highly attractive with large system dimensions such as those in the time-duplexing high-speed packet access system. In [17], the equalization algorithm for non-return-to-zero high-speed transceivers is explored. A low-complexity MLSE algorithm is introduced based on traditional MLSE principles. This algorithm significantly reduces the cumulative computations compared to traditional MLSE, requiring only straightforward assessments to accomplish equalization. The resource utilization of the proposed low-complexity MLSE algorithm has been significantly decreased, facilitating its practical implementation. In [18], a novel MLSE receiver for Internet of Things (IoT) devices is proposed, which exhibits low complexity and close MLSE performance. The reduction in complexity was achieved through a transformation of the received signal. This processing allows the use of two simplified metrics in implementing the VA. The proposed metrics attain a reduction of computational complexity and hardware resources. In [19], to achieve low-complexity equalization in single-carrier underwater acoustic communications, a suboptimal MLSE equalization algorithm, namely the fractional interval decision–feedback sequence estimation equalization algorithm, is proposed. In the simulation experiments, the proposed equalization method is superior to the symbol-by-symbol adaptive channel-estimation-based equalization by approximately 1.2 dB, showing a good performance/complexity tradeoff.

For the optimum receiver, a whitening filter must first be built before MLSE detection [20,21]. The above literature mainly reduces the complexity of the optimum receiver from the perspective of detection, and this paper further reduces the complexity of the optimum receiver from the perspective of building a whitening filter. In order to achieve this goal, we propose a multipath channel classification method according to the zero-pole distribution of the channel impulse response; that is, the multipath channel is divided into a minimum phase channel and a non-minimum phase channel. Unlike the traditional channel classification method, which divides the channel into line-of-sight (LOS) and non-line-of-sight (NLOS) channels [22,23,24], this channel classification method can effectively represent the characteristics of the channel. It is applied to the design of the MLSE optimum receiver. This idea of channel classification provides a reliable channel identification method, which can reduce the complexity of optimum receiver design. The contributions are as follows:

(1) Based on the zero distribution of the channel impulse response, the multipath channel is divided into a minimum phase channel and non-minimum phase channel, and numerical results prove that the zero distribution reflects the channel characteristics. When the zeros of two channels are symmetric about the Z domain unit circle, it will have the same impact on the signal, even if the time domain impulse response is different.

(2) A channel identification method based on zero distribution is proposed, based on which the design of the MLSE optimum receiver is simplified, and the framework of the MLSE optimum receiver is constructed under the conditions of minimum phase and non-minimum phase channels, respectively. Numerical results prove that the MLSE optimum receiver performs similarly to the conventional one and is better than MMSE equalization.

The remainder of the paper is structured as follows. In Section 2, the MLSE conventional optimum receiver model is given. The channel classification based on zero distribution and the MLSE optimum receiver framework is provided in Section 3. Numerical results are shown in Section 4. Conclusions are presented in Section 5.

2. Conventional Optimum Receiver Model

The conventional MLSE optimum receiver model is shown in Figure 1, which consists of the transmitter component, including the shaping filter, and the receiver components, including the waveform matching filter, channel matching filter, whitening filter tap calculator, noise whitening filter, and MLSE detector. The channel module includes the multipath channel and additive Gaussian white noise (AWGN).

2.1. Signal Model

The signal model is represented in discrete signal form. Therefore, the shaping filter and waveform matching filter cannot be considered. Assuming that the input information symbol sequence is $x\left(n\right)$, through the multipath channel $h\left(n\right)$ and adding Gaussian white noise, the received signal $y\left(n\right)$ can be denoted as

$$y\left(n\right)=h\left(n\right)\otimes x\left(n\right)+w\left(n\right),$$

(1)

where ⊗ represents a convolution operation, and $w\left(n\right)$ is a complex AWGN sequence with zero mean. The length of $h\left(n\right)$ is L. At the receiving end, the received signal first passes through the channel matching filter, which is represented as

$$r\left(n\right)=\stackrel{⌣}{h}\left(n\right)\otimes y\left(n\right),$$

(2)

where $\stackrel{⌣}{h}\left(n\right)$ represents the channel matching filter of $h\left(n\right)$. The channel matching filter here is different from the typical waveform matching filter. The waveform matching filter is generally the same as the root-raised cosine shape filter at the transmitting end, which is a low-pass filter. The response of the channel filter here is related to the pulse response of the channel, and the channel matching filter will make the white noise become colored noise, which is not conducive to the detection of subsequent signals. However, the channel matching filter can improve the signal-to-noise ratio (SNR), and the multipath energy can be concentrated to achieve complete multipath diversity.

Without considering noise, then

$$r\left(n\right)=x\left(n\right)\otimes h\left(n\right)\otimes \stackrel{⌣}{h}\left(n\right).$$

(3)

The conventional MLSE optimum receiver first needs to calculate the autocorrelation function of the channel, i.e.,

$$f\left(n\right)=h\left(n\right)\otimes \stackrel{⌣}{h}\left(n\right),$$

(4)

and its bilateral Z transform

$$F\left(z\right)=\sum _{n=-L}^{L}f\left(n\right)z^{-n},$$

(5)

where $f\left(n\right)=f^{*}\left(-n\right)$. Superscripts ^* denote a conjugate, so

$$\begin{array}{cc}\hfill F^{*}\left(1/z^{*}\right)& ={\left(\sum _{n=-L}^{L}f\left(n\right){\left({\displaystyle \frac{1}{z^{*}}}\right)}^{-n}\right)}^{*}=\sum _{k=-L}^L{f}^{*}\left(n\right){\left({\displaystyle \frac{1}{{\left(z^{*}\right)}^{*}}}\right)}^{-n}\hfill \\ \hfill & =\sum _{n=-L}^L{f}^{*}\left(n\right){\left({\displaystyle \frac{1}{z}}\right)}^{-n}\hfill \\ \hfill & \underline{\underline{m=-n}}\sum _{m=-L}^L{f}^{*}\left(-m\right){\left({\displaystyle \frac{1}{z}}\right)}^{-(-m)}\hfill \\ \hfill & =\sum _{m=-L}^{L}f\left(m\right)z^{-m}=F\left(z\right).\hfill \end{array}$$

(6)

$F\left(z\right)$ has L sets of pairwise symmetric roots about the unit circle in the Z domain: if $\lambda$ is a root, then $1/{\lambda }^{*}$ is also a root, so $F\left(z\right)$ can be factored into

$$F\left(z\right)=E\left(z\right)E^{*}\left(1/z^{*}\right),$$

(7)

where $E\left(z\right)$ is a polynomial of degree L with roots ${\lambda }_1\sim {\lambda }_L$, and $E\left(1/z\right)$ is a polynomial of degree L with roots $1/{{\lambda }_1}^{*}\sim 1/{{\lambda }_L}^{*}$.

After the channel-matching filter, the white noise becomes colored noise, which is not conducive to the subsequent detection, so the MLSE optimum receiver needs to add a noise whitening filter, and the Z transform of an appropriate noise whitening filter satisfies $1/E^{*}\left(1/z^{*}\right)$. There are $2^L$ possible options for its roots. It is necessary to choose based on the distribution of the roots of $F\left(z\right)$. In general, the unique $E^{*}\left(1/z^{*}\right)$ with zeros outside the unit circle is chosen, which implies the zeros of $E\left(z\right)$ are inside the unit circle.

Therefore, the sequence $r\left(n\right)$ passes through the whitening filter, yielding

$$v\left(n\right)=\sum _{k=0}^{L}e\left(k\right)x\left(n-k\right)+\eta \left(n\right),$$

(8)

where $\eta \left(n\right)$ is an AWGN sequence.

2.2. MLSE Detector

In the case of intersymbol interference (ISI) caused by multipath transmission covering L symbols, the MLSE criterion can be equivalent to the state estimation problem of a finite state machine (FSM). According to the above, the FSM is equivalent to a discrete-time channel $\left\{e_n\right\}$. At any moment, its state is determined by the L nearest inputs; i.e., at the moment k, the state is

$$S_n=\left\{s_{n-1},s_{n-2},\cdots ,s_{n-L-1}\right\}.$$

(9)

Therefore, if the information symbol is M elements, the channel filter has $M^L$ states. Thus, the channel can be described by an $M^L$-state grid, and the VA can be used to find the most likely path through the network.

VA utilizes dynamic programming (DP) to solve the problem of predicting hidden Markov models, specifically to find the path with the maximum probability. The idea of DP is to decompose the problem into multiple stages, each corresponding to a decision. We record the reachable state of each stage and then derive the state of the next stage from the current stage, moving forward dynamically until we reach the final stage. So, a path corresponds to a sequence of states. Based on the principle of DP, if the optimum path passes through node $i_t^{†}$ at time t, the partial path from node $i_t^{†}$ to the end node $i_T^{†}$ must be optimum. Superscripts ^† indicate an optimum path. Then, it just needs to begin at time $t=1$ and recursively compute the maximum probability of each partial path with state i at time t. Eventually, the maximum probability of each path with state $i_T$ at time $t=T$ is acquired. The maximum probability of time $t=T$ is the probability $P^{†}$ of the optimum path, and the end node $i_T^{†}$ of the optimum path is also obtained. P denotes the probability of an event. Then, to find each node of the optimum path, starting from the end node $i_T^{†}$, nodes $i_{T-1}^{†},i_{T-2}^{†},\cdots ,i_1^{†}$ are gradually obtained from the back to the front, and the optimum path $I^{†}=\left(i_1^{†},i_2^{†},\cdots ,i_T^{†}\right)$ is obtained, which is the VA.

Define the maximum probability of all paths with state $\left(i_1,i_2,\cdots ,i_t\right)$ at the time t as

$$\begin{array}{cc}\hfill {\delta }_t\left(i\right)& =\underset{i_1,i_2,\cdots ,i_{t-1}}{max}P\left(i_t=i,i_{t-1},\cdots ,i_1,o_t,\cdots ,o_1\left|\lambda \right.\right)\hfill \\ \hfill & i=1,2,\cdots ,N.\hfill \end{array}$$

(10)

According to the VA’s relevant definition and solving principle, the recursive formula of variable $\delta$ can be obtained as

$$\begin{array}{cc}\hfill {\delta }_{t+1}\left(i\right)& =\underset{i_1,i_2,\cdots ,i_t}{max}P\left(i_{t+1}=i,i_t,\cdots ,i_1,o_{t+1},\cdots ,o_1\left|\lambda \right.\right)\hfill \\ \hfill & =\underset{1\le j\le N}{max}\left[{\delta }_t\left(j\right)a_{ji}\right]b_i\left(o_{t+1}\right),\hfill \\ \hfill & i=1,2,\cdots ,N;t=1,2,\cdots ,T-1.\hfill \end{array}$$

(11)

As in the previous calculations for each node, ${\delta }_t\left(j\right)$ is multiplied by the transition probability to obtain the probability of the transition from j to i at time t, and then it is multiplied by the observation probability to obtain the maximum probability for all individual paths with state i at time t. The $\left(t-1\right)$-th node of the path $\left(i_1,i_2,\cdots ,i_{t-1},i\right)$ with the most significant probability at time t and state i is

$${\mathrm{\Psi }}_t\left(i\right)=arg\underset{1\le j\le N}{max}\left[{\delta }_{t-1}\left(j\right)a_{ji}\right],i=1,2,\cdots ,N.$$

(12)

The above analysis shows the solution flow of prediction using the VA, as shown in Algorithm 1.

Algorithm 1 Solution flow of prediction using the VA

Input:  Model $\lambda =\left(A,B,\pi \right)$ and observed sequence $O=\left\{o_1,o_2,\cdots ,o_T\right\}$, where A is the state transition probability matrix and B is the observation probability matrix. Output:  Optimum path $I^{†}=\left(i_1^{†},i_2^{†},\cdots ,i_T^{†}\right)$; 1: Initialization: ${\delta }_1\left(i\right)={\pi }_i{b}_i\left(o_1\right),i=1,2,\cdots ,N$ and ${\mathrm{\Psi }}_1\left(i\right)=0,i=1,2,\cdots ,N$; 2: Recursion: For $t=2,3,\cdots ,T$ ${\delta }_t\left(i\right)={\displaystyle \underset{1\le j\le N}{max}}\left[{\delta }_{t-1}\left(j\right)a_{ji}\right]b_i\left(o_t\right),i=1,2,\cdots ,N$ ${\mathrm{\Psi }}_t\left(i\right)=arg{\displaystyle \underset{1\le j\le N}{max}}\left[{\delta }_{t-1}\left(j\right)a_{ji}\right],i=1,2,\cdots ,N$; 3: Termination: $P^{†}={\displaystyle \underset{1\le j\le N}{max}}{\delta }_T\left(i\right)$ $i_T^{†}=arg{\displaystyle \underset{1\le j\le N}{max}}\left[{\delta }_T\left(i\right)\right]$; 4: Optimum path backtracking: $i_t^{†}={\mathrm{\Psi }}_{t+1}\left(i_{t+1}^{†}\right),t=T-1,T-2,\cdots ,1$.

3. Channel Classification Based on Zero Distribution and MLSE Optimum Receiver Framework

3.1. Channel Classification Based on Zero Distribution

Considering the zero distribution of the multipath channel in the Z domain, we divide the channel into a minimum phase channel and non-minimum phase channel. If all the zeros are distributed in the unit circle, the channel is called the minimum phase channel. A channel is classified as a non-minimum phase channel when all or some of its zeros are located outside the unit circle. The zeros of the real channel and complex channel are observed, respectively.

Taking a real channel $h\left(n\right)=\left\{1,0.5,0.3\right\}$ as an example, its zero-pole distribution diagram is shown in Figure 2a. All zeros are in the unit circle, which is a minimum phase channel. The zero-pole distribution in real channel $h\left(n\right)=\left\{0.5,1,0.4\right\}$ is shown in Figure 2b. Some zeros are outside the unit circle, which is a non-minimum phase channel.

Taking a complex channel $h\left(n\right)=\left\{1+0i,-0.4-1.3i,-0.4+0.4i\right\}$ as an example, its zero-pole distribution diagram is shown in Figure 3a. All zeros are in the unit circle, which is a minimum phase channel. The zero-pole distribution in the complex channel $h\left(n\right)=\left\{1+0i,-0.76-1.4i,-0.56+0.56i\right\}$ is shown in Figure 3b. Some zeros are outside the unit circle, which is a non-minimum phase channel.

3.2. MLSE Optimum Receiver Framework with Channel Identification Based on Zero Distribution

In the conventional MLSE optimum receiver, to obtain the whitening filter, the autocorrelation function $F\left(z\right)$ needs to be calculated first after the channel matching filter. Then, $F\left(z\right)$ is factored and the whitening filter $1/E^{*}\left(1/z^{*}\right)$ is determined based on the root of the factor. Its design process is shown in Figure 4a. The whole process is complicated. The MLSE optimum receiver framework with channel identification based on zero distribution proposed in this paper directly obtains the whitening filter through the characteristics of the zero distribution of the channel without calculating the autocorrelation function, simplifying the design process. Its design process is shown in Figure 4b.

The following two cases are analyzed. For the minimum phase channel, there is no need to perform any processing; we can directly obtain the whitening filter $1/H^{*}\left(1/z^{*}\right)$. From the perspective of an autocorrelation function, in the case of the minimum phase channel, it is equivalent to the factorization of autocorrelation function to

$$\left(z\right)=H\left(z\right)H^{*}\left(1/z^{*}\right).$$

(13)

For a non-minimum phase channel, the zeros not in the unit circle are first symmetrically moved into the unit circle to obtain channel $h_a\left(n\right)$, in which case the whitening filter is $1/{H_a}^{*}\left(1/z^{*}\right)$. From the perspective of the autocorrelation function, in the case of the non-minimum phase channel, it is equivalent to the factorization of the autocorrelation function to

$$\left(z\right)=H_a\left(z\right){H_a}^{*}\left(1/z^{*}\right).$$

(14)

The channel identification based on zero distribution can simplify the construction of the MLSE optimum receiver.

From the above analysis, the optimum receiver design proposed in this paper reduces the correlation calculation of the multipath channel compared with the traditional optimum receiver design. The distribution of zero can be obtained directly by calculating the Z transform of the channel. Thus, the complexity is reduced, and the number of the multiplication reduced is $N(2N+1)$. In addition, $2N+1$ memory units storing the autocorrelation sequence are reduced. In the next section, we will show that although the proposed optimum receiver design reduces complexity, the performance remains the same as the traditional optimum receiver.

4. Numerical Results

4.1. Simulation Related to Zero Distribution

In the Z domain, the zero position can be represented by polar coordinates. Assuming that the zero is $\rho =\left|\rho \right|e^{j\theta }$, two parameters affect the zero position: amplitude $\left|\rho \right|$ and phase $\theta$. According to the zero position, we construct several groups of channels for simulation experiments.

Select $\left|\rho \right|=0.5(0.7,0.9)$, $\theta =0\left(\pi /2,\pi ,3\pi /2\right)$ for comparison simulation, and the numerical results are illustrated in Figure 5. It can be seen from the figure that for a two-path channel, there is only one zero, and $\left|\rho \right|$ determines the BER performance. The closer $\left|\rho \right|$ is to the unit circle, the worse the performance is. If the phase of the zero of the channel is different but the mode value of the zero is the same, the performance is consistent. In the first simulation experiment in this section, we simulate the BER under the two-path channel.

We use a three-path channel with two zeros, ${\rho }_1$ and ${\rho }_2$, to simulate the BER in the second simulation experiment. Select a combination with an amplitude of 0.2 (0.5, 0.7, 0.9). The phase of the two zeros takes the same value: $\pi /4$. The simulation results are shown in Figure 6. As can be seen from the figure, as the two zeros gradually approach the unit circle, the system performance deteriorates.

In the third simulation experiment, the three-path channel, which has two zeros, is still selected for simulation. We fix the position of one zero and the amplitude of the other zero and select different phases for simulation where ${\rho }_1=0.7e^{j\pi /4}$, $\left|{\rho }_2\right|=0.9$ and ${\theta }_2=0,\pi /4,\cdots ,7\pi /4$. The numerical results are displayed in Figure 7, in which different positions of zeros in the Z domain are marked with different colors; the same color means that two zeros have the same relative positions. From the BER results, the system performance is related to the relative phase relationship between channel zeros, and the more significant the phase difference between two zeros, the better the performance, while the smaller the phase difference, the worse the performance.

In the fourth simulation experiment, a three-path channel is also selected. Compared with the third simulation experiment, the simulation under the multipath channel with zeros symmetric about the unit circle is added. It can be seen from Figure 8 that if the zeros of the two channels are symmetric about the unit circle, the performance through the two channels is the same.

4.2. Simulation for MLSE Optimum Receiver

In this subsection, the performance of the proposed MLSE optimum receiver design with channel identification based on zeros is compared with the conventional MLSE optimum receiver and MMSE equalizer. The simulation is based on the assumption that perfect synchronization has been attained. The theoretical curve represents the ideal error performance curve of QPSK modulation in the AWGN channel in simulation.

Three three-path channels with two zeros are used in the simulation: Channel 1 is the minimum phase channel, the zeros are all in the unit circle, and the amplitudes are 0.5/0.7. Channel 2 is the minimum phase channel; its zeros are all in the unit circle, and the amplitudes are 0.7/0.9; compared with channel 1, its zeros are closer to the unit circle. Channel 3 is a non-minimum phase channel, with one zero in the unit circle and one outside the unit circle, and the amplitudes are 1.43/0.9, respectively. The zero with an amplitude of 1.43 is symmetric with the zero with an amplitude of 0.7 about the unit circle in channel 2. The zero distribution of the three channels is shown in Figure 9.

The simulation error performance curves under the above three channels are illustrated in Figure 10. It can be seen from Figure 10 that the newly proposed MLSE optimum receiver with channel identification based on zero distribution can achieve similar performance to the conventional MLSE optimum receiver in both the minimum phase channel and the non-minimum phase channel. Still, its complexity is low, and the correlation function does not need to be calculated. It only needs to realize channel identification based on the zero distribution of the channel and construct the whitening filter.

By comparing Figure 10a,b, it can be seen that the system performance is worse in channel 2, where the zeros are closer to the unit circle. Therefore, the system performance is related to the zeros position, and the closer the zeros are to the unit circle, the worse the performance is. By comparing Figure 10b,c, it can be seen that when the zeros position of the two channels are the same or symmetric about the unit circle, the performance of the two channels is the same. From a computational point of view, the number of multiplications reduced by the proposed method is $N(2N+1)$. In terms of processing time, the proposed method decreases the time required to recover a frame signal by 0.018 s in Matlab 2019b.

5. Conclusions

In this paper, a novel MLSE optimum receiver design is suggested to reduce the complexity of the conventional optimum receiver. We first propose a multipath channel classification method according to the zero distribution of the channel impulse response; that is, the multipath channel is divided into a minimum phase channel and a non-minimum phase channel. The simulation results indicate that this channel classification method effectively represents the characteristics of the channel. Then, an MLSE optimum receiver design with channel identification based on zero distribution is also proposed. In this design, the distribution of zero can be obtained directly by calculating the Z transform of the channel. Thus, the complexity is reduced. The number of the multiplication reduced is $N(2N+1)$. In addition, $2N+1$ memory units storing the autocorrelation sequence are reduced. The simulation results show that the MLSE optimum receiver proposed in this paper performs similarly and is less complex than the conventional one. In terms of processing time, the proposed method decreases the time required to recover a frame signal by 0.018 s in Matlab. Compared with the MMSE equalizer, the performance is greatly improved. This design can be used in various scenarios, such as UAV communication, optical communication and ultraviolet communication. As with all other receivers, the performance of the proposed optimum receiver is related to the accuracy of the channel estimation. Especially in deep fading scenarios, channel estimation is a technical challenge.