An Overview of FIR Filter Design in Future Multicarrier Communication Systems

Abstract

Future wireless communication systems are facing with many challenges due to their complexity and diversification. Orthogonal frequency division multiplexing (OFDM) in 4G cannot meet the requirements in future scenarios, thus alternative multicarrier modulation (MCM) candidates for future physical layer have been extensively studied in the academic field, for example, filter bank multicarrier (FBMC), generalized frequency division multiplexing (GFDM), universal filtered multicarrier (UFMC), filtered OFDM (F-OFDM), and so forth, wherein the prototype filter design is an essential component based on which the synthesis and analysis filters are derived. This paper presents a comprehensive survey on the recent advances of finite impulse response (FIR) filter design methods in MCM based communication systems. Firstly, the fundamental aspects are examined, including the introduction of existing waveform candidates and the principle of FIR filter design. Then the methods of FIR filter design are summarized in details and we focus on the following three categories—frequency sampling methods, windowing based methods and optimization based methods. Finally, the performances of various FIR design methods are evaluated and quantified by power spectral density (PSD) and bit error rate (BER), and different MCM schemes as well as their potential prototype filters are discussed.

1. Introduction

The rapid increase of mobile devices and the emergence of new technologies as well as services demand more efficient wireless cellular networks [1,2,3,4]. The 5th generation (5G) communication networks need to support abundant business scenarios, such as Internet of Vehicles [5,6,7], Internet of Things [8,9,10], virtual reality [11,12], device-to-device communications [13,14,15,16], and so forth. Therefore, plenty of important technologies are worth studying in future communication networks, for example, multicarrier modulation, massive multiple-input multiple-output (MIMO), millimeter wave, and so forth [1,2,3,4]. Due to the critical role of the prototype filter played in multicarrier modulations, that is, it determines the system performance such as stopband attenuation, inter-symbol interference (ISI), inter-channel interference (ICI) and phase noise caused by high operating frequencies, herein various prototype filter design methods in multicarrier communication networks are summarized.

It is known that cyclic-prefix orthogonal frequency division multiplexing (CP-OFDM) as the air interface of the 4th generation (4G) communication networks is capable of avoiding ICI and ISI. The modulated and demodulated signals are respectively generated through inverse fast Fourier transform (IFFT) and fast Fourier transform (FFT), thereby greatly reducing the system complexity and improving the signal transmission rate. OFDM has been used in conjunction with other technologies, for example, wavelet OFDM (WOFDM) [17,18], orthogonal frequency division multiple access (OFDMA) [19,20,21], Alamouti coded OFDM (AC-OFDM) [22,23,24,25], and MIMO-OFDM [26,27,28]. Although OFDM has been widely applied, it also has the following limitations: serious out-of-band leakage characteristic; strict synchronization and orthogonality among the subcarriers are needed; high peak to average power ratio (PAPR) resulting in non-linear distortion of the signal; sensitive to frequency offset, which has a significant impact on the system performance [29,30].

The above situations drive the urgency to conceive an appropriate modulation scheme in future communication networks. Faced with the requirements of the ultra-low latency, high spectrum efficiency, high transmission rate and business diversity in the future networks, researchers and practitioners in related fields have exerted a tremendous fascination on a number of alternative single-carrier or multicarrier modulation techniques [31], including CP-Discrete Fourier Transform spread OFDM (CP-DFT-s-OFDM) [32], filter bank multicarrier (FBMC) [33,34,35,36,37,38,39], generalized frequency division multiplexing (GFDM) [40,41], universal filtered multicarrier (UFMC) [42,43,44], and filtered-orthogonal frequency division multiplexing (F-OFDM) [45,46,47], and so forth. Apart from typical waveform design methods where waveform parameters are obtained manually, with the development of machine learning, waveform design methods with data-driven models have attracted increasing attention [48,49]. These modulation waveforms have their own merits and drawbacks in 5G communication scenarios, respectively. The prototype filter determines the performance of a specific modulation waveform, thus the evaluation standard of prototype filter is primarily characterized by minimizing the stopband energy, minimizing the maximum stopband ripple, minimizing the total interference (ISI and ICI) [50,51], and so forth. Digital filters are very important in digital communications and are generally divided into two categories—infinite impulse response (IIR) filter and finite impulse response (FIR) filter [52]. Based on the fact that only linear phase FIR filters are suitable for wireless communication systems [53], the FIR filter design has been the major research topic to realize prototype filters in the literature.

There have already been a few representative survey works related to prototype filters. In Reference [4], a survey of multicarrier communications about prototype filters, lattice structures and the implementation aspects is reported by A. Şahin et al., and this work provides four classes of filters according to the design criteria—energy concentration, rapid decay, spectrum nulling and channel/hardware characteristics. In Reference [54], the methods of prototype filter design for cosine modulated filter banks (CMFBs) with nearly perfect reconstruction (NPR) are reviewed by K. Shaeen et al., and these methods are categorized into nonlinear optimization methods [55,56,57,58,59,60,61,62,63], spectral factorization methods [64], linear search methods [65,66,67,68,69,70], interpolated finite impulse response (IFIR) methods [71,72,73], and frequency response masking (FRM) methods [58,74,75,76,77,78,79], and so forth. The authors of Reference [80] provide the basic introduction of the FIR filter, some relevant works on FIR and various factors which effect the performance of FIR filter in communication systems.

In this paper, we pay more attention to the literature from more recent years and provide a survey on various FIR filter design methods in multicarrier systems. Compared with the existing surveys in References [4,54,80], this paper provides an in-depth introduction to advanced filter design methods, which are not categorized based on the design criteria. In addition, this work can serve as an extension of the survey in Reference [54], which is dedicated to the aspect of CMFBs. Specifically, we firstly introduce the FIR filter design criteria, that is, the objective to be optimized. The evaluation criteria and the implementation methods are summarized in canonical form. Subsequently, the design methods of realizing the objectives are divided into three categories: frequency sampling methods, windowing based methods and optimization based methods. Lastly, the reviewed filter design methods are analyzed for multicarrier modulation schemes in terms of power spectral density (PSD) [81,82], bit error rate (BER) [83,84,85], spectral efficiency, latency, computational complexity, and so forth, which are important measurements to characterize the performance of modulation waveforms. We expect that this survey can provide a reference on how to select multicarrier modulation schemes and their corresponding prototype filters in future wireless communication systems.

The remainder of this overview paper is organized as follows. Section 2 introduces the concepts of different multicarrier modulations and the evaluation criteria of FIR filter design. In Section 3, the classification of FIR prototype filter design methods and corresponding implementation procedures are described in details. The discussions of different filter design methods in multicarrier modulation systems in terms of PSD and BER performances are presented in Section 4. Finally, Section 5 concludes the paper.

2. Multicarrier Modulations and FIR Filter

By splitting the transmitting data into several components and sending each of these components over separate carrier signals, multicarrier modulation (MCM) has numerous advantages compared to single carrier modulation, including the relative immunity to multipath fading, less susceptibility to interference caused by impulse noise, and enhanced immunity to ISI. In the following subsection, five important MCM waveforms—OFDM, FBMC, GFDM, UFMC and F-OFDM are introduced.

2.1. Modulation Waveforms

2.1.1. OFDM

Compared with conventional OFDM, windowed orthogonal frequency division multiplexing (W-OFDM) has a similar transceiver structure but non-rectangular transmit windows are utilized to smooth the edges of rectangular pulse, accordingly provide better spectral localization and reduce ICI [86]. For a raised cosine shape based non-rectangular window used in W-OFDM, the CP needs to be extended to maintain the orthogonality and the spectral efficiency will decrease for W-OFDM compared to OFDM [87]. In order to utilize non-contiguous spectrum fragments, resource block filtered OFDM (RB-F-OFDM) aims to split the available spectrum fragments into several resource blocks which make a chunk of some contiguous subcarriers. It generates and filters the signal transmitted on each resource block individually [88]. Analogous to the subcarrier filtering based modulation, spectrum leakage among these resource blocks is unavoidable in the case that the spacing between these blocks is narrow [89]. In CP-OFDM with weighted overlap and add (WOLA), some data parts are copied and added to the right and left part of conventional OFDM, then a pulse with soft edges takes place in the rectangular prototype filter, in the meanwhile, the soft edges are added to the cyclic extension by a time domain windowing and this results in a sharper side-lobe decay in frequency domain [90]. Therefore, CP-OFDM with WOLA is a special case of OFDM, aiming to improve the prototype filter such that it has more reasonable pulse shape used in regular CP-OFDM. Furthermore, for the sake of addressing the sacrifice of time resource resulting from added parts and avoiding possible data collision before transmission due to windowing process, overlap process is employed [91].

2.1.2. FBMC

The FBMC modulation technique, as one of the waveform candidates in 5G communication networks, has attracted a great amount of research attention. In the 1960s, the concept of FBMC modulation was firstly proposed by Chang and Saltherg [92]. However, it did not receive much attention by researchers due to its complexity. The well-known discrete multi-tone (DMT) modulation and discrete wavelet multi-tone (DWMT) modulation reported in the 1990s are two specific cases of FBMC modulation [93]. Currently, FBMC based systems have been extensively studied from the aspects of spectrum efficiency analysis [94,95,96], system complexity analysis [97], prototype filter design [98,99,100,101], frequency offset estimation [102,103], MIMO [104,105,106], and so forth. Ever-emerging research projects investigate the application of FBMC modulation in practical scenarios, for example, PHYDYAS [107], METIS [108], 5GNOW [109]. FBMC is robust against frequency offset because of its negligible spectrum leakage. Furthermore, FBMC does not need the guard band in frequency domain, which greatly improves the spectrum efficiency, thus FBMC can flexibly control the interference between adjacent subcarriers, and the synchronization requirement among subcarriers is relaxed [36]. Simultaneously, there are still some underlying issues of FBMC, for instance, high PAPR, which may well result in signal nonlinearities. Though many existing methods have been dedicated to lowing PAPR of FBMC [110,111,112,113], it is actually a trade-off between low PAPR and other characteristics, that is, in Reference [110], the PAPR reduction is at the expense of higher computation complexity. In spite of this, advantages above still make FBMC popular in the academic field, and FBMC has already been considered as the waveform candidate in future MCM networks [33].

2.1.3. GFDM

The GFDM modulation waveform, as a new 5G multicarrier modulation technique, was proposed by Fettweis et al. in 2009 [41]. Compared to OFDM, GFDM aiming to generalize traditional OFDM is based on separate block modulation, which makes it more flexible by configuring different subcarriers and symbols. To accommodate various types of services, GFDM can work with different prototype filters and different types of CPs. The design of the conforming prototype filter reduces the sidelobe interference, and the modulation process is converted from linear convolution to cyclic convolution through tail biting operation, thus shortening the length of CP. Similar to FBMC, GFDM is also robust against time-frequency offset but has a low PAPR. However, each subcarrier in GFDM modulation does not keep the orthogonality in the frequency domain and ICI is introduced even under ideal channel, which increases the complexity of the receiver algorithm and meanwhile raises BER.

2.1.4. UFMC

UFMC modulation technique was proposed by Vida Vakilian et al. to solve the ICI problem in OFDM systems [42]. UFMC modulation enjoys the following advantages of relaxing the requirement of CP, having high spectrum efficiency, relaxing carrier synchronization, and being suitable for fragmented debris spectrum utilization. In addition, UFMC supports short burst asynchronous communication because the filter lengths depend on the sub-band widths. Nevertheless, similar to OFDM, the performance of UFMC is also inevitably affected by the carrier frequency offset (CFO), which has attracted a lot of research interest in order to mitigate the significant impact of CFO on UFMC based system performance [114,115,116].

2.1.5. F-OFDM

The basic principle of F-OFDM technique is to divide the carrier bandwidth of OFDM into sub-bands with different parameters, filter the sub-bands, and leave less isolation bands in the sub-bands [46]. Firstly, to support diverse businesses, F-OFDM modulation supports flexible sub-band configurations for different subcarrier spacing. Secondly, it supports different sub-band configurations allowing different CP lengths to better adapt transmission channels, and different sub-bands on asynchronous signal transmission are also supported, thus saving the signaling overhead. Finally, with better out of band suppression characteristics compared to OFDM, it saves the cost of protection zone, that is, the spectral efficiency is improved. However, filters need to be dynamically designed for each fragment, which makes the use of F-OFDM systems challenging [88].

2.2. FIR Filter

In this subsection, we introduce the basic principle of digital FIR filters as well as the evaluation criteria of a designed filter. It is known that the condition of distortionless transmission and filtering is that the amplitude response of the system should be constant in the effective spectral range of the signal, and the phase response should be a linear function of the frequency (i.e., linear phase). The prototype filter is required to have linear phase characteristic in wireless communication applications. In contrast to the IIR filter, the FIR filter has the ability to achieve linear phase filtering [117]. In addition, as an all-zero filter, the hardware and software structures of the FIR filter could be established without considering the stability problem. Therefore, FIR filters have been widely used in the field of wireless communications [53].

Assuming the length of the impulse response $h\left(n\right)$ of a linear phase FIR filter is N, then the frequency response function is defined as

$$H\left(e^{j\omega }\right)=\sum _{n=0}^{N-1}h\left(n\right)e^{-j\omega n},$$

(1)

where $\omega$ is the angular frequency. According to the parity of N and the symmetry of $h\left(n\right)$, there are four types of FIR transfer functions. The condition for the most widely used low-pass FIR filter is given by

$$h\left(n\right)=h(N-1-n),0\le n\le N-1.$$

(2)

The ripples are observed in both the passband and stopband, and the maximum approximation error ${\delta }_p$ is described by

$$1-{\delta }_p\le \mid H\left(e^{j\omega }\right)\mid \le 1,\mid \omega \mid \le {\omega }_p.$$

(3)

The amplitude of the stopband is approximated by the maximum error ${\delta }_s$, that is,

$$\mid H\left(e^{j\omega }\right)\mid \le {\delta }_s,{\omega }_s\le \mid \omega \mid \le \pi ,$$

(4)

where ${\omega }_p$ and ${\omega }_s$ are passband and stopband boundary frequencies, ${\delta }_p$ and ${\delta }_s$ are passband and stopband ripples, respectively (Gibbs effect), and ${\omega }_s-{\omega }_p$ is the transition bandwidth. In order to make designed filters close to the ideal low-pass filter, several evaluation criteria are given in [50,51], including least-squares (LS), minimax, peak-constrained least-squares (PCLS), minimum total interference (ICI and ISI), and and so forth.

• LS criterion
  The goal of the least-squares criterion is to minimize the stopband energy of the filter, whose objective function is

  $$J={\int }_{{\omega }_s}^{\pi }\mid H\left(e^{j\omega }\right){\mid }^{2}d\omega .$$

  (5)
• Minimax criterion
  The goal of the minimax criterion is to minimize the maximum stopband ripple, and its objective function can be written as

  $$J=\underset{\omega \in [{\omega }_s,\pi ]}{max}\mid H\left(e^{j\omega }\right)\mid .$$

  (6)
• PCLS criterion
  The PCLS criterion establishes a trade-off between the LS and the minimax criteria. The PCLS criterion can be described as below

  $$\begin{array}{c}J={\int }_{{\omega }_s}^{\pi }\mid H\left(e^{j\omega }\right){\mid }^{2}d\omega \hfill \\ \mathrm{s}.\mathrm{t}.\mid H\left(j{e}^{j\omega }\right)\mid \le \delta ,\hfill \end{array}$$

  (7)

  where $\delta$ is a prescribed value. If $\delta$ is close to zero, the PCLS criterion approaches the minimax criterion, while in the limit of $\delta$, that is, $\delta$ is up to infinite, the criterion turns out the LS criterion.
• Minimum total interference criterion
  This criterion is to minimize the total interference of ICI and ISI for filter bank structure. Its objective function is defined as

  $$J=\mathrm{ISI}+\mathrm{ICI},$$

  (8)

  where

  $$\mathrm{ISI}=\underset{k}{max}\left(\sum _n{({\left[T_{\mathrm{TMUX}}\left(n\right)\right]}_{k,k}-\delta (n-▵))}^2\right),$$

  (9)

  $$\mathrm{ICI}=\underset{k}{max}\left(\sum _{l=0,l\ne k}^{N-1}\sum _n{\left({\left[T_{\mathrm{TMUX}}\left(n\right)\right]}_{k,l}\right)}^2\right),$$

  (10)

  where $T_{\mathrm{TMUX}}\left(n\right)$ is the transfer matrix, the element ${\left[T_{\mathrm{TMUX}}\left(n\right)\right]}_{a,b}$ represents the relationship between the input signal $X\left(z^N\right)$ and the output signal $Y\left(z^N\right)$, where z is the complex argument of z-transform. $\delta \left(n\right)$ is the ideal impulse, and △ denotes the delay of the TMUX system. Thus the transfer function between input and output signals is given by

  $$Y\left(z^N\right)=T_{\mathrm{TMUX}}\left(z^N\right)·X\left(z^N\right),$$

  (11)

  where

  $$X\left(z^N\right)={[X_0\left(z\right)X_1\left(z\right)\cdots X_{N-1}\left(z\right)]}^T,$$

  (12)

  $$Y\left(z^N\right)={[Y_0\left(z\right)Y_1\left(z\right)\cdots Y_{N-1}\left(z\right)]}^T.$$

  (13)

3. FIR Prototype Filter Design

As discussed in Section 2, there are some design criteria to customize the performance of FIR. Thus, there must be some specific methods of FIR design corresponding to these criteria. In this work, the filter design methods are divided into three major categories: the frequency sampling methods, the windowing based methods, and the optimization based methods. For each type of method, we firstly introduce their basic concepts and subsequently some typical design examples are given. At last the advantages and disadvantages of the summarized methods are discussed.

3.1. Frequency Sampling Methods

The idea of frequency sampling is conceived from the frequency domain perspective. This kind of method takes uniform spacing sampling of an ideal frequency response $H_d\left(e^{j\omega }\right)$, which is given by

$$H_d\left(k\right)=H_d\left(e^{j\omega }\right){|}_{\omega =\frac{2\pi }{N}k}.$$

(14)

Then $H_d\left(k\right)$ is used as the sampled values of the actual linear phase FIR filter, written as

$$H\left(k\right)=H_d\left(k\right),k=0,1,\cdots ,N-1.$$

(15)

The N-point inverse discrete Fourier transform (IDFT) for $H\left(k\right)$ yields the following impulse response

$$\begin{array}{cc}\hfill h\left(n\right)=& \mathrm{IDFT}\left[H\left(k\right)\right]\hfill \\ \hfill =& \frac{1}{N}\sum _{k=0}^{N-1}H\left(k\right)W_N^{-kn},n=0,1,\cdots ,N-1,\hfill \end{array}$$

(16)

where $W_N=e^{-j\frac{2\pi }{N}}$.

The exponential form of $H\left(k\right)$ is given by

$$H\left(k\right)=A\left(k\right)e^{j\theta \left(k\right)},$$

(17)

where $A\left(k\right)$ and $\theta \left(k\right)$ are amplitude sampling and phase sampling, respectively. To design a linear phase FIR filter in practical applications, the condition in Equation (2) should be satisfied. Thus certain constraint conditions are imposed on $A\left(k\right)$

$$\left\{\begin{array}{cc}A\left(k\right)=A(N-k),\hfill & \hfill N\mathrm{is}\mathrm{odd},\\ A\left(k\right)=-A(N-k),\hfill & \hfill N\mathrm{is}\mathrm{even},\end{array}\right.$$

(18)

and also on $\theta \left(k\right)$

$$\theta \left(k\right)=-\omega \left(\frac{N-1}{2}\right){|}_{\omega =\frac{2\pi k}{N}}=-\left(\frac{N-1}{N}\right)\pi k.$$

(19)

A series of frequency sampling based filter design methods have been reported according to the above formulations. Some of commonly used methods are described in the following subsections.

3.1.1. Bellanger’s Method

In [118], Bellanger et al. use a typical frequency sampling method to design a filter, which is chosen as the prototype filter of FBMC modulation system in PHYDYAS project. Specific implementation is as follows [119]: assuming an integer K, the number of subcarriers is F and the number of samples is $KF$, two conditions are proposed:

Condition 1: To approximately meet the Nyquist criterion, the following equation should be satisfied

$$\left\{\begin{array}{c}{H}_0=1,\hfill \\ H_k^2+H_{K-k}^2=1,\hfill \\ H_{KF-k}=H_k,\hfill & \hfill 1\le k\le K-1,\\ H_k=0,\hfill & \hfill K\le k\le KF-K,\end{array}\right.$$

(20)

where $H_k(0\le k\le KF-1)$ is the $k^{th}$ frequency weighting coefficient.

Condition 2: To ensure stopband performance, the following equation is satisfied

$$H_0+2\sum _{k=1}^{K-1}{(-1)}^k{H}_k=0.$$

(21)

When $K=3$ and 4, the filter weighting coefficients are

$$\left\{\begin{array}{c}K=3:H_0=1;H_1=0.9144;H_2=0.4114,\hfill \\ K=4:H_0=1;H_1=0.9720;H_2=\sqrt{2};H_3=0.2351.\hfill \end{array}\right.$$

(22)

Then the impulse response is obtained by IDFT

$$h\left(t\right)=\left\{\begin{array}{cc}\hfill 1+2\sum _{k=1}^{K-1}{(-1)}^k{H}_{k}cos\left(\frac{2\pi t}{KT}\right),& -\frac{KT}{2}\le t\le \frac{KT}{2},\hfill \\ \hfill 0,& \mathrm{otherwise}.\hfill \end{array}\right.$$

(23)

3.1.2. Viholainen’s Method

Similar to the design in [118], Condition 1 is also considered in Viholainen’s method [51], but Condition 2 is relaxed and it is assumed that $H_1=\chi$. When $K=3$ and $K=4$, the filter weighting coefficients are

$$\left\{\begin{array}{c}K=3:H_0=1;H_1=\chi ;H_2=\sqrt{1-{\chi }^2},\hfill \\ K=4:H_0=1;H_1=\chi ;H_2=1/\sqrt{2};H_3=\sqrt{1-{\chi }^2}.\hfill \end{array}\right.$$

(24)

According to a specific objective function, the optimal solution can be obtained by a simple global search algorithm.

In [51], the LS criterion and the minimax criterion are selected for comparison with the method in [118]. The frequency responses of prototype filters in [51,118] based on the LS criterion are shown in Figure 1a, where the number of subcarriers is 16. As can be seen, the side-lobes of the prototype filters when $K=4$ are lower than those with $K=3$, that is, better stopband performance is benefited by a larger value of K. Furthermore, the first side-lobe of the prototype filter in [51] is a little lower than that in [118]. Figure 1b shows the frequency responses of prototype filters in [51,118] based on the minimax criterion. Similarly, the prototype filters in [118] have higher first side-lobe levels. It is also observed from Figure 1, the prototype filters based on the minimax criterion improve the stopband performance at the cost of sacrificing the main-lobe width.

3.1.3. Cruz-Roldán’s Method I

A frequency sampling method for arbitrary length FIR filters is proposed in [120]. Define the frequency response of the prototype filter $H\left(e^{j\omega }\right)$, let $H\left[k\right]=H\left(e^{j{\omega }_k}\right)$, where ${\omega }_k=(k+\alpha )·2\pi /N$, $0\le k\le (N-1)$ and $\alpha =0$ or $1/2$. The authors in [120] assume $h\left(n\right)$ is real and symmetric, thus the filter coefficients are obtained according to [117] when $\alpha =0$

$$h\left[n\right]=\frac{1}{N}\left\{P\left[0\right]+2\sum _{k=1}^{\lfloor N/2\rfloor -1}P\left[k\right]cos\left((n+1/2)·\frac{2\pi k}{N}\right)\right\},$$

(25)

where $P\left[k\right]=H\left[k\right]·e^{-jk\pi /N}$, $\lfloor \mathrm{M}\rfloor$ denotes the largest integer less than $\mathrm{M}$. While if $\alpha =1/2$, the filter is expressed as

$$h\left[n\right]=\frac{2}{N}\sum _{k=0}^{\lfloor N/2\rfloor -1}P\left[k\right]sin\left((n+1/2)·\frac{2\pi }{N}·(k+1/2)\right),$$

(26)

where $P\left[k\right]=H\left[k\right]·e^{-j(N-2k-1)\pi /\left(2N\right)}$.

Initial values of $H\left[k\right]$ are determined before optimization, and the center of the transition band is ${\omega }_r=(r+\alpha )·2\pi /N,r\in Z^{+}$. In the transition band, the number of samples $L(L>1)$ should be defined. Thus the magnitude response is defined as

$$\mid H\left[k\right]\mid =\left\{\begin{array}{cc}1,\hfill & \hfill 0\le k\le r-\lceil L/2\rceil ,\\ \hfill & \hfill \left(\mathrm{passband}\right)\\ f\left(k\right),\hfill & \hfill r-\lceil L/2\rceil +1\le k\le r+\lfloor L/2\rfloor ,\\ \hfill & \hfill \left(\mathrm{transition}\mathrm{band}\right)\\ 0,\hfill & \hfill r+\lfloor L/2\rfloor +1\le k\le \lfloor (N-1)/2\rfloor ,\\ \hfill & \hfill \left(\mathrm{stopband}\right)\end{array}\right.$$

(27a)

$$\mid H[N-1-k]\mid =\mid H\left[k\right]\mid ,\lfloor (N-1)/2\rfloor +1\le k\le N-1,$$

(27b)

where $\lceil \mathrm{M}\rceil$ denotes the smallest integer more than $\mathrm{M}$. And the phase response $arg\left\{H\right[k\left]\right\}$ is defined as

$$arg\left\{H\left[k\right]\right\}=\left\{\begin{array}{c}-\frac{N-1}{2}·\frac{2\pi }{N}·(k+\alpha ),\hfill \\ 0\le k\le \lfloor (N-1)/2\rfloor ,\hfill \\ \frac{N-1}{2}·\frac{2\pi }{N}·\left(N-(k+\alpha )\right),\hfill \\ \lfloor (N-1)/2\rfloor +1\le k\le N-1.\hfill \end{array}\right.$$

(28)

The function $f\left(k\right)$ in (27a) is to obtain the magnitude values of the transition band samples. In [121], $f\left(k\right)$ is chosen as

$$f\left(k\right)=\frac{{\omega }_s-k·2\pi /N}{{\omega }_s-{\omega }_p}.$$

(29)

While in [120] $f\left(k\right)$ has a different expression as below

$$f\left(k\right)=0.95-{\left(\frac{{\omega }_s-(L+1-k)·2\pi /N}{{\omega }_s-{\omega }_p}\right)}^2.$$

(30)

When the initial values are determined, the specific steps of the optimization procedure are as follows:

(a) 

Initialize the filter length N and the required number of samples L in the transition band.

(b) 

Initialize the frequency response in (27) and (28). The resulting vector $\left|\mathbf{H}\left[k\right]\right|$ is presented as follows

$$\begin{array}{c}\hfill |{\mathbf{H}}_{opt}\left[k\right]|=\left[\underset{\mathrm{passband}}{\underbrace{1...1}}\underset{\mathrm{transition}}{\underbrace{f[q+1]...f[q+L]}}\underset{\mathrm{stopband}}{\underbrace{0...0}}\right],\end{array}$$

(31)

where $0\le k\le \lfloor (N-1)/2\rfloor ,q=r-\lceil L/2\rceil$.

(c) 

Let $\mathbf{f}=\left[f[q+1]f[q+2]\cdots f[q+L]\right]$ be the vector whose elements are the samples of the magnitude response at the transition band. Find ${\mathbf{f}}_{opt}$ and minimize an objective function $\psi$ defined as [122]

$$\psi =\underset{n,n\ne 0}{max}\mid g\left[2Mn\right]\mid ,$$

(32)

where M is related to the number of channels, $G\left(e^{j\omega }\right)$ is the DFT of $g\left[n\right]$ and defined as $G\left(e^{j\omega }\right)=\mid H\left(e^{j\omega }\right){\mid }^2$.

(d) 

Calculate the optimum values of the frequency response samples ${\mathbf{H}}_{opt}\left[k\right]$. These values are obtained from (27) and (28), by replacing the initial values of the magnitude response in the transition band in (27a) by the optimised values ${\mathbf{f}}_{opt}$ obtained in the previous step

$$\begin{array}{c}\hfill |{\mathbf{H}}_{opt}\left[k\right]|=\left[\underset{\mathrm{passband}}{\underbrace{1...1}}\underset{\mathrm{transition}}{\underbrace{f_{opt}[q+1]...f_{opt}[q+L]}}\underset{\mathrm{stopband}}{\underbrace{0...0}}\right],\end{array}$$

(33)

$$\begin{array}{cc}\hfill |{\mathbf{H}}_{opt}[N-& 1-k]|=|{\mathbf{H}}_{opt}\left[k\right]|,\hfill \\ \hfill & \lfloor (N-1)/2\rfloor +1\le k\le N-1.\hfill \end{array}$$

(34)

(e) 

Based on ${\mathbf{H}}_{opt}\left[k\right]$, the prototype filter coefficients are obtained through (25) or (26).

In [65], the objective function $\psi$ in step 3 is defined as

$$\psi =\underset{\omega }{max}\left\{|H\left(e^{j\omega }\right){|}^2+{|H\left(e^{j(\omega -\pi /M)}\right)|}^2-1\right\}.$$

(35)

The minimization algorithms in MATLAB Optimization Toolbox [120] can be used to solve this optimization problem.

3.1.4. Cruz-Roldán’s Method II

In [123], a multi-objective optimization technique based on the Cruz-Roldán’s method I in [120] is proposed. The difference between [120,123] lies in the objective function, and the objective function of Cruz-Roldán’s method II is

$$\begin{array}{cc}\hfill & \psi =\frac{1}{2\pi }{\int }_{{\omega }_s}^{2\pi -{\omega }_s}{|H\left(e^{j\omega }\right)|}^{2}d\omega \hfill \\ \hfill & ={\mathbf{hSh}}^{\mathrm{T}}\hfill \\ \hfill & =\frac{1}{N^2}{\mathbf{H}}^{\mathrm{T}}{\left({\mathbf{W}}_N^{-1}\right)}^{\mathrm{T}}\mathbf{S}{\mathbf{W}}_N^{-1}\mathbf{H},\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\left\{\begin{array}{c}\underset{\omega \in [0,\pi ]}{max}(|T_0\left(e^{j\omega }\right)|)-\underset{\omega \in [0,\pi ]}{min}(|T_0\left(e^{j\omega }\right)|)\le {\delta }_{pp}^{ini},\hfill \\ MSA\ge MS{A}^{ini},\hfill \end{array}\right.\hfill \end{array}$$

(36)

where $\mathbf{h}=\left[h\right(0),h(1),\cdots h(N-1\left)\right]$, ${\mathbf{h}}^{\mathrm{T}}=(1/N){\mathbf{W}}_N^{-1}\mathbf{H}$, ${\mathbf{W}}_N^{-1}$ is the IDFT matrix, and the elements of $N\times N$ matrix $\mathbf{S}$ are given by

$$\left[{\mathbf{S}}_{i,j}\right]=\left\{\begin{array}{cc}1-\frac{{\omega }_s}{\pi },\hfill & \hfill i=j,\\ -\frac{sin\left[{\omega }_s(i-j)\right]}{\pi (i-j)},\hfill & \hfill \mathrm{otherwise}.\end{array}\right.$$

(37)

The values of ${\delta }_{pp}^{ini}$ (amplitude distortion [121]) and $MS{A}^{ini}$ (minimum stopband attenuation) are the initially fixed goals of the problem. The overall distortion transfer function $T_0\left(e^{j\omega }\right)$ is obtained as

$$T_0\left(e^{j\omega }\right)=\frac{e^{-j\omega (N-1)}}{F}\sum _{k=0}^{2F-1}{|H\left(e^{j(\omega -k·\pi /F)}\right)|}^2,$$

(38)

where F is the number of subcarriers. This method requires the values of ${\delta }_{pp}^{ini}$ and $MS{A}^{ini}$ to be selected appropriately, and the MATLAB function fgoalattain can solve the multi-objective optimization problem.

3.1.5. Salcedo-Sanz’s Method

In [124], the authors propose the variable limits evolutionary programming (VLEP) based on the evolutionary programming (EP) to design the prototype filter, including variable limits classical evolutionary programming (VLCEP) and variable limits fast evolutionary programming (VLFEP). The final optimum solutions are sensitive to the initial values of $f\left(k\right)$ in [120], while the algorithm in [124] can solve this problem.

Three design methods in [120,123,124] are analyzed in terms of amplitude distortion peak, aliasing error and minimum stopband attenuation, as presented in

Table 1. Analysis of prototype filter design in [120,123,124] by amplitude distortion peak, aliasing error, minimum stopband attenuation.

Prototype Filter Design	Amplitude Distortion Peak	Aliasing Error	Minimum Stopband Attenuation
Ref. [120]	$7.0449\times {10}^{-4}$	−139.48 dB	78 dB
Ref. [123]	$3.5001\times {10}^{-5}$	−151.39 dB	108 dB
Ref. [124]	$1.2848\times {10}^{-4}$	−93.27 dB	69 dB

. The number of subcarriers is 128 and the length of the prototype filter is 2049. We select the VLFEP algorithm in [124]. Generally, all of the three methods can obtain nearly perfect reconstruction (NPR) filter banks. By comparison, the method in [123] achieves the best filter design performance. To summarize, the essence, pros and cons of aforementioned frequency sampling methods are presented in

Table 2. Summary of frequency sampling methods.

Methods	Comments	Pros and Cons
Bellanger’s method [118]	Design the filter parameter under the constraints of controlling stopband performance and satisfying Nyquist criterion.	Pros: Favorable stopband attenuation. Cons: Long filter length (Long latency).
Viholainen’s method [51]	Optimize the filter parameters according to different evaluation criteria under the condition of Nyquist criterion.	Pros: Good stopband attenuation; Flexibly select suitable filter parameters according to different evaluation criteria. Cons: Need long filter length.
Cruz-Roldán’s method I [120]	An optimization scheme based on frequency sampling is proposed to obtain the filter parameters.	Pros: Low computational complexity; Design a filter with arbitrary length. Cons: Difficult to choose the sampling values of transition band for fast convergence.
Cruz-Roldán’s method II [123]	Based on the optimization algorithm proposed in [120], the objective function is improved to achieve multi-objective optimization.	Pros: The stopband energy and stopband attenuation are minimized simultaneously. Cons: Need to appropriately select initial parameters to ensure the performance.
Salcedo-Sanz’s method [124]	The VLEP algorithm [124] is proposed based on the classical and fast evolutionary programming algorithm.	Pros: Robust to the initial conditions; The minimum of the objective function can be reliably obtained. Cons: High computational complexity.

.

3.2. Windowing Based Methods

The idea of windowing based methods is to use the digital FIR filter to approximate the desired filtering characteristics. Assuming that the desired filter frequency response function is $H_d\left(e^{j\omega }\right)$, the impulse response is denoted as $h_d\left(n\right)$. Considering the windowing based methods focus on the perspective of time domain, the ideal $h_d\left(n\right)$ with a certain shape of window function is intercepted as $h\left(n\right)$ with finite length, whose frequency response $H\left(e^{j\omega }\right)$ approximates the desired frequency response $H_d\left(e^{j\omega }\right)$. The specific design steps of windowing based methods are as follows:

step 1:

Taking linear phase low-pass FIR filter as an example, the general selection of $H_d\left(e^{j\omega }\right)$ is

$$H_d\left(e^{j\omega }\right)=\left\{\begin{array}{cc}{e}^{-j\omega \tau },\hfill & \hfill |\omega |\le {\omega }_c,\\ 0,\hfill & \hfill {\omega }_c\le |\omega |\le \pi ,\end{array}\right.$$

(39)

where $\tau$ is a constant.

step 2:

Determine $h_d\left(n\right)$ via IDFT

$$\begin{array}{c}\hfill h_d\left(n\right)=\frac{1}{2\pi }{\int }_{-{\omega }_c}^{{\omega }_c}{H}_d\left(e^{j\omega }\right)e^{j\omega n}d\omega =\frac{sin\left[{\omega }_c(n-\tau )\right]}{\pi (n-\tau )}.\end{array}$$

(40)

step 3:

The impulse response $h\left(n\right)$ of the linear phase FIR filter is obtained by multiplying a specific window $w\left(n\right)$, as below

$$h\left(n\right)=h_d\left(n\right)w\left(n\right).$$

(41)

Thus the corresponding $H\left(e^{j\omega }\right)$ is obtained by DFT.

3.2.1. Jain’s Method

Like the concept of Bartlett-Hanning window, a new window consisting of a Hamming window and a Blackman is proposed in [125], which is given by

$$\begin{array}{cc}\hfill w\left(n\right)=& \lambda \left(0.54-0.46cos\frac{2\pi n}{N-1}\right)\hfill \\ \hfill & +(1-\lambda )\left(0.42-0.5cos\frac{2\pi n}{N-1}+0.8cos\frac{4\pi n}{N-1}\right),\hfill \end{array}$$

(42)

where $0\le |n|\le (N-1)/2$.

Note that the formula in (42) becomes a Hamming window if $\lambda =1$ and a Blackman window if $\lambda =0$. Figure 2 shows the frequency responses of Hamming window, Blackman window and this new window, where the variable $\lambda$ is evaluated as $0.0625$ to ensure superior performance than using Blackman window or Hamming window alone, and the length of the filters is $N=128$. Compared with Blackman and Hamming windows, the new window has the best first side-lobe level, the best maximum side-lobe level and the best spectral efficiency.

3.2.2. Kumar’s Method

In [126], the Hamming and Gaussian windows are combined as a new window function, whose expression is

$$\begin{array}{c}\hfill w\left(n\right)=\left[0.54+0.46cos\left(\frac{2\pi n}{N-1}\right)\right]e^{-\frac{1}{2}{\left(\frac{2\alpha n}{N-1}\right)}^2},\end{array}$$

(43)

where $0\le |n|\le (N-1)/2$, and the value of the parameter $\alpha$ determines the performance of the filter.

Figure 3 shows the frequency responses of Hamming window, Gaussian window and the new window proposed in [126] for $N=64$. Note that the window in [126] has the best side-lobe attenuation compare with Hamming and Gaussian windows. However, it increases the width of the main-lobe.

3.2.3. Mottaghi-Kashtiban’s Method

In [127], a special case of raised cosine windows is presented. The proposed window function is

$$w\left[n\right]=a_0-a_{1}cos\left(\frac{2\pi n}{N-1}\right)-a_{3}cos\left(\frac{6\pi n}{N-1}\right),0\le n\le N-1.$$

(44)

For normalization, that is, $w\left[\frac{N-1}{2}\right]=1$, then

$$a_0+a_1+a_3=1.$$

(45)

This new window is symmetric about $(N-1)/2$, and it has linear phase, thus the window function is obtained by

$$w\left[n\right]+w[n-(N-1)/2]=2a_0,\frac{N-1}{2}\le n\le N-1.$$

(46)

The expression in Equation (44) is a 4th order raised cosine window [128] and its third term is zero

$$w\left[n\right]=\sum _{i=0}^3{a}_{i}cos\left(\frac{2i\pi n}{N-1}\right),0\le n\le M,a_2=0.$$

(47)

The optimal values are found by optimization and approximation. The relationship between the parameters in (47) and the length of window function N can be expressed as

$$\left\{\begin{array}{cc}\hfill a_0& =0.537-\frac{0.3}{N+14},\hfill \\ \hfill a_1& =0.46+\frac{0.25}{N+14},\hfill \\ \hfill a_3& =1-a_0-a_1.\hfill \end{array}\right.$$

(48)

Figure 4 shows the frequency responses of Hamming window, Gaussian window and the new window proposed in [127] for a typical filter length of $N=41$. All of the windows have approximately equal mainlobe width. Compared with Hamming window, the new window has better peak level of maximum side-lobes, meanwhile it has better performance of side-lobe attenuation than Gaussian window.

3.2.4. Rakshit’s Method

In [129], a new form of adjustable window function combining a tangent hyperbolic function and a weighted cosine series is proposed. In this paper, the modified tangent hyperbolic function is given as

$$\begin{array}{cc}\hfill y_1=tan& \left\{\frac{n-\frac{N-1}{2}+{cosh}^2\left(\alpha \right)}{B}\right\}\hfill \\ \hfill & -tan\left\{\frac{n-\frac{N-1}{2}-{cosh}^2\left(\alpha \right)}{B}\right\},\hfill \end{array}$$

(49)

and the weighted cosine function is expressed as

$$y_2=0.375-0.5cos\left(\frac{2\pi n}{N-1}\right)+0.125cos\left(\frac{4\pi n}{N-1}\right),$$

(50)

where $\alpha$ and B are the constants, and the symbol $n=0,1,2,3,\cdots ,(N-1)$. Then the new window function can be expressed as

$$w\left(n\right)=\left\{\begin{array}{cc}{(y_1\times y_2)}^{{\gamma }^{\frac{1}{5}}},\hfill & \hfill 0\le n\le N,\\ 0,\hfill & \hfill \mathrm{otherwise},\end{array}\right.$$

(51)

where $\gamma$ is a variable which controls the shapes and frequency response of window function, and “$\times$” denotes dot product.

Figure 5 shows the frequency responses of Gaussian window and the new window proposed in [129]. The length of the filters is $N=65$, and $B=1$, $\alpha =2.5$, $\gamma =1.5$ for the new window in [129]. It can be observed that the new window has much less side-lobe peak level than Gaussian window while its main-lobe width keeps exactly the same.

3.2.5. Martin-Martin’s Method

In [130], a window designing method for cosine-modulated transmultiplexers is proposed, which is called generalized windowing method for transmultiplexers (GWMT). A 4th order generalized cosine window function is expressed as

$$w\left(n\right)=\sum _{i=0}^3{(-1)}^i{A}_{i}cos\left(\frac{2\pi in}{N-1}\right),$$

(52)

where $n=0,1,2,\cdots ,N-1$, and $A_i$ are the weights of the terms for $i=0,1,2,3$. The designed window function is normalized as

$$\sum _{i=0}^3{A}_i=1.$$

(53)

The weights $A_i$ and the cut-off frequency ${\omega }_c$ of the ideal low-pass filter can be adjusted by minimizing the objective function

$$\varphi \left(\mathbf{x}\right)=-\frac{1}{F}\sum _{f=0}^{F-1}\frac{E_f\left(\mathbf{x}\right)}{\beta E_{\mathrm{ICI}}^{\left(f\right)}\left(\mathbf{x}\right)+(1-\beta )E_{\mathrm{ISI}}^{\left(f\right)}\left(\mathbf{x}\right)},$$

(54)

and

$$\left\{\begin{array}{c}{E}_f=\frac{1}{\pi }{\int }_0^{\pi }(|T_{ff}\left(e^{j\omega }\right){|}^2)d\omega ,\hfill \\ E_{ICI}^{\left(f\right)}=\frac{1}{\pi }{\int }_0^{\pi }\left(\sum _{l=0,l\ne f}^{F-1}{|T_{fl}\left(e^{j\omega }\right)|}^2\right)d\omega ,\hfill \\ E_{ISI}^{\left(f\right)}=\frac{1}{\pi }{\int }_0^{\pi }{\left\{\parallel T_{ff}{\parallel }_1-|T_{ff}\left(e^{j\omega }\right)|\right\}}^{2}d\omega ,\hfill \\ \parallel T_{ff}{\parallel }_1=\frac{1}{\pi }{\int }_0^{\pi }|T_{ff}\left(e^{j\omega }\right)|d\omega ,\hfill \end{array}\right.$$

(55)

where F is the channel number, $\beta$ is the compromise factor satisfying $0\le \beta \le 1$, T describes the transfer function, $\mathbf{x}=[A_0,A_1,A_2,{\omega }_c]$ denotes the adjustable parameter vector, and $A_3$ can be obtained by the Equation (53). This optimization problem is solved by the Nelder-Mead simplex minimization algorithm.

Figure 6 displays the frequency responses of Blackman window, Kaiser window and the generated window by GWMT method. The length of the filters is $N=2KF$, where $K=3$, $F=32$. It is seen that the GWMT method based window has the highest signal-to-overall-interference ratio levels. The comments, pros and cons of the above windowing based methods are summarized in

Table 3. Summary of windowing based methods.

Methods	Comments	Pros and Cons
Jain’s method [125]	Connecting Hamming window and Blackman window by a parameter $\lambda$.	Pros: Best first side-lobe level and spectrum efficiency compared to Hamming and Blackman windows. Cons: Large transition bandwidth.
Kumar’s method [126]	Product of Hamming window and Gaussian window.	Pros: Excellent performance of stopband attenuation compared to Hamming and Gaussian windows. Cons: Under the same parameter setting, the width of main-lobe will increase.
Mottaghi-Kashtiban’s method [127]	Design a four-semester raised cosine window by optimizing window parameters.	Pros: Narrower main-lobe under similar conditions compared to Hamming window. Cons: Although with improved performance of stopband attenuation, the performance gain is inconspicuous.
Rakshit’s method [129]	A window function combining tangent hyperbolic function and weighted cosine series using an adjustable parameter $\gamma$.	Pros: Higher side-lobe roll-off ratio under the same main-lobe width compared to Gaussian window. Cons: Need to constantly adjust the parameters $\gamma$.
Martin-Martin’s method [130]	The parameters of a four-term generalized cosine window are optimized on the basis of a given objective function.	Pros: Best performance of signal-to-overall-interference ratio compared to Kaiser and Blackman windows. Cons: High algorithm complexity.

.

3.3. Optimization Based Methods

In optimization based methods, some design parameters are usually given, including the filter length, the passband cutoff frequency, the stopband cutoff frequency, the number of channels, the maximum allowable distortion, and so forth. According to corresponding evaluation criteria, different objective functions and algorithms are proposed to optimize the filter parameters. This type of filter design methods is more flexible due to adjustable objective functions and constraints. Both constrained and unconstrained optimization problems can be formulated for the filter design.

3.3.1. Ababneh’s Method

In Reference [131], the authors propose designing the FIR filter by the particle swarm optimization (PSO) algorithm. For each particle i, the position ${\mathbf{P}}_i$ and the velocity ${\mathbf{V}}_i$ are shown as

$${\mathbf{P}}_i=(p_{i,1},p_{i,2},\cdots ,p_{i,D}),$$

(56a)

$${\mathbf{V}}_i=(v_{i,1},v_{i,2},\cdots ,v_{i,D}),$$

(56b)

$$\begin{array}{cc}\hfill v_{i,d+1}=\mu v_{i,d}& +c_1{\beta }_1(p{b}_{i,d}-p_{i,d})+c_2{\beta }_2(g{b}_d-p_{i,d}),\hfill \end{array}$$

(56c)

$$p_{i,d+1}=p_{i,d}+v_{i,d+1},$$

(56d)

where D is the iteration of velocity and position, $\mu$ is the weighting function, $c_1$ and $c_2$ are the acceleration constants, ${\beta }_1$ and ${\beta }_2$ are random numbers in the range $[0,1]$. In addition, the pbest $p{b}_{i,d}$ denotes the personal best of the $i_{th}$ particle vector at the $d_{th}$ dimension. The gbest $g{b}_d$ denotes the group best at the $d_{th}$ dimension. During each iteration, the position and the velocity are calculated by the Equations (56c) and (56d), respectively.

3.3.2. Luitel’s Method

In Reference [132], the differential evolution particle swarm optimization (DEPSO) algorithm combined by differential evolution and PSO is proposed to design the FIR filter. In the DEPSO algorithm, the offspring is generated by the nutation of the parent, the Gaussian distribution is considered and the parent can be represented by the gbest. For mutation, there are four random particles chosen from the population to produce an offspring. A mutation operator is expressed as below

$$\begin{array}{c}\hfill if(rand<RRord==m)\\ \hfill T_{i,d}=g{b}_d+{\delta }_{2,d},\end{array}$$

(57)

$${\delta }_{2,d}=\frac{(p{b}_{1,d}+p{b}_{2,d})+(p{b}_{3,d}+p{b}_{4,d})}{2},$$

(58)

where $RR$ is the reproduction rate, $rand$ is a random number in the range of $[0,1]$, m means a dimension which is randomly chosen, $T_{i,d}$ is the offspring and ${\delta }_{2,d}$ is the weighted error in different dimensions.

3.3.3. Gupta’s Method

In Reference [133], the restart PSO (RPSO) is used to design the FIR filter. Compared with PSO, the RPSO algorithm depends on the two following criteria:

Criterion 1: Terminate if the fitness’s standard deviation of swarm is smaller than ${10}^{-3}$. In this case, the particles are randomly redistributed in the search space with a probability of $1/D$.

Criterion 2: Terminate if the change in fitness of the objective function is below ${10}^{-8}$ for certain generations, then the particles are restarted by calculating derivatives to gbest.

Figure 7a shows the frequency responses of the PSO and RPSO algorithms, and Figure 7b shows the frequency responses of the PSO and DEPSO algorithms. Note that the filters designed by the three algorithms have similar performance. Compared to the PSO algorithm, the RPSO algorithm can prevent the results from falling into a local optimal solution, while the DEPSO algorithm has better convergence performance.

3.3.4. Li’s Method

In Reference [134], a method to design the FIR filter based on genetic algorithm (GA) is proposed. We summarize this algorithm in Algorithm 1.

Algorithm 1 The GA Algorithm.

Input: Initialize parameters: population size $NP$, current generation $g=1$, maximum generation   $G_{max}$, swarm S; Output: The best resolution $BS$; 1: while$g\le G_{max}$do 2:     for $i=1toNP$ do 3:         Evaluate fitness of $S_g$; 4:     end for 5:     for $i=1toNP$ do 6:         Select operation to $S_g$; 7:     end for 8:     for $i=1toNP/2$ do 9:         Crossover operation to $S_g$; 10:     end for 11:     for $i=1toNP$ do 12:         Mutation operation to $S_g$; 13:     end for 14:     for $i=1toNP$ do 15:         $S_{g+1}=S_g$; 16:     end for 17:     $g=g+1$; 18: end while 19: return The best resolution $BS$.

3.3.5. Karaboga’s Method

In Reference [135], an FIR filter is designed based on the differential evolution (DE) algorithm. The main difference between DE algorithm and GA algorithm is that GA algorithm relies on the crossover, while DE algorithm relies on the mutation operation. The specific steps of the DE algorithm are provided in Algorithm 2.

In order to evaluate the performance of the designed FIR filters by GA and DE algorithms, the least mean squared error (LMSE) is used, and the fitness function is defined as

$$\mathrm{Fitness}=\frac{1}{\mathrm{LMSE}},$$

(59)

and the $\mathrm{LMSE}$ is given as

$$\mathrm{LMSE}={\left(\sum _k(|H_i\left(e^{-j{\omega }_k}\right)|-|H_d\left(e^{-j{\omega }_k}\right){|)}^2\right)}^{\frac{1}{2}},$$

(60)

where $H_i\left(e^{-j{\omega }_k}\right)$ is the frequency response of ideal filter, and $H_d\left(e^{-j{\omega }_k}\right)$ is the frequency response of designed filter.

Figure 8 shows the frequency responses of the FIR filters designed by GA and DE algorithms. The length of the two filters is $N=21$, and 

Table 4. The parameters of GA and DE algorithms.

GA Algorithm	DE Algorithm
Population size = 100	Population size = 100
Crossover rate = 0.8	Crossover rate = 0.8
Mutation rate = 0.01	Scaling factor = 0.8
Generation number = 500	Combination factor = 0.8
—	Generation number = 500

shows the control parameters of GA and DE algorithms, which have a significant effect on the performance of these two algorithms. As can be seen from Figure 8, the FIR filters designed by GA and DE algorithms have similar performance, that is, the stopband attenuation using GA is slightly better than that using DE. But the convergence speed of the DE algorithm is faster than GA.

Algorithm 2 The DE Algorithm.

Input: Initialize parameters: population size $NP$, current generation $g=1$, maximum generation  $G_{max}$, dimension D, tolerance $\epsilon$, swarm S, base vector s, mutant vector v, trial vector u; Output: The best resolution $BS$;  for $i=1toNP$ do    for $d=1toD$ do           $s_{i,g}^d=s_{min}^d+rand·(s_{max}^d-s_{min}^d)$;    end for   end for   while $(|f\left(BS\right)|\ge \epsilon )or(g\le G_{max})$ do    for i = 1 to NP do           for d = 1 to D do              $v_{i,g}^d=\mathrm{Mutation}\left(s_{i,g}^d\right)$;              $u_{i,g}^d=\mathrm{Crossover}(s_{i,g}^d,v_{i,g}^d)$;           end for           if $f\left(u_{i,G}\right)<f\left(s_{i,g}\right)$ then              $s_{i,g}=u_{i,g}$;              if $f\left(s_{i,g}\right)<f\left(BS\right)$ then                $BS=s_{i,g}$;              end if           else              $s_{i,g}=s_{i,g}$;           end if    end for    $g=g+1$;  end while  return The best resolution $BS$.

3.3.6. Chen’s Method

In Reference [98], the prototype filter is designed by directly optimizing the filter coefficients. This method is mainly designed for FBMC based systems, aiming to minimize the stopband energy and constrain the ISI/ICI. The optimization problem is formulated as

$$\begin{array}{cc}\hfill & \underset{h\left(0\right),h\left(1\right),\cdots h(N-1)}{min}{\int }_{{\omega }_0}^{\pi }{|H\left(e^{j\omega }\right)|}^{2}d\omega \hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\left\{\begin{array}{cc}h\left(n\right)=h(N-n-1),\hfill & \hfill C1\\ \mathrm{ISI}\mathrm{and}\mathrm{ICI}\le T_1,\hfill & \hfill C2\\ \sum _{n=0}^{N-1}{\left(h\left(n\right)\right)}^2=1,\hfill & \hfill C3\end{array}\right.\hfill \end{array}$$

(61)

where $T_1$ is a threshold. However, this optimization problem is non-convex and non-linear, which greatly increases the computational complexity. Through a series of approximation of the constraints, the number of unknowns in the optimization problem is enormously reduced. Therefore, the optimization problem can be solved and the computational complexity becomes acceptable.

3.3.7. Hunziker’s Method

In Reference [136], the design of filter banks for maximal time-frequency (TF) resolution is proposed. Two properties should be satisfied to compute the optimal DFT filter banks. Firstly, the window under the filter operation is orthogonal, thus for white input processes, the sampling at the output of the filter banks generates uncorrelated random variables. Secondly, to ensure the minimum leakage of signal components outside the target area in TF plane, the prototype window shows the best TF localization feature. To satisfy the first property, the parametrization of paraunitary filter banks in Reference [137] is used and longer pulses are extended in Reference [138]. As for the second property, the Rihaczek distribution is used in TF region, and the objective function along with the constraints is transformed into a form which can be solved by semi-definite programming (SDP) algorithm.

3.3.8. Dedeoğlu’s Method

In Reference [139], the FIR filter design based on SDP is proposed. In order to satisfy both constraints on magnitude and phase responses of the filter, a non-convex quadratic constrained quadratic programming (QCQP) is constructed. Then, by relaxing the QCQP, a convex SDP is obtained, where the variables of optimization are limited to rank-1. Finally the global optimum can be found by using a convex solver. To obtain the rank-1 solution, a novel directed iterative rank refinement (DIRR) algorithm providing monotonic improvement is proposed, and a sequence of convex optimization problems are solved to minimize an adaptively chosen cost function. The performance of this design method is extensively illustrated over design cases under a variety of constraints.

3.3.9. Kobayashi’s Method

A relatively new method of convex optimization is proposed to obtain the filter coefficients for FBMC, in pursuit of superior spectrum features while maintaining high symbol reconstruction quality [140]. The objective function is constructed as the minimization of out-of-band pulse energy. At the meanwhile the constraints include a maximum tolerable self-interference level and a fast spectrum decay. The main idea of the work in Reference [140] locates in transforming the formulated problem, which belongs to a non-convex QCQP, into a convex QCQP. In order to circumvent the non-convexity, a relaxation is utilized to transform the norm-2 equality constraint into a norm-1 equality constraint. Thus the transformed problem benefits from existent optimization tools. The comments, pros and cons of the above optimization based methods are summarized in

Table 5. Summary of optimization based methods.

Methods	Comments	Pros and Cons
Ababneh’s method [131]	Design the filter using the PSO algorithm.	Pros: Simple and intuitive. Cons: Poor stopband attenuation performance of the filter under low-order conditions.
Luitel’s method [132]	Design the filter using the DEPSO algorithm.	Pros: Compared with PSO algorithm, the filter has better convergence consistency. Cons: Improvement of the stopband attenuation is not obvious.
Gupta’s method [133]	Design the filter using the RPSO algorithm.	Pros: Compared with PSO algorithm, the filter has better convergence and can avoid falling into local optimum. Cons: Improving the performance of the stopband attenuation is not obvious.
Li’s method [134]	Design the filter using the GA algorithm.	Pros: Obtain near global optimum solutions. Cons: Slow convergence rate and poor stopband attenuation performance.
Karaboga’s method [135]	Design the filter using the DE algorithm.	Pros: Better convergence and acceptable computational complexity compared to GA. Cons: Improvement of the stopband attenuation is not obvious.
Chen’s method [98]	Directly optimize filter coefficients.	Pros: Minimize the ISI/ICI and the stopband energy for FBMC modulation. Cons: High computational complexity.
Hunziker’s method [136]	An optimization algorithm aiming at minimizing TF resolution.	Pros: Minimize the TF resolution. Cons: High first sidelobe.
Dedeoğlu’s method [139]	Design the filter by convex optimization using DIRR algorithm.	Pros: Robust design under the phase and group delay constraints. Cons: Approximated solution.
Kobayashi’s method [140]	Minimize the out of band pulse energy through a relaxed QCQP.	Pros: High symbol reconstruction performance and desirable spectral features. Cons: Approximated solution.

.

4. Discussion

Future wireless communication networks will impose strict requirements on multicarrier modulation schemes in terms of spectral efficiency, latency, computational complexity, and so forth. As shown in

Table 6. Respective characteristics and applicability of different multicarrier modulation waveforms.

	OFDM	FBMC	GFDM	UFMC	F-OFDM
Spectral Efficiency	Medium	High	Medium	High	Medium
Out of Band	High	Low	Low	Low	Low
Cyclic Prefix	Yes	No	Yes	No	Yes
Synchronized Requirement	High	Low	Medium	Low	Low
Latency	Short	Long	Short	Short	Short
Effect of Frequency Offset	Medium	Low	Medium	Medium	Medium
PAPR	High	High	Low	Medium	High
Computational Complexity	Low	High	High	High	Medium
Short-Burst Traffic	No	No	Yes	Yes	No

, crucial characteristics and applicability of five promising modulation waveforms are clearly reflected, including spectral efficiency, out of band, CP, synchronization requirement, latency, effect of frequency offset, PAPR, computational complexity, and short-burst traffic, which implicate a concrete limit to the real use of different MCM schemes. To further learn about the merits and defects of various MCM waveforms, interested readers are referred to the related works in Reference [141] for more detailed information. Since prototype filter design directly affects most characteristics of modulation waveforms such as spectral efficiency [142,143], out of band [144], it has attracted extensive research attention in the field of MCM communication. In recent years, various prototype filter design methods are designed in accordance with the above practical requirements of modulation waveforms in Table 6, and some commonly used FIR design methods for different multicarrier modulation waveforms are concluded in

Table 7. Summary of filter design methods used in existing multicarrier modulation waveforms.

	OFDM	FBMC	GFDM	UFMC	F-OFDM
Frequency Sampling Methods	—	[51,118]	[148,149]	—	—
Windowing based Methods	[150,151]	[4]	[40,152]	[42]	[45,46,145,153]
Optimization based Methods	[154]	[140,155,156,157]	[158,159,160,161]	[162,163]	—

. The contents of this table will be continuously enriched as time goes on. In addition, we analyze the MCM system performance using three types of FIR design methods in terms of PSD and BER, which are two another important evaluation criteria [81,82,83,145,146,147]. The simulation results of PSD and BER performances are presented in Figure 9, where the FBMC is selected as the system modulation type, the number of subcarriers for FBMC is 1024, and the offset quadrature amplitude modulation (OQAM) technology is adopted. In the following, some critical characteristics and application scope of the three categories of FIR design methods are discussed and summarized, respectively.

4.1. Frequency Sampling Methods

The performances of PSD and BER about different prototype filters using frequency sampling techniques are given in Figure 9a,b, respectively. From the Figure 9a, it can be seen that the frequency sampling methods can achieve good PSD performance. The prototype filter designed by Bellanger’s method has the best performance compared to other frequency sampling techniques, on the other hand, the prototype filter designed in Reference [123] has the worst BER performance while the others have the similar performance, as shown in Figure 9b.

Generally, a small number of parameters are simply designed in frequency domain, thus frequency sampling methods are inclined to optimal design and narrowband filter design where only a few non-zero values are needed. Furthermore, it is obvious that there are few side-lobes which result in better spectrum utilization and improving spectral efficiency. For time complexity, Bellanger’s method achieves the order of magnitude $\mathcal{O}(NlogN)$ as most of other methods are its improved versions, and N denotes the filter length. When the value of N increases, low side-lobes may appear in the frequency domain and adjacent channel interference can be reduced, thus frequency sampling methods can be taken into consideration while low interference is required. In addition, frequency sampling techniques can satisfy the NPR condition, the research projects of 5G related networks, such as PHYDYAS and 5GNOW, consider the frequency sampling technique for FBMC modulation. Frequency sampling methods are also applied for GFDM modulation [148,149] as concluded in Table 7. However, frequency sampling methods generally have long filter length, which results in long latency, that is, there is a trade-off between high spectrum efficiency and low latency. Moreover, the position of the frequency control point is limited by N-sampling points on the frequency domain. This implies that the sampling frequency can only be an integral multiple of $2\pi /N$, resulting in that the cutoff frequency of filter is hard to control. If the cutoff frequency is freely selected, we must increase the number of sampling points N, that is, increasing the filter length, which is not conducive to short uplink burst communication in 5G scenarios.

4.2. Windowing Based Methods

The performances of PSD and BER about different prototype filters using windowing based techniques are given in Figure 9c,d, respectively. The aforementioned windowing based techniques have similar PSD and BER performances while the method in Reference [129] performs relatively poor compared with other methods.

The windowing based FIR filters are designed in time domain, and the basic idea is that the linear phase FIR filter multiplies by a specific windowing function, thus the time complexity is proportional to $\mathcal{O}\left(N\right)$. Generally, windowing based techniques can be seen as improved versions of classical windowing functions, nevertheless the performance improvement is not very remarkable compared with typical techniques such as Hamming window, Hanning window, Blackman window and so on. From the view of practical use, windowing based functions are easy to implement and bring little computational complexity burden on systems, therefore they are able to be used for the prototype filter design in almost all multicarrier modulation systems, as shown in Table 7. However, it is difficult to accurately control the passband cutoff frequency of the filter. The windowing process leads to a truncation effect and hence produces a transition band. Although increasing the length of the window can reduce the transition band, the amplitude of fluctuation cannot be suppressed due to Gibbs phenomenon. Changing the windowing shape can reduce the passband/stopband attenuation, but at the expense of increasing the width of transition band.

4.3. Optimization Based Methods

The performances of PSD and BER about different prototype filters using optimization based techniques are given in Figure 9e,f, respectively. The evolutionary algorithms in References [131,132,133,134,135] are used for linear phase FIR filter design in recent years, and they could be taken into account in multicarrier modulation applications. In this paper, FBMC based systems using five different evolutionary algorithms are simulated, as shown in Figure 9e,f. By analyzing the simulation of the filters with the same order, it is noted that different evolutionary algorithms achieve similar PSD and BER performances.

Compared with frequency sampling techniques and windowing based techniques, the corresponding filter parameters of optimization based methods, that is more concerned with the local optimization and algorithm convergence problem, can be optimized depending on different evaluation criteria, and the selected design parameters determine the effectiveness of the solution to the optimization problem. The complexity of this type of design techniques seems to be much higher than other two techniques, for instance, the complexity of PSO based method is about the order of $LP\mathcal{O}\left(N^2\right)$, where L is the iteration size and P is the population size. This complexity is much less than QCQP based methods or non-convex optimization methods, of which the complexity is up to $\mathcal{O}\left(N^3\right)$ or higher [164]. Nevertheless, optimization based methods are much more flexible as they focus on local optimization to support the constraints imposed by various scenarios such as cognitive radio [157], massive MIMO [156] and so on. For example, when FBMC is applied for opportunistic spectrum sharing, in order to avoid interfering with other bands, well localized filters in time and frequency are prone to be designed to minimize Out of Band [140]. Similarly, there are also some optimization based methods for different waveform candidates, for example, OFDM [154], FBMC [140,155,156], GFDM [158,159,160,161], UFMC [162,163]. In general, the establishment and constraints imposed on objective functions are often highly non-linear, which increases the computational complexity and also sensitive to the selection of initial values. In addition, it is possible to fall into local optimal range and fail to identify global optimal solution, which is one of the major issues to be solved in optimization based design methods.

As discussed earlier, since multicarrier modulation can enlarge the system capacity and have the relative immunity to defend the multipath fading effect, it dominates the main stream in future communication technology. The critical part of multicarrier modulation lies in the FIR filter design. A good filter design scheme can improve some important performances not only including PSD and BER but also some other crucial aspects, consequently increasing the competitiveness of waveform candidates and improving the communication quality to suit various application scenarios of 5G or future networks. FIR filters have been widely used in many engineering applications involved with digital signal processing, such as the communication signal [165,166], the speech signal [167,168] and the medical signal [169]. We can conclude without exaggeration that where digital signal processing is needed, where there is a shadow of FIR.

5. Conclusions

The prototype filter plays an important role in multicarrier modulation systems and the FIR filter is considered to be the suitable choice in wireless communication systems. This paper has reviewed existing FIR filter design methods which are categorized into frequency sampling methods, windowing based methods and optimization based methods. The concept and principle of each method are described in detail, and the merits and drawbacks of corresponding prototype filters are summarized. Finally, the performances of FIR design methods in different multicarrier modulation systems are evaluated and discussed in terms of PSD and BER. It is expected that this survey work can provide a basis for the selection of prototype filters in future wireless communication systems.