Modified Decision Feedback Equalizers for Nonlinearity Compensation in Coherent PAM4 Transmission System

Abstract

To address chromatic dispersion (CD) and nonlinear impairments in coherent optical four-level pulse amplitude modulation (PAM4) systems, we propose a magnitude-assisted decision feedback equalizer (MA-DFE). The proposed scheme utilizes signal amplitude information to construct the error function, which is robust to carrier phase noise. Therefore, no additional carrier phase recovery operation is required during digital signal processing (DSP). Under conditions without CD pre-compensation, MA-DFE achieves 80 Gb/s single-wavelength transmission over a 25 km standard single-mode fiber (SSMF) in the C-band. When considering a bit error rate (BER) of 1 × 10⁻² for the soft decision threshold, the link budget achieves 27 dB. In addition, we incorporate the phase into the error function, proposing the phase-assisted decision feedback equalizer (PA-DFE). PA-DFE is also unaffected by carrier phase noise and demonstrates better performance than MA-DFE when equalizing the more severe signal impairments caused by SOA gain saturation. Ultimately, we achieve a link budget of 29 dB using PA-DFE.

1. Introduction

With the continuous growth of high-bandwidth access network service demands, communication networks based on passive optical network (PON) technology have gained increasing attention from researchers. The 50 G PON standard has now been launched and commercialized [1], and the current research focus is to achieve higher capacity, lower cost, and lower latency.

In order to meet the requirements of low cost and simple implementation for PONs, advanced modulation formats based on intensity modulation and direct detection (IM/DD), including discrete multitone (DMT), carrier-less amplitude phase (CAP) and four-level pulse amplitude modulation (PAM4) have attracted much attention due to the high spectral efficiency [2,3,4]. Among these, the low-complexity PAM4 has been identified by the IEEE as the 802.3 bs 400 G standard [5].

IM-DD systems operating in the C-band are susceptible to the effects of chromatic dispersion (CD) leading to signal distortion. Fiber Bragg gratings (FBGs) are commonly used as CD compensation modules. Additionally, methods such as single-sideband or vestigial sideband (SSB/VSB) modulation [6] and CD pre-compensation are often employed. However, these approaches require complex system configurations and expensive costs, limiting their commercial viability. In contrast, digital signal processing (DSP)-based equalization techniques enhance dispersion tolerance while maintaining cost-effectiveness. Common equalization methods include feedforward equalizers (FFEs), decision feedback equalizers (DFEs), and maximum-likelihood sequence estimation (MLSE). Based on this, some modified equalization algorithms have been proposed. In [7], a DFE based on a nonlinear filtering structure is proposed for a 40 Gb/s PAM4-PON system. In [8], pre-equalizers based on Volterra and post-equalizers based on LMS were used to achieve 50 Gb/s transmission over 20 km with a power budget of 27 dB. However, due to the low sensitivity of IM-DD, further increasing the transmission rate would lead to a decrease in link budget, which limits the application of IM-DD in high-speed PON systems.

In contrast to IM-DD, coherent detection technology offers superior resistance to interference and higher sensitivity. It serves as a powerful method to mitigate channel impairments and enhance receiver sensitivity. Therefore, applying coherent detection to PON-based access networks holds significant potential for development [9,10]. In addition, considering the cost of coherent optical systems, researchers propose to combine a coherent receiver with an intensity-modulated transmitter [11,12], which promotes the application of PAM4 in coherent systems. Signal distortions including CD, linear, and nonlinear impairments also exist in coherent systems. With the decrease in DSP costs, many equalization algorithms based on coherent detection have also been widely discussed [13,14]. In [15], the researchers used a Volterra equalizer and MLSE for equalizing PAM4 signals, but additional phase recovery operations were required, which significantly increased the computational complexity. The multimodulus algorithm (MMA) can balance PAM4 signals without the need for carrier phase recovery operation. However, once the carrier phase noise is too large, the equalization effect will deteriorate, and the MMA cannot perform nonlinear compensation [16]. Furthermore, the application prospects of equalizers based on neural network (NN) models are noteworthy. In [17], a multi-purpose complex-valued neural network was used to equalize the QAM signal for a 200 Gb/s/λ PON with 20 km of transmission. Nevertheless, the use of these neural networks increases system complexity, leading to higher power consumption. DFEs, compared to FFEs, can compensate for nonlinear distortions and have lower complexity than Volterra and NN equalizers, making them a promising choice for PON. In [18], a DFE with error detection capability was employed to equalize signals in 100 Gb/s coherent 16QAM systems, but this equalizer was sensitive to phase impairments, necessitating additional carrier phase recovery operations for achieving lower error rates. To reduce equalization complexity, a carrier phase recovery-free constant modulus algorithm (CMA)-DFE is proposed in [19] for compensating nonlinear impairments in coherent QPSK systems, achieving a minimum bit error rate (BER) of 6.8 × 10⁻⁴. However, the CMA is only suitable for signals with arbitrary symmetric two-dimensional signal constellations, such as QPSK and M-PSK, and is unsuitable for PAM4 signals. Therefore, there is a need to develop an equalizer for coherent optical PAM4 systems with low complexity and excellent equalization performance.

In this paper, we focus on an access network based on PON, and propose two modified DFEs for coherent optical PAM4 systems, i.e., magnitude-assisted decision feedback equalizers (MA-DFEs) and phase-assisted decision feedback equalizers (PA-DFEs). They can compensate for CD and nonlinear impairments, while being completely robust to carrier phase impairments. Among them, MA-DFEs construct the error function considering the signal amplitude information, whereas PA-DFEs incorporate phase into the error function. We experimentally demonstrate a single-wavelength 80 Gb/s coherent optical PAM4 system in the C-band and conduct comparative analysis of the equalizer’s performance parameters. The experimental results indicate that without CD pre-compensation or carrier recovery operations, using only the PA-DFE and MA-DFE effectively compensates for CD and nonlinear impairments over 25 km SSMF. The BER meets the soft decision threshold of 1 × 10⁻² at a received optical power of −23 dBm and the link budget reaches 27 dB. The experiment also demonstrates different operating states of the SOA at the transmitter. The results show that, when dealing with severe signal impairments caused by SOA gain saturation, the equalization performance of PA-DFE is superior to that of MA-DFE, ultimately achieving a link budget of 29 dB.

2. Principle of the Proposed Electrical Equalization

In coherent systems, complex signal equalization can effectively compensate for CD [20], and the output current $I\left(t\right)$ of the PAM4 signal after intradyne detection is expressed as

$$I\left(t\right)={\displaystyle \frac{1}{2}}R{A}_0{A}_s\left(t\right)exp(j({{\omega }_{IF}t+\theta }_n(t))),$$

(1)

where $R$ is the responsivity of the photodiode, $A_0$ and $A_s\left(t\right)$ are the magnitude of the local oscillator and the received signal, respectively, ${\omega }_{IF}$ represents the frequency difference between the carrier and the local oscillator, and ${\theta }_n\left(t\right)$ denotes the phase noise. It is known that the PAM4 signal carries only intensity information $A_s\left(t\right)$. As mentioned above, conventional adaptive equalization algorithms are sensitive to phase, and the CMA is also unsuitable for PAM signals with asymmetric two-dimensional constellations. Therefore, we aim to propose an equalizer that is not affected by phase impairment. As shown in Figure 1, the constellation points of the received PAM4 signal are rotated due to the carrier frequency offset and the laser linewidth. The equalizer focuses only on the magnitudes of the signal, using the values D1, D2, D3, and D4 as the four reference levels for PAM4, and equalizes the signal to make it converge onto the circles defined by the reference levels. Furthermore, to compensate for the nonlinear impairments of the system, we incorporate decision feedback. In traditional DFEs, the decision value is directly fed back to the equalizer [21]. However, for coherent PAM4 signals, since the decision value is a real signal, directly feeding it back will result in phase differences with the equalized signal. As illustrated in Figure 1, the red points represent the constellation points of the k-th decision feedback, while the blue points represent the constellation points of the k-th equalized signal. There exists a phase difference $\theta \left(k\right)$ between them, which diminishes the equalizer’s robustness against carrier frequency offset and laser linewidth.

Based on the issues analyzed above, we propose a magnitude-assisted decision feedback equalizer (MA-DFE) and a phase-assisted decision feedback equalizer (PA-DFE). Their structure is represented in Figure 2. It consists of two parts: feedforward and feedback section. Note that in the feedback part, to eliminate the phase difference, we extract the phase of the equalized signal and assign this phase to the decision value before feeding it back into the equalizer.

Therefore, the output of the MA-DFE and PA-DFE can be expressed as

$$y\left(k\right)=\sum _{P=-{\displaystyle \frac{N}{2}}}^{{\displaystyle \frac{N}{2}}-1}{w}_p^{F}x\left(k-p\right)+\sum _{q=1}^M{w}_q^{B}d\left(k-q\right)e^{j\theta \left(k-q\right)}$$

(2)

where $x\left(k\right)$ is the input value, $d\left(k\right)$ is the decision value, $\theta \left(k\right)$ is the phase angle of $y\left(k\right)$, $w_p^F$ and $w_q^B$ are the feedforward and feedback tap coefficients, respectively, and $N$ and $M$ are the numbers of the feedforward and feedback taps, respectively. If $M=0$, MA/PA-DFE are converted to PA/MA-FFE.

When constructing the error function, to eliminate the interference caused by the phase difference, the MA-DFE formulates a real error function based on the magnitude of $y\left(k\right)$. Meanwhile, the PA-DFE assigns the phase of $y\left(k\right)$ to $d\left(k\right)$ to create a complex error function. The formulas are expressed as follows:

$${e\left(k\right)}_{MA\text{-}DFE}=d\left(k\right)-\left|y\left(k\right)\right|$$

(3)

$${e\left(k\right)}_{PA\text{-}DFE}=d\left(k\right)e^{j\theta \left(k\right)}-y\left(k\right)$$

(4)

The recursive least squares (RLS) algorithm is chosen for updating tap coefficients in equalizers due to its robust and rapid convergence. First, $w_p^F$ and $w_q^B$ are represented by the vector $W\left(k\right)$ as

$$W\left(k\right)=[w_{-{\displaystyle \frac{N}{2}}}^F\dots w_0^F\dots w_{{\displaystyle \frac{N}{2}}-1}^F{w}_1^B{w}_2^B\dots w_M^B]$$

(5)

Afterward, the $W\left(k\right)$ of the MA-DFE and PA-DFE can be updated as follows:

$${W\left(k\right)}_{MA\text{-}DFE}=W\left(k-1\right)+G\left(k\right)e\left(k\right)y^{*}\left(k\right)$$

(6)

$${W\left(k\right)}_{PA\text{-}DFE}=W\left(k-1\right)+G\left(k\right)e^{*}\left(k\right)$$

(7)

where ${\left(\right)}^{\mathrm{*}}$ denotes complex conjugation, and $G\left(k\right)$ is a gain vector. Let

$$\begin{array}{c}U\left(k\right)=[x\left(k+{\displaystyle \frac{N}{2}}\right)\dots x\left(0\right)\dots x\left(k-{\displaystyle \frac{N}{2}}+1\right)d(k-1)e^{j\theta \left(k-1\right)}\\ {\dots d\left(k-M\right)e^{j\theta \left(k-M\right)}]}^T\end{array}$$

(8)

$G\left(k\right)$ is specifically computed as

$$G\left(k\right)={\displaystyle \frac{P(k-1)U\left(k\right)}{\lambda +U^H\left(k\right)P(k-1)U\left(k\right)}}$$

(9)

where ${\left(\right)}^H$ denotes the conjugate transpose, and $\lambda$ is the forgetting factor $(\lambda \in (\mathrm{0,1}\left]\right)$. $P\left(k\right)$ is an inverse matrix, updated as follows:

$$P\left(k\right)={\displaystyle \frac{1}{\lambda }}[P\left(k-1\right)-G(k)U^H(k)P(k-1)]$$

(10)

In the MA-DFE and PA-DFE equalization process, there are two stages. The initial stage involves the training convergence process, where the decision value $d\left(k\right)$ directly corresponds to the training sequence:

$$d\left(k\right)=Training\left(k\right)$$

(11)

where the training sequence is an ideal PAM4 signal. By continuously training and iterating, the error function $e\left(k\right)$ converges, thereby obtaining the optimal tap coefficients for the equalizer.

Following the convergence of the error function, the second stage commences. During this stage, the decision value d(k) is determined by the magnitude of the equalized signal

$$d\left(k\right)=\left|y\left(k\right)\right|$$

(12)

It should be emphasized that after the tap coefficients are updated in the first stage, the error function has already converged, so the tap coefficients are not updated in the second stage.

3. Experimental Setup

The C-band 80 Gb/s coherent optical PAM4 transmission system setup and DSP block diagram are shown in Figure 3a. At the transmitter, a pseudo-random sequence of the length 2¹¹–1 was generated using MATLAB (R2022b) and mapped into PAM4 symbols. PAM4 symbols were upsampled by a factor of 4 and pulse-shaped using a root-raised cosine filter with a roll-off factor of 0.01. Then, the resulting PAM4 signal was loaded into an arbitrary waveform generator (AWG) to generate a 40 GBaud PAM4 electrical signal. After that, the 40 GBaud PAM4 electrical signal from AWG was amplified by a linear electrical amplifier (EA) and then fed into an optical modulator for electrical–optical conversion. Due to the insufficient bandwidth of the electro-absorption modulated laser (EML) in our laboratory for this experiment, we used a Mach–Zehnder modulator (MZM) with a 3 dB bandwidth of 28 GHz as the optical modulator. The laser source was an external cavity laser (ECL) with a center wavelength of 1550 nm and linewidth of 500 Hz. Semiconductor optical amplifiers (SOAs) are considered a promising solution for increasing output power and are commonly integrated with distributed feedback (DFB) lasers and electro-absorption modulators (EAMs) to form EML, thus compensating for modulator losses [22]. Consequently, the SOA is used to amplify the signal power before the PAM4 signal enters the standard single-mode fiber (SSMF). Figure 3b shows the experimentally measured SOA input–output curves, and it can be seen that excessive input power provides higher output power but also leads to SOA gain saturation.

At the receiver, the received optical power (ROP) of the signal was adjusted using a variable optical attenuator (VOA), and the polarization state was controlled by a polarization controller (PC). The tunable semiconductor laser (TSL) with an output power of 12 dBm and a linewidth of 100 kHz was used as the local oscillator. Subsequently, the signal was intradyne-detected by a 25 GHz integrated coherent receiver (ICR), generating two IQ signals. Next, the detected signal was digitized at a rate of 160 GSa/s using a real-time digital signal analyzer (DSA). We performed offline DSP on the received data using MATLAB. Initially, the PAM4 complex signal formed by the combination of the two IQ signals was subjected to matched filtering and then downsampled by a factor of 4. Next, we determined the starting point of the corresponding training sequence by calculating the correlation between the received signal and the training sequence. Then, the training sequence was used to optimize and obtain the best tap coefficients for the PA-DFE and MA-DFE, thereby equalizing the PAM4 signal. Finally, PAM4 signals are hard-determined and demapped, and then the BER is calculated.

The experimentally measured MZM transmission curve is depicted in Figure 4a. When employing it for intensity modulation, the DC bias voltage setting affects the extinction ratio of the PAM4 signal. Typically, setting the DC bias point at the quadrature point ensures linear output from the MZM. However, this configuration typically results in a lower extinction ratio of the PAM4 signal, thereby reducing its resistance to interference. If the DC bias voltage is adjusted to achieve a high extinction ratio, the offset of the bias point will cause a part of the MZM output to enter the nonlinear region, resulting in signal distortion. Similarly, this issue arises when using EAM for PAM4 modulation.

It is worth emphasizing that our proposed PA-DFE and MA-DFE can mitigate these nonlinear impairments to some extent, so that the optimal bias point may not be at the quadrature point. To determine the optimal bias voltages for different equalizers in this system, we incrementally increased the bias voltage from 3.5 V to 4.5 V during experiments. The PA-DFE, MA-DFE, PA-FFE, and MA-FFE were used to equalize the PAM4 signals under different bias voltages, which were captured in a back-to-back (BtB) case with an ROP of −22 dBm. Figure 4b illustrates the bias voltage versus BER curves for different equalizers. It can be seen that the PA-FFE and MA-FFE achieve the lowest BER at a bias voltage of 4.175 V (orthogonal point: Q1), while the PA-DFE and MA-DFE reach their lowest BER at 3.825 V (non-orthogonal point: Q2).

We define the extinction ratio of PAM4 as the ratio of the maximum amplitude to the minimum amplitude of the signal. To show the difference in extinction ratio more clearly, Figure 4c,d show the constellation diagrams of the received PAM4 signals at low rates when the bias points are Q1 and Q2. It can be seen that the amplitude of the PAM4 signal at the quadrature point Q1 is in the range of [0.5, 3], while the amplitude of the PAM4 signal at the non-quadrature Q2 point is in the range of [0, 3]. Therefore, the extinction ratio of the PAM4 signal at Q2 is higher. Although the non-orthogonal point introduces nonlinear distortion, the MA-DFE and PA-DFE can effectively compensate for signal nonlinear distortions, and thus the advantage of high extinction ratio is presented.

4. Experimental Results and Analysis

4.1. The Parameters Affecting Equalizer Performance

After determining the optimal MZM bias voltage for the equalizer in this system, we explored the parameters affecting the performance of the PA-DFE and MA-DFE in the case of BtB transmission, at an ROP of −20 dBm. Figure 5a shows the effect of the number of feedforward taps on the performance of the equalizer when the number of feedback taps is 0 and 10. It can be seen that they have a similar decreasing trend, with the equalizer containing 10 feedback taps having a lower BER. After the number of feedforward taps exceeded 80, all equalizers achieved stable BER performance. Figure 5b shows the effect of the number of feedback taps on the performance when feedforward taps are fixed at 85, and the BER can be significantly decreased as the feedback taps increase. When the feedback taps exceed 15, the BER almost reaches stability. Considering that increasing the number of taps raises computational complexity, the number of taps should be constrained while seeking the optimal equalizer performance.

Then, we investigated the convergence properties of the PA-DFE and MA-DFE (see Figure 6a,b). It can be observed that the forgetting factor $\lambda$ in the RLS algorithm affects both the convergence speed and stability from the figure. For $\lambda =1$, the convergence curves of the PA-DFE and MA-DFE are similar, requiring 1000 iterations for the MSE to stabilize. When $\lambda =0.98$, the convergence is faster and only 500 iterations are needed, but this leads to larger residuals and less stable equilibrium. We also investigate the effect of tap configuration on convergence properties. From the curves, it is observed that their variations are almost the same for tap configurations of (85, 16) and (45, 6). Therefore, it can be concluded that the effect of the number of taps on the convergence rate is relatively weak.

When analyzing the computational complexity of an equalizer, since multipliers require more computational resources compared to adders, we measured the computational complexity by calculating the number of multiplications per output symbol [23]. It is important to note that we did not include the computational complexity of the training process, as once the equalizer converges, the tap training is completed and the equalizer no longer updates. According to the equalizer output expression in Equation (1), in the feedforward section, given that the number of feedforward taps is $N$, and both the tap coefficients $w_p^F$ and the received PAM4 signal $x\left(k\right)$ are complex numbers, multiplying two complex numbers involves four multiplication operations. Therefore, the computational complexity of the feedforward equalization is $4N$. Similarly, in the feedback section, given that the number of taps is $M$, the tap coefficients $w_q^B$ are complex numbers, the phase $e^{j\theta \left(k-q\right)}$ is a complex number, and the decision value $d\left(k\right)$ is a real number, the computational complexity of the feedback equalization is $4M+2M=6M$. Thus, the overall computational complexity of the equalizer can be expressed as:

$$C_{PA/MA\text{-}DFE}=4N+6M$$

(13)

In order to balance the equalizer’s performance and computational complexity, we tested the BER performance and computational complexity for different tap combinations. As shown in Figure 7, it can be observed that as the number of feedback taps increases, the BER decreases, but the computational complexity increases. When the tap numbers are (85, 16), there is a good balance between the equalizer’s performance and computational complexity.

4.2. Equalization Performance

Based on the above analysis, the taps were configured as (85, 16) to balance the performance and computational complexity in our experiments. Due to the low time-varying rate of the channel system, $\lambda =1$ was chosen to maintain the accuracy of the equilibrium convergence, and 1000 training sequences were used to ensure convergence stability. Figure 8a shows the BER versus ROP curves obtained using PA/MA-DFE (85, 16) and PA/MA-FFE (85) under BtB transmission. Note that they are both measured at their respective MZM optimal bias voltages. From the curves, it can be observed that the BER of the PA-DFE and MA-DFE are nearly identical. Using a BER of 1 × 10⁻² as the FEC threshold, both the PA-DFE and MA-DFE achieve a receiver sensitivity of −24 dBm, which is a significant increase of 7 dB compared with the PA-FFE and MA-FFE. Figure 8b–e display the constellation diagrams after equalization by the four types of equalizers, where the circles representing the constellations of the PA-DFE and MA-DFE appear more convergent and distinct.

To increase the link budget of the system, we used an SOA to boost the launch power. However, it is important to note that when the SOA is in gain saturation state, it can introduce nonlinear distortions to the signal. For this reason, we investigated the performance of the PA-DFE and MA-DFE when the SOA was operating under the linear amplification and gain saturation states. We adjusted the input power to change the operating state of the SOA based on its input–output characteristics. When the input power is −12.8 dBm, the SOA exhibits an output power of 4 dBm and operates within the linear amplification regime. Conversely, when the input power is −6.4 dBm, the SOA outputs 8 dBm and transitions into the gain saturation state. The normalized power spectra of the signals in both states are shown in Figure 9a. The new frequency components generated by gain saturation can be clearly observed outside the band, and the in-band signal spectrum is also obviously distorted. These spectral impairments can lead to significant signal distortion. Figure 9b shows the BER performance of the PA-DFE and MA-DFE in two operating states after 25 km of SSMF transmission. Under linear amplification, the PA-DFE and MA-DFE achieve a sensitivity of −23 dBm, and the system link budget is 27 dB. In the gain saturation region, the SOA experiences noise and nonlinear issues. The BER of the MA-DFE at the measured ROP fails to meet the FEC threshold of 1 × 10⁻². After increasing the number of taps to (105, 26), the MA-DFE meets FEC when the ROP is above −17 dBm. However, a sensitivity of −20 dBm is achieved when using the PA-DFE with tap numbers of (85, 16) and the sensitivity can be further improved by 1 dB when increasing the tap numbers to (105, 26). Considering the launch power of 8 dBm, a 29 dB link budget is obtained. Therefore, the PA-DFE demonstrates superior equalization performance compared to the MA-DFE in mitigating PAM4 signal impairments caused by SOA gain saturation.

Finally, the data rate was limited to 80Gb/s due to the bandwidth limitation of the 25 G ICR, and a higher data rate can be achieved by using higher bandwidth devices or applying polarization multiplexing techniques. Also, the link budget can be further improved by using an SOA with higher saturation power.

5. Conclusions

For coherent optical PAM4 systems, we propose the PA-DFE and MA-DFE, which are designed to compensate for CD and nonlinear impairments. Moreover, the proposed equalizer is robust to carrier phase noise, thus eliminating the need for additional carrier phase recovery operations. We built an 80 Gb/s coherent optical PAM4 system operating at a single wavelength in the C-band for demonstration and set the appropriate equalizer parameters based on the experiments. The experimental results show that the MA-DFE and PA-DFE effectively compensated for CD and nonlinear impairments, achieving a transmission over 25 km of SSMF with a sensitivity of −23 dBm, and the link budget reached 27 dB. Additionally, we explored the performance of the equalizers under different operating states of the SOA at the transmitter. The results show that in mitigating signal distortion caused by SOA gain saturation, the PA-DFE exhibits superior equalization performance compared to the MA-DFE and ultimately achieves a link budget of 29 dB.