Advanced Neural Network-Based Equalization in Intensity-Modulated Direct-Detection Optical Systems: Current Status and Future Trends

Abstract

Intensity-modulated direct-detection (IM/DD) optical systems are most widely employed in short-reach optical interconnects due to their simple structure and cost-effectiveness. However, IM/DD systems face mixed linear and nonlinear channel impairments, mainly induced by the combination of square-law detection and chromatic dispersion, as well as the utilization of low-cost non-ideal transceivers. To solve this issue, recent years have witnessed a growing trend of introducing machine learning technologies such as neural networks (NNs) into IM/DD systems for channel equalization. NNs usually present better system performance than traditional approaches, and various types of NNs have been investigated. Despite the excellent system performance, the associated high computational complexity is a major drawback that hinders the practical application of NN-based equalizers. This paper focuses on the performance and complexity trade-off of NNs employed in IM/DD systems, presenting a systematic review of the current status of NN-based equalizers as well as a number of effective complexity reduction approaches. The future trends of leveraging advanced NN in IM/DD links are also discussed.

1. Introduction

With the exponential growth of Internet Protocol (IP) traffic, there is an ever-increasing demand for capacity in data centers. According to International Data Corporation, global data center traffic will reach 175 Zettabytes (ZB) per year by the end of 2025, up from 33 ZB per year in 2018. Inspired by this dramatic demand for capacity, data centers have become a hot topic for both academia and industry, which drive the research on short-reach optical fiber interconnects within the data centers [1,2,3,4,5,6,7,8,9]. Compared to coherent detection, intensity-modulated direct-detection (IM/DD) optical links are ideal for such short-reach applications due to their cost-effectiveness and simple structure [10,11,12,13,14,15,16,17]. However, the intensity-only direct-detection, or the simple square-law detection of the optical field, creates a nonlinear channel when combined with channel chromatic dispersion (CD). Additionally, to maintain low costs, bandwidth-limited transceivers and inexpensive lasers, such as directly modulated lasers (DMLs), are preferred, which present non-ideal frequency responses and chirp impairments [18,19,20,21,22]. These mixed linear and nonlinear impairments can significantly degrade bit error rate (BER) performance and limit the system’s achievable capacity. Therefore, effective nonlinear equalization techniques are crucial to ensure the desired system BER.

Traditional digital signal processing (DSP) approaches such as decision feedback equalization (DFE) and Volterra series-based equalization, are decades old. They have been widely applied in IM/DD systems to deal with the nonlinear impairments [23,24,25,26,27,28,29]. In recent years, advances in machine learning (ML) [30,31,32,33,34,35,36,37,38,39] have led to the introduction and growing popularity of numerous ML algorithms in the field of optical fiber communication. These algorithms have found applications across various aspects of optical communications, such as optical performance monitoring [40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60] and channel equalization for different types of optical systems [61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87]. For IM/DD channel equalization, ML algorithms, especially neural networks (NNs), have been found superior to traditional approaches in terms of system performance. Due to the introduction of different nonlinear activation functions and the layered DSP design, NNs are extremely suitable to solve nonlinear problems. Among the broad topic of applying ML in optical communications, this paper specifically focuses on leveraging NN for nonlinear equalization in short-reach IM/DD systems. Different types of NNs and their variants are presented targeting at improved system performance.

While introducing the cutting-edge NNs trying to explore better system performance, it is also important to pay special attention to the computational complexity (CC) [88,89,90]. Complicated NN equalizers with increased CC can lead to higher latency and greater power consumption in the receiver, which hinders their practical implementation. CC is particularly relevant for NN-based equalization, where it impacts both the training and equalization (inference) processes. The training process for NNs typically requires a substantial number of training symbols and epochs. When the link scenario changes, the performance of the previously trained NNs may degrade, necessitating retraining to adapt to the new conditions, which is computationally inefficient. During the equalization process, the computational load is significant as well, with the number of multiplications per equalized symbol needing to be limited to a few tens to enable real-time DSP implementation [91,92,93]. Given these considerations, it is highly desirable to reduce CC in both NN training and the equalization processes. We can make a trade-off between the performance and CC of NN-based equalizers according to different link requirements.

In this paper, we provide a systematic review of the application of NN for equalization in short-reach IM/DD optical links, taking both system performance and CC into account. The remainder of this paper is organized as follows. Section 2 provides the introduction and the mathematical model of typical IM/DD systems, discussing the benefits and bottlenecks. Section 3 presents different performance-oriented advanced NN-based equalization structures, providing a comprehensive summary of existing works. Section 4 gives a detailed overview of a number of techniques effectively addressing both training and equalization CC of NN-based equalizers. Finally, Section 5 concludes this paper and discusses future perspectives.

2. Short-Reach IM/DD Systems

2.1. IM/DD System Structure

This paper discusses the traditional double-sideband (DSB) IM/DD systems which possess the simplest structure among various designs of optical transmission systems. A general illustration of a typical IM/DD communication system as well as the associated DSP processes are depicted in Figure 1. In this system, a laser serves as the light source, and the transmitted electrical signal is directly modulated onto the optical intensity. Various types of laser/modulator modules can be employed at the transmitter, including DML; a vertical-cavity surface-emitting laser (VCSEL); an electro-absorption modulated laser (EML), which consists of a laser combined with a separate electro-absorption modulator (EAM); a laser combined with a Mach-Zehnder Modulator (MZM); and other advanced silicon photonic modulators. Pulse amplitude modulation (PAM) formats with different levels are usually adopted for intensity-only optical transmission, such as PAM-2, PAM-4, and PAM-8. At the transmitter DSP, the PAM signals are generated and passed through a root raised cosine (RRC) filter for pulse shaping. The signals are sampled at a proper sampling rate before being sent out for transmission. The choice of fiber can vary based on the transmitter type. A single-mode fiber (SMF) is commonly used for most of the transmitters, while a multi-mode fiber (MMF) is selected for systems utilizing a VCSEL-based transmitter.

At the receiver, as shown in Figure 1, only a single-ended photo-detector (PD) is employed to convert the optical signal into electrical power. Unlike coherent detection, which could preserve both amplitude and phase of the signal, the simple square-law direct-detection can only preserve the amplitude information, and that is why PAM is usually employed for IM/DD systems. The received electrical signal is further processed by a series of commonly employed DSP procedures such as resampling, synchronization, and matched filtering. The signals are then fed into the NN-based equalization module for nonlinearity mitigation. Finally, hard-decision is performed and the system BER is calculated.

2.2. IM/DD System Model

Most IM/DD systems face intrinsic nonlinearity problem when performing square-law detection over signals affected by a dispersive channel [94]. When the IM/DD system is not operated at zero-dispersion wavelength, the CD effects is not negligible. The frequency response of CD can be expressed by

$$H(\omega )=e^{j\frac{1}{2}{\beta }_2{\omega }^{2}L},$$

(1)

where ${\beta }_2$ is the group velocity dispersion coefficient, $L$ denotes the fiber length, and $\omega$ denotes the signal angular frequency. Assuming an ideal transmitter is employed, after intensity-modulation, the output optical power of the transmitter laser, denoted by $P_{Tx}(t)$, is given by

$$P_{Tx}(t)=\eta \left(S_0+S_{Tx}(t)\right),$$

(2)

where $S_{Tx}(t)$ represents the transmitted electrical signal, $S_0$ denotes the bias current, and $\eta$ denotes the modulation coefficient. If we omit the phase impact, the optical field of the transmitter laser, denoted by $E_{Tx}(t)$, can be written as

$$E_{Tx}(t)=\sqrt{P_{Tx}(t)}=\sqrt{\eta \left(S_0+S_{Tx}(t)\right)}=\sqrt{\eta S_0}\sqrt{1+{\displaystyle \frac{S_{Tx}(t)}{S_0}}}.$$

(3)

Note that the bias current $S_0$ normally needs to be large enough to make the signal located at the linear modulation range of lasers. As such, we can perform Taylor series expansion over $E_{Tx}(t)$, and $E_{Tx}(t)$ can be rewritten as

$$E_{Tx}(t)=\sqrt{\eta S_0}\left(1+{\displaystyle \sum _{n=1}^{\infty }{c}_n}{\left({\displaystyle \frac{S_{Tx}(t)}{S_0}}\right)}^n\right),$$

(4)

where the Taylor expansion coefficients are calculated by

$$c_n={\displaystyle \frac{{(-1)}^{n-1}(2n)!}{2^{2n}{(n!)}^2(2n-1)}}.$$

(5)

After transmission through the optical fiber channel, the received optical field, denoted by $E_{Rx}(t)$, is modeled by the convolution of the transmitted optical field $E_{Tx}(t)$ and the CD response in time domain denoted by $h(t)$, which is shown as

$$\begin{array}{c}{E}_{Rx}(t)=E_{Tx}(t)\otimes h(t)\\ =\sqrt{\eta S_0}\left(1+{\displaystyle \sum _{n=1}^{\infty }{c}_n}{\left({\displaystyle \frac{S_{Tx}(t)}{S_0}}\right)}^n\right)\otimes h_R(t)+j\sqrt{\eta S_0}\left(1+{\displaystyle \sum _{n=1}^{\infty }{c}_n}{\left({\displaystyle \frac{S_{Tx}(t)}{S_0}}\right)}^n\right)\otimes h_I(t)\end{array},$$

(6)

where $h_R(t)$ and $h_I(t)$ represents the real and imaginary part of $h(t)$, and $\otimes$ represents the convolution operation. Equation (6) can be simplified by calculating $1\otimes h_R(t)$ and $1\otimes h_I(t)$ based on Equation (1), where the simplified version is written by

$$E_{Rx}(t)=\sqrt{\eta S_0}\left(1+{\displaystyle \sum _{n=1}^{\infty }{c}_n}{\left({\displaystyle \frac{S_{Tx}(t)}{P_0}}\right)}^n\otimes h_R(t)\right)+j\sqrt{\eta S_0}{\displaystyle \sum _{n=1}^{\infty }{c}_n}{\left({\displaystyle \frac{S_{Tx}(t)}{P_0}}\right)}^n\otimes h_I(t).$$

(7)

The received square-law detected electrical signal, denoted by $S_{Rx}(t)$, is shown as

$$S_{Rx}(t)=R{\left|E_{Rx}(t)\right|}^2,$$

(8)

where $R$ represents the responsivity of the PD. With simple mathematical derivation, $S_{Rx}(t)$ can be expanded and written as (note that $c_1={\displaystyle \frac{1}{2}}$)

$$\begin{array}{ll}{S}_{Rx}(t)=\hfill & R\eta S_0+R\eta S_{Tx}(t)\otimes h_R(t)+2R\eta S_0{\displaystyle \sum _{n=2}^{\infty }{c}_n}{\left({\displaystyle \frac{S_{Tx}(t)}{S_0}}\right)}^n\otimes h_R(t)+\hfill \\ & R\eta S_0\left[{\left({\displaystyle \sum _{n=1}^{\infty }{c}_n}{\left({\displaystyle \frac{S_{Tx}(t)}{S_0}}\right)}^n\otimes h_R(t)\right)}^2+{\left({\displaystyle \sum _{n=1}^{\infty }{c}_n}{\left({\displaystyle \frac{S_{Tx}(t)}{S_0}}\right)}^n\otimes h_I(t)\right)}^2\right]\hfill \end{array}.$$

(9)

As shown in Equation (9), the received signal $S_{Rx}(t)$ is separated into four parts. The first term denotes the direct current, which is constant and can be easily removed. The second term is a linear convolution of the transmitted signal $S_{Tx}(t)$ and the real part of time-domain CD response $h_R(t)$. This is known as the power fading effect, where the IM/DD signals suffer from destructive frequencies especially when the data rate and fiber length increase. The third term is the convolution of the high order signal term with the real part of time-domain CD response $h_R(t)$, while the fourth term shows the signal-to-signal beating interference (SSBI). The first two terms are linear, while the last two terms show nonlinear impacts. Even in the ideal case, we find that IM/DD channel is intrinsically nonlinear. In practical applications, the laser, modulator, and PD can introduce more severe nonlinear impairments. The mixed linear and nonlinear impairments significantly degrade system performance, which necessitate advanced equalization methods such as powerful NNs.

3. Performance-Oriented NN-Based Equalizers

3.1. FNN-Based Equalizer

The NN-based equalizers for IM/DD links are first investigated targeting at improved system BER performance. As the simplest form of NN, feedforward NNs (FNNs) are widely employed for equalization in IM/DD systems [95,96,97,98,99,100,101,102,103,104,105,106]. A typical two-layer FNN equalization structure is depicted in Figure 2. Assuming $n^{[0]}$ inputs and $n^{[1]}$ hidden neurons are employed for the FNN, the DSP process is operated by

$$y=f^{[2]}\left(W^{[2]}{f}^{[1]}(W^{[1]}x+b^{[1]})+b^{[2]}\right),$$

(10)

where $x\in {ℝ}^{n^{[0]}}$ and $y$ represents the inputs and output of FNN, $W^{[1]}\in {ℝ}^{n^{[1]}\times n^{[0]}}$/$b^{[1]}\in {ℝ}^{n^{[1]}}$/$f^{[1]}$ and $W^{[2]}\in {ℝ}^{n^{[2]}\times n^{[1]}}$/$b^{[2]}$/$f^{[2]}$ denotes the weights/biases/activation functions of the hidden and the output layer. The NN is operated in a sliding-window manner, which predicts the received symbol sequentially. Different NN parameters can be selected and optimized to yield different system performance.

The first application introducing FNN into IM/DD systems is observed in [95], where a simple two-layer FNN is deployed to infer simultaneously the linear and non-linear channel response. The NN has four outputs, each corresponding to one level of the PAM-4 signal and the entire NN function as a classifier. With the help of the FNN, a 168-Gb/s PAM-4 signal is successfully transmitted over 1.5-km SMF, achieving up to 10 times BER reduction over conventional FFE. In [96], FNN is implemented to attain a 64-Gb/s PAM4 4-km MMF link employing 850-nm VCSEL. FNN outperforms 3rd order Volterra series in their VCSEL-based IM/DD setup. Recorded high 256-Gb/s∙km data-rate distance product is achieved supported by FNN-based equalization. In [97], FNN is used for nonlinear equalization in IM/DD passive optical network (PON) scenarios. With the aid of FNN, 50-Gb/s PAM4 IM/DD PON transmission via 20 km SMF is realized using 10-GHz class optical devices, where the end-to-end 3-dB bandwidth is only 3.6 GHz. FNN shows its superiority over conventional approaches, and shows its effectiveness in resolving bandwidth problems. In [98], a DML-based IM/DD link is shown using FNN at the receiver. A 20-Gb/s 18-km O-band PAM4 transmission is realized, where the FNN nonlinear equalizer is found to increase the channel capacity and drastically reduce the impact of nonlinear penalties. In [99], the authors extend their [98] and increase the data rate to 54 Gb/s. Different modulation formats are used, where the FNN-based equalizers work well for all the cases. FNN is adopted in wavelength division multiplexing (WDM) IM/DD links in [100], where 4 × 50-Gb/s PAM4 signal is transmitted over 80-km SMF. In this work, a dispersion compensation fiber (DCF) is used to pre-compensate the CD impacts. FNN shows about 2 dB power sensitivity improvement over conventional nonlinear DSP methods.

The demonstration of FNN in IM/DD systems has not gone away in years. More recently, FNN is applied in 137-Gb/s PAM4 link using 25-GHz class 850-nm optical devices [101]. The signal is transmitted over an in-house fabricated 40 cm optical backplane. The 112-Gbps 100-m VCSEL-MMF optical interconnects are demonstrated in [102], and the FNN achieve more than one order of magnitude BER improvement compared with Volterra series in such system. Similar as [97], a 50-Gb/s 20-km link is shown in [103] using bandwidth-limited transceivers. Under bandwidth constraints, the FNN-based equalizer again presents superior performance. IM/DD link using the simple OOK modulation format is shown in [104], where a 50-Gb/s OOK signal is transmitted over 30-km SMF. The FNN is also successfully demonstrated in real-time field-programmable gate arrays (FPGAs) in this work. The IM/DD link data rate is increased to as high as 160 Gb/s in [105], employing a GeSi EAM. The highest single-wavelength PAM4 data rate is achieved based on a single EAM, supported by FNN-based nonlinear mitigation. A more generalizable FNN-based equalizer is shown in [106], where a 56-Gb/s PAM4 signal is transmitted over 20/30/40-km SMFs using the proposed FNN. All the above works prove that FNN is effective in mitigating the channel impairments of short-reach IM/DD systems.

3.2. CNN-Based Equalizer

Following the introduction of FNN, more powerful NNs are employed for equalization in IM/DD systems. Convolutional NNs (CNNs) are employed to explore deeper into the system performance in [107,108,109,110,111]. CNN is a regularized type of FNN that learns feature engineering by filter optimization with the help of convolutional and pooling layers, which are widely used for image classification tasks. The schematic of a CNN-based equalizer is illustrated in Figure 3. Considering only one-dimension data (which is the case of signal processing for channel equalization), assuming the input, filter, and output of the convolutional layer are represented by $x$, $f$, and $y$, the convolution operation is expressed as

$$y_i={\displaystyle \sum _{k=1}^L{x}_{i+k-1}{f}_k},$$

(11)

where $L$ denotes the filter length. The pooling operation reduces the number of data by combining the outputs of neuron clusters at one layer into a single neuron in the next layer. Max pooling and average pooling are the most commonly used, which take either the maximum or the average value of each local cluster of neurons in the feature map. A typical CNN consists of many stacks of convolutional and pooling layers, where each stack represents one feature of the input data. The features are collected and fed into fully connected layers same as the FNN to give the final outputs.

CNN is applied for equalization in a 112-Gb/s 40-km PAM4 optical link using EML in [107]. The CNN has one input layer, three convolutional layers, two fully connected layers, and one output layer for classification of PAM signals. It has been shown that the performance of the proposed CNN model outperforms Volterra series and FNN equalizers. In [108,109,110], the same group use different types of CNN for equalization in different IM/DD systems. The system is varied in modulation formats (PAM-4, PAM-8, PAM-16), transmission bands (C-band, O-band), data rates (56 Gb/s, 100 Gb/s) and system bandwidth (10-G class, 20-G class). The CNN is changed with different number of convolutional layers and the number of neurons in each layer. All the different demonstrations show strong equalization ability of CNN. A temporal CNN (TCNN) is proposed in [111], which introduce dilated convolutions and residual connections onto the traditional CNN. Better system performance is observed compared with traditional CNN equalization architecture, and the proposed scheme enables as far as 100-km SMF IM/DD transmission of a 56-Gb/s PAM4 signal.

3.3. RNN-Based Equalizer

Although CNN presents better performance, it also requires much larger network structures, which is deemed too complex for real-time application. The investigation on CNN-based equalization seems to have disappeared in recent years, and researchers focus more on recurrent NN (RNN)-based equalization [112,113,114,115,116,117,118,119,120,121,122,123,124]. Four types of RNNs are found in IM/DD applications, which are auto-regressive RNN (AR-RNN), layer-recurrent NN (L-RNN), long short-term memory (LSTM) and gate recurrent unit (GRU) networks. These RNNs are built on top of the traditional FNN.

The schematic of a two-layer AR-RNN-based equalizer is shown in Figure 4. On top of the FNN, a few output delays are sent back to the hidden layer and serve as new inputs. Assuming the number of output delays used in the feedback loop is denoted by $k$, and the output delays and the associated weights are represented by $y_d$ and $W^d\in {ℝ}^{n^{[1]}\times k}$, the DSP process is given by

$$y=f^{[2]}\left(W^{[2]}{f}^{[1]}\left(\left[W^{[1]},W^d\right]{\left[x^T,y_d^T\right]}^T+b^{[1]}\right)+b^{[2]}\right).$$

(12)

In terms of the equalization process, the operation of using past predicted output symbols as additional inputs provides more information when predicting the current output symbol. As such, better performance can normally be achieved with the help of this information. AR-RNN is first used for equalization in IM/DD systems in [112]. The 50-Gb/s PAM-2 and 100-Gb/s PAM-4 signals are transmitted over 20-km SMF, where the receiver adopt AR-RNN with seven feedbacks to read the historical decision results. In [113,114], the AR-RNN is implemented using FPGAs in a parallel manner, and a 100-Gb/s IM/DD PON system is demonstrated. It is shown that the AR-RNN can beat FNN equalizers with the same input/output size and the number of training parameters, achieving better receiver sensitivity performance.

The structure of L-RNN is depicted in Figure 5. Different from AR-RNN, which uses output feedbacks, L-RNN collects the delays from the outputs of hidden neurons and sends them back to the hidden layer again for data processing. Assuming the number of rounds of hidden layer delays used in the feedback loop is denoted by $k$, and the hidden layer delays and the associated weights are represented by $h_d$ and $W^h\in {ℝ}^{n^{[1]}\times k{n}^{[1]}}$, the DSP process of L-RNN is given by

$$y=f^{[2]}\left(W^{[2]}{f}^{[1]}\left(\left[W^{[1]},W^h\right]{\left[x^T,h_d^T\right]}^T+b^{[1]}\right)+b^{[2]}\right).$$

(13)

Similar to AR-RNN, additional useful information about former predictions is also provided in L-RNN when predicting the current symbol. An L-RNN-based equalizer is proposed for equalization in a VCSEL-MMF optical interconnect in [115]. It has been shown that L-RNN is more powerful than ANN in dealing with sequential signals, and has the potential of reaching much lower BER with similar complexity. In [116], the authors extend their work in [115] by employing hidden feature extraction before sequence training. The input features are first extracted using principal component analysis or other dimensionality reduction approaches before sending into the L-RNN equalizer. Aided by the feature-enhanced L-RNN, single-lane 288-Gb/s PAM-8 signal transmission over 100-m MMF is realized with BER well below the 20% SD-FEC threshold.

The architecture of LSTM and GRU networks are given in Figure 6. Compared with traditional FNN, an LSTM/GRU layer is added, where inside contains a number of LSTM/GRU cells. Both LSTM and GRU address the vanishing gradient problem in traditional RNNs by introducing gating mechanisms that allow them to capture long-term dependencies more effectively. The detailed complicated LSTM/GRU cell structure will not be discussed in this paper. Interested readers can refer to [117,118,119,120,121,122,123,124] for more information.

LSTM is used for equalization in both classification and regression manners in [117]. Both cases work well for a 50-Gb/s 100-km PAM4 optical system. In [118,119], a 160-Gb/s 1-km PAM4 link is conducted using a silicon-microring-modulator (Si-MRM) and an LSTM-based equalizer. Two LSTM layers and two fully connected layers are employed. The nonlinearity induced by the modulator is effectively mitigated by the proposed powerful equalizer. In [120], the authors extend their work in [118,119] with updated experimental configuration. The LSTM now supports 270-Gb/s PAM-8 signal to transmit 1-km SMF using the Si-MRM, which greatly increases the achievable data rate. LSTMs are also employed in [121,122] to achieve 200+ Gb/s per single lane. Note that non-zero dispersion-shifted fiber (NZDSF) is used to reduce the impact of CD. In addition to LSTM, the performance of GRU is also tested in [122], where it achieves slightly higher BER than LSTM. More works on GRU-based equalization can be found in [123,124], where the GRU is proposed to resolve the patterning effect of the semiconductor optical amplifiers (SOAs) applied in IM/DD systems. The input power dynamic range of SOA can be greatly extended with the help of the GRU-based equalization.

3.4. Cascade NN-Based Equalizer

The introduction of different recurrent structures of RNN significantly improves the system performance. However, the training and equalization complexity also increase, especially for the LSTM and GRU ones. Another variant of FNN is the cascade NN, which is computationally friendly. The structure of cascade FNN is shown in Figure 7. On top of the traditional FNN, cascade connections are included, which connect the input and every previous layer to the following layers. For a two-layer cascade FNN, the input layer is simply connected to the output layer. Assuming the cascaded weights and are represented by $W^c\in {ℝ}^{n^{[0]}+k}$, the equalization process of cascade FNN is given by

$$y=f^{[2]}\left(\left[W^{[2]},W^c\right]{\left[{\left(f^{[1]}\left(W^{[1]}x+b^{[1]}\right)\right)}^T,x^T\right]}^T+b^{[2]}\right).$$

(14)

The cascade connections produce a pure linear path for direct mapping of the inputs to the output. This enables an efficient joint linear and nonlinear effect estimation and results in better system performance when used for equalization. Both cascade FNN and cascade RNN are proposed in [125,126]. A 100-Gb/s 15-km PAM4 link is built using a band-limited DML, where the cascade structure help improves the receiver sensitivity by 1 dB compared with NNs without cascade connections. It is also demonstrated that cascade NN-based equalizers have a much faster training speed. A more recent work is found in [127], where the cascade structure is shown as “skip connections”. The experimental setup is similar as used in [123,124], where the NN performs well with skip connections. The effect of simplified training is also verified in this work.

3.5. Other Types of NN-Based Equalizers

In addition to the above-mentioned NNs, there are also many different type of NN-based equalizers demonstrated in IM/DD systems. Radial basis function NN (RBF-NN) is shown in [128] in a 4 × 50-Gb 80-km PAM-4 IM/DD link. The RBF-NN employs Gaussian activation function in the hidden layer, and achieves better network stability and fitting ability compared with traditional Volterra series or FNN. There are many discussions on the application of spiking NN (SNN) in IM/DD systems recently, as shown in [129,130,131,132,133]. In addition to neuronal and synaptic state used in traditional NNs, SNNs incorporate the concept of time into their operating model. The SNN-based equalizers have been implemented in application-specific integrated circuits (ASICs), and have been verified in both simulation and experiments. Interested readers can refer to [129,130,131,132,133] for more details about SNN-based equalization in IM/DD systems. The different types of NN-based equalizers and IM/DD links are summarized and shown in

Table 1. Different types of NN-based equalization for various short-reach IM/DD links.

NN Type	Ref.	Modulation	Data Rate	Fiber Length	Tx Type	Wavelength
FNN	[95]	PAM4	168 Gb/s	1.5 km SMF	MZM (35 GHz)	~1550 nm
	[96]	PAM4	64 Gb/s	4 km MMF	VCSEL (25 GHz)	~850 nm
	[97]	PAM4	50 Gb/s	20 km SMF	MZM (10 GHz)	~1550 nm
	[98]	PAM4	20 Gb/s	18 km SMF	DML (10 GHz)	~1310 nm
	[99]	PAM2/PAM4/PAM8	54 Gb/s	25 km SMF	DML (10 GHz)	~1550 nm
	[100]	PAM4	4 × 50 Gb/s	80 km SMF	DML (20 GHz)	~1550 nm
	[101]	PAM4	137 Gb/s	40 cm MMF	MZM (25 GHz)	~850 nm
	[102]	PAM4	112 Gb/s	100 m MMF	VCSEL (NA)	~850 nm
	[103]	PAM4	50 Gb/s	20 km SMF	DML (10 GHz)	~1310 nm
	[104]	PAM2	50 Gb/s	30 km SMF	MZM (35 GHz)	~1310 nm
	[105]	PAM4	160 Gb/s	2 km SMF	GeSi EAM (30 GHz)	~1550 nm
	[106]	PAM4	56 Gb/s	20/30/40 km SMF	MZM (40 GHz)	~1550 nm
CNN	[107]	PAM4	112 Gb/s	40 km SMF	EML (25 GHz)	~1310 nm
	[108]	PAM4	56 Gb/s	25 km SMF	DML (10 GHz)	~1310 nm
	[109,110]	PAM8/PAM16	100 Gb/s	25 km SMF	DML (20 GHz)	~1310/1550 nm
	[111]	PAM4	56 Gb/s	100 km SMF	MZM (40 GHz)	~1550 nm
RNN	[112]	PAM2/PAM4	60/100 Gb/s	20 km SMF	MZM (40 GHz)	~1550 nm
	[113,114]	PAM4	100 Gb/s	20 km SMF	MZM (NA)	~1310 nm
	[115]	PAM4	56 Gb/s	100 m MMF	VCSEL (18 GHz)	~850 nm
	[116]	PAM8	288 Gb/s	100 m MMF	VCSEL (23 GHz)	~850 nm
	[117]	PAM4	50 Gb/s	100 km SMF	DML (18 GHz)	~1550 nm
	[118,119]	PAM4	160 Gb/s	1 km SMF	Si MRM (47 GHz)	~1550 nm
	[120]	PAM8	270 Gb/s	1 km SMF	Si MRM (55 GHz)	~1550 nm
	[121,122]	PAM4	212 Gb/s	1 km NZDSF	EML (40 GHz)	~1550 nm
	[123,124]	PAM4	100 Gb/s	5.4 km SMF	MZM (NA)	~1550 nm
Cascade NN	[125,126]	PAM4	50/100 Gb/s	25/15 km SMF	DML (16 GHz)	~1550 nm
	[127]	PAM4	100 Gb/s	4.8 km SMF	MZM (33 GHz)	~1550 nm
RBF-NN	[128]	PAM4	4 × 50 Gb/s	80 km SMF	DML (18 GHz)	~1550 nm
SNN	[129,130]	PAM4	224 Gb/s	4 km SMF	NA	~1270 nm
	[131,132]	PAM4	100 Gb/s	2 km SMF	NA	~1310 nm
	[133]	PAM4	200 Gb/s	5 km SMF	NA	~1270 nm

.

Besides the direct utilization of NN for equalization, NNs are often combined with sequence decoders to achieve better system performance. Maximum likelihood sequence estimation (MLSE) based on NN is proposed in [134], where the NN is used to estimate the nonlinear channel responses and to calculate the metrics for the Viterbi algorithm. Similarly, an NN-BCJR equalization scheme is proposed in [135], where an NN-based nonlinear channel emulator is adopted to calculate the transition metric in the BCJR algorithm. In [136,137], duobinary training strategy is proposed. The NN equalizer is first trained targeting at the duobinary form of the signal, and MLSE is followed to recover the enforced ISI. This approach is particularly effective in addressing the bandwidth limitations problems in IM/DD systems. In [138], the NN equalizer is trained targeting at adaptive duobinary form of the signal. The optimal partial-response parameter is learned through NN training, where the system performance can be further improved compared with equalization with conventional partial-response target.

3.6. Performance and Complexity Comparison of FNN-, L-RNN-, Cascade FNN-, and AR-RNN-Based Equalizers

An IM/DD experiment is conducted to verify the performance and complexity of above-mentioned NN-based equalizers. Here we only show the results of four types, i.e., FNN-, L-RNN-, cascade FNN-, and AR-RNN-based equalizers, since other types such as CNN- or LSTM/GRU-based ones are considered much more complex, which makes them difficult to be applied in real-time applications. Interested readers can refer to [95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133] for more details on the performance of different types of NN-based equalizers. The IM/DD experiment is based on a DML with a 3-dB bandwidth around 16 GHz, where a 50-Gb/s PAM4 signal is generated and transmitted over 25-km SMF [125,126]. A variable optical attenuator (VOA) is applied at the receiver to tune the received optical power (ROP). The NNs only have two layers, where tanh activation function is selected for the hidden layer and linear activation function is used for the output layer. The NNs are used in a regression manner which means that only one output is adopted. A total of 20,000 random PAM4 symbols are used for training the NNs, while an additional 1.2 million PAM4 symbols are collected for NN-based nonlinear equalization and BER calculation.

We first show the best system performance of each NN-based equalizers, where the complexity constraint is omitted. As many as 15 inputs and nine hidden neurons are selected, which can guarantee that the NNs achieve their best performance. The BER-ROP curves of different NN-based equalizers are shown in Figure 8a. We use the form (the number of inputs, the number of hidden neurons) to represent the size of the different NNs, as shown in the figure. It can be observed that the performance of NNs follows the order of AR-RNN, cascade FNN (denoted by C-FNN in the figure), L-RNN, and FNN. Compared with traditional FNN, the three FNN variants all improve the system performance. The receiver sensitivity is improved by approximately 2/1/0.5 dB by AR-RNN/cascade FNN/L-RNN. Figure 8b illustrates the number of multiplications (denoted by $N_{mul}$) of all the NNs adopted in Figure 8a. Considering the same number of inputs and hidden neurons, L-RNN is obviously more complicated than the other employed NNs. Cascade FNN and AR-RNN, however, show limited additional complexity compared with traditional FNN.

Figure 9a depicts the system BER performance of the NN-based equalizers under the complexity constraint, where the $N_{mul}$ of all the NNs are all kept below 100, shown in Figure 9b. The number of inputs and hidden neurons of the different types of NNs are carefully chosen to achieve the best system performance with only a few tens of multiplications involved to recover one symbol, showing the potential for real-time implementation. When the $N_{mul}$ of NNs are lower than 100, L-RNN becomes the worst equalizer since its size is affected most by the complexity constraint. Cascade FNN and AR-RNN, however, still present superior BER performance over FNN, increasing the receiver sensitivity by about 1.5 and 2 dB, respectively.

3.7. Possible Pitfalls When Applying NN-Based Equalizers

One thing we need to pay special attention to when using NNs is the so-called possible pitfalls or overestimation traps [139,140,141,142,143]. It has been observed that NNs are capable of learning the operational logic of pseudo-random bit sequences (PRBSs). This ability may lead to an overestimation of the NNs’ performance, as the performance improvements might stem from predicting the sequence patterns rather than from mitigating channel impairments. Such overestimation is not limited to PRBS but can also occur with other data types that follow specific patterns, including short repeated sequences.

To mitigate this overestimation problem, several approaches can be adopted. First is employing pure random data, i.e., true random numbers generated through unpredictable physical processes, to ensure that each transmitted symbol is independent. This method prevents NNs from learning any underlying patterns. Second is using different mixtures of PRBSs with varying orders to train the NNs. This can also prevent the recognition of consistent patterns across the combined sequences. Lastly, ensuring that the number of NN inputs does not exceed the PRBS order can naturally address the issue. For PRBS transmission, the NNs require at least as many inputs as the PRBS order to fully grasp the PRBS operational logic. In the context of equalization in small-scale optical transmission systems such as IM/DD links, where only a limited number of NN inputs are needed for equalization, one viable strategy is to use sufficiently long PRBSs.

4. Computationally Efficient NN-Based Equalizers

The NNs indeed greatly exploit the performance of IM/DD system. However, it is also obvious that the CC is largely increased, which makes NN receivers less practicable for real-time implementation. Much progress has been made on resolving the complexity issue when applying NN for IM/DD equalization. This section will review all the techniques, focusing on both NN training and equalization.

4.1. Transfer Learning

The training process of NN-based equalizers is usually time-consuming, which involves many iterations of forward- and backward-propagation calculations. When there are many optical links needed for equalization, the training of different NN-based equalizers becomes a big problem. Transfer learning is proposed to speed up the NN training process [144]. Transfer learning is a machine learning strategy that involves repurposing a model designed for one task to serve as the foundation for a different, but related, task. This method capitalizes on the insights gained from the initial task and applies them to a new challenge. It is especially advantageous when the new task has a limited amount of labeled data, as it enables the model to utilize the extensive data and computational resources already invested in training the original model. Transfer learning has been introduced into optical communications for optical performance monitoring [145,146,147], and for equalization of coherent or single-sideband (SSB) signals [148,149,150,151]. It is first introduced for equalization in IM/DD systems in [152,153], where the flow diagram is given in Figure 10. For NN training of the target IM/DD system, we can leverage the NN trained from different source IM/DD systems and use transfer learning. Since the source NN-based equalizers preserve channel information that are related to the target system, they can serve as a better starting point for NN training in the target system, instead of training purely from scratch. Transfer learning-aided fast equalization is demonstrated for both FNN and RNN-based equalizers in a 50-Gb/s 20-km PAM-4 target IM/DD system [152,153]. The target system equalization is accelerated by adopting NNs from a number of source systems with different data rates and fiber lengths. The 60-Gb/s 15-km source system is found closest to the target one, where significant reduction of 90%/87.5% in training epochs and 62.5%/53.8% in training symbols are achieved. The study also reveals that FNNs can be smoothly transferred to RNNs for equalization in the target system, whereas the reverse adaptation is not practical.

In addition to FNN- and RNN-based equalizers, transfer learning can be smoothly applied for CNN-based equalization in IM/DD systems, as shown in [154]. Similar to [152,153], source systems with varying data rates and fiber lengths are employed, and transfer learning again shows its effectiveness in reducing the number of training epochs and the size of the training dataset. The iterative pruning technique is introduced into the transfer learning-aided equalization for IM/DD links in [155,156], where the convergence speed can be further enhanced during TL between the source and target links. By fine-tuning the pruning parameters, an optimal balance between performance stability and complexity can be attained. Transfer learning is set to be pivotal in the advancement of optical-switched data center networks, where the dynamic reconfiguration of optical link parameters is crucial. Utilizing transferred NN receivers, new optical interconnects can be rapidly deployed.

4.2. Pruning

When the optical links are fixed and do not change dynamically, the training complexity of NNs can be omitted since the well-trained equalizer can be stably used without the need for retraining. The equalization complexity becomes the primary concerns. Pruning is one possible technique to reduce the size and equalization complexity of an NN by removing less important weights. As shown in Figure 11, after pruning, a sparse NN structure is presented compared with its fully connected counterpart. This process helps in making the model more efficient, often leading to faster inference times and reduced computational resources, without significantly compromising the model’s performance. Pruning can also enhance generalization by preventing overfitting and is particularly useful for deploying models on resource-constrained devices. The pruning techniques have been applied in Volterra-series-based equalization [157,158,159,160] for IM/DD systems, as well as in NN-based equalization for coherent [161] and SSB signals [162]. Pruning of NN-based equalizers in IM/DD systems is found in [98,163,164,165,166,167].

The pruning process include the importance assessment of all the weights in the NN-based equalizer. This is commonly performed by setting a threshold, where the weights with absolute values lower than the threshold are considered insignificant and can be pruned. In [98], the traditional pruning method is applied in a DML-based IM/DD link. It has been demonstrated that the BER curves of the pruned NN are close to that of the unpruned NN, showing the ability of pruning in reducing receiver complexity without degrading much of the system performance. In [163], an iterative pruning algorithm is proposed for NN-based equalization in VCSEL-based IM/DD links. Compared with traditional one-shot pruning, which prunes the NN only one time, the iterative pruning method prunes the NN many times. The NN can be fine-tuned accordingly, which leads to a better complexity reduction efficiency. Ref. [164] presents the real-time pruned NN in FPGAs for VCSEL-based optical interconnects. The included hardware resources are minimized by pruning, showing the potential of applying NN receivers in practical applications. In [165], pruning is applied in a different cascade RNN-based equalization structure in a DML-based IM/DD link. It is shown that the receiver complexity is largely decreased, despite the utilization of NN structures. The importance of cascade and recurrent connections are also verified in the pruning process. In [166,167], adaptive L2-regularization is introduced to facilitate pruning in EML-based optical interconnects. A two-step training scheme is proposed, where the first step involves using L2-regularization during training to encourage sparsity in weight representations, and the second step applies the traditional pruning mechanism to remove the insignificant weights. The proposed L2-regularization-aided pruning approach shows better performance compared with conventional direct pruning.

4.3. Multi-Task Learning

Multi-task learning is also an efficient technique to address the equalization complexity issue. Multi-task learning is a machine learning approach where a model is trained simultaneously on multiple related tasks, leveraging shared representations to improve performance on each task. By learning commonalities and differences among tasks, the model can lead to improved accuracy and efficiency. Considering the equalization tasks in optical transmission systems, multi-task learning is referred to as multi-symbol prediction, shown in Figure 12, where multiple symbols are recovered simultaneously rather than processed sequentially, one at a time. By dealing with multiple symbol using only one NN, a better utilization of weights and biases can be realized. The information provided by the weights and biases in the traditional single-output NN to recover the current symbol can still be useful for predicting the following symbols. Part of the NN parameters can be shared to enable a more efficient equalization structure.

NNs with multi-outputs are adopted in IM/DD links in [104,113,114,124,168], where the main purpose is to enable high throughputs. By increasing the number of NN outputs, parallel computing can be realized. At the same FPGA clock frequency, higher throughputs can be achieved while the number of employed FPGAs remains the same. For the complexity reduction purpose, multi-symbol IM/DD equalization is first proposed in [169,170], where FNN-, cascade NN-, and RNN-based multi-symbol equalizations are demonstrated. All the cases reduce the number of multiplications for one symbol recovery to about a few tens, which indicates the potential real-time implementation of NNs with multi-output selections. The work also finds that there exist an optimal number of NN outputs that reduce the computational complexity most. The multi-symbol equalization idea is then introduced into LSTM and GRU-based IM/DD equalization in [121,122], as well as reservoir computing-based equalization [171,172], where similar conclusions about complexity reduction are given. The multi-symbol equalization scheme can even be combined with pruning techniques to jointly reduce the receiver complexity [173]. Recent hardware demonstrations of multi-output NN-based equalizers further indicates their effectiveness in reducing the equalization complexity [174,175], where the chip areas are considerably saved.

4.4. Quantization

Another approach to relax the equalization complexity requirement of NN is quantization. NN quantization is a method that reduces the computational and memory demands of an NN by changing its parameters and activations from high-precision (e.g., 32-bit floating-point) to lower-precision formats (e.g., 8-bit integers). A quantized NN-based equalizer is shown in Figure 13, where bit shifters and quantizers are adopted between each layer. The quantization approach shrinks the model size and speeds up inference, enhancing efficiency for hardware deployment. Although quantization can lead to some loss of accuracy, careful tuning helps preserve the performance while significantly lowering computational costs and power usage. A few works [176,177,178,179] have already demonstrated the computationally efficient quantized NN-based equalization in coherent optical transmission systems.

For equalization in short-reach IM/DD scenarios, fixed-precision quantization of NNs is employed in [114,168], targeting FPGA implementation NN receivers. Both works quantize the floating-point (32-bit)-based NNs to integer-based ones, where only a few bits are used to represent each weight and bias. Negligible BER penalty is observed when reducing the quantized bits, which suggests the floating-point-based NNs are actually redundant in precisions. By reducing the number of bits, the floating-point calculations all change to integers, and the memory needed for NN parameter storage is drastically decreased. In [180,181], a mixed-precision quantization method is proposed for IM/DD equalization to further decrease the number of quantized bits compared with the fixed-precision counterpart. A straightforward input neuron partitioning approach is applied to determine the high- and low-precision weights. The proposed mixed-precision quantization is verified in both traditional FNN and advanced cascade RNN scenarios. In [182], joint mixed-precision quantization and pruning is proposed to squeeze out more bits in the NN-based equalizer. The connections of NN are either directly eliminated or represented by a suitable number of quantization bits through weight clustering, creating a hybrid compressed sparse network structure that computes much faster and consumes less hardware resources. The system performance can still be upheld using the pruned mixed-precision-quantized NN receivers.

5. Conclusions and Future Perspectives

This paper presents a comprehensive overview of the current status of applying NNs for equalization in short-reach IM/DD optical links, considering both system performance and complexity. Traditional FNN and a series of advanced NNs are adopted to effectively mitigate the linear and nonlinear impairments in IM/DD channels. Transfer learning, pruning, multi-task learning, and quantization approaches are introduced to make the NN-based equalizer more computationally efficient, considering both the training and equalization phases.

Future directions of NN-based equalization in short-reach IM/DD systems still focus on improving the performance and reducing the complexity. One key area of interest is the continuous exploration of more advanced and powerful NN-based equalizers to enhance system performance. One thing we need to mention is that the works present in this paper mainly consider only post-equalization for simplicity. Joint optimizations of both pre- and post-NN-based equalization, or the so-called end-to-end learning structures, may be viable solutions to further improve BER. Considering different short-reach links, it is also important to develop approaches to improve the ability of generalization for NN-based equalizers. Another area of interest lies in the development of more efficient and intelligent approaches for complexity reduction. Algorithms that enable faster NN training and equalization are vital for realizing real-time receiver implementations. Moreover, the NN-based equalizers shown in this paper are all used as black boxes, relying purely on data-driven methods. It is also important to incorporate physical interpretations into the equalization model and develop physics-informed NN receivers for short-reach IM/DD applications.