Recent Progress on Novel DSP Techniques for Mode Division Multiplexing Systems: A Review

Abstract

This paper provides an overview of latest progress on the novel advanced digital signal processing (DSP) techniques for long-haul mode division multiplexing (MDM) systems with high capacity. Space-division multiplexing (SDM) techniques have been developed for a period to increase the capacity of optical communication system by at least one order of magnitude through MDM techniques using few-mode fibers (FMFs) or multi-core multiplexing (MCM) using multi-core fibers (MCFs). The signals in MDM links are mainly impaired by the linear and nonlinear effects in FMFs, making DSP techniques become necessary to undo these impairments. In this paper, we not only review the advanced multiple-input multiple-output (MIMO) DSP techniques for compensating linear impairments in FMFs, but also enclose the state of the art of novel DSP techniques to deal with nonlinear effects. Firstly, we introduce the MIMO schemes for equalizing modal crosstalk and modal dispersion. Then, we focus on the fast tracking of time-varying (TV) channels in FMF links through frequency-domain (FD) recursive least square (RLS) algorithm. Besides, we also cover the mainstream DSP solutions for mode-dependent loss (MDL) and several possible methods to compensate nonlinearity in FMF. Moreover, artificial intelligence (AI) technologies are also discussed for its high nonlinearity tolerance and may bring a revolution in MDM systems on the process of channel equalization, link monitoring, etc. In the end, a brief conclusion and perspective will be provided.

1. Introduction

SDM has gained great attention over the last decades as a potential technology to accommodate the demand for future long-haul and power-efficient optical fiber communication systems with extremely high capacity [1,2]. In SDM transmission systems, fibers with parallel spatial transmission channels are used to carry independent data-streams in the same frequency band, which can be either uncoupled multi-core fibers (MCFs) or few- and multi-mode fibers (FMFs and MMFs) [3].

Each type of fiber for SDM systems has its own special advantages. Uncoupled MCFs are easy to be expanded from current single mode fiber (SMF) schemes [4] since each core is used as an independent spatial channel where crosstalk from other cores is well suppressed [5]. While strongly coupled MCFs have been proved with better nonlinear tolerance compared to SMFs, which makes longer transmission distance or higher transmitting power possible [6]. FMFs or MMFs on the other hand can support much high spatial channel-count in a fixed cross-section area compared with other type of fibers, using additional DSP to undo the linear, nonlinear and other impairments that recoverable after several kilometers of transmission.

The impairments over long-distance FMF systems mainly include modal dispersion (MD), linear coupling or crosstalk (XT) among different fiber modes, time-varying (TV) effect in transmission channels, mode dependent loss (MDL) and nonlinear effects.

A combination of MD and linear coupling causes visible inter-symbol interference between different modes and different time slots. Accumulated MD is also called differential mode group delay (DMGD), which means different spatial modes propagate along the fiber with different speed. While linear coupling either appears at discrete points like fiber splices and connectors or is distributed along the fiber due to bends or fabrication imperfections. Generally, linear coupling and MD are mitigated by multiple-input-multiple-output (MIMO) equalizers [7]. To decrease the high computational complexity of MIMO equalizers, frequency-domain equalization (FDE) algorithms were utilized to recover the signals that have experienced impairments in fibers [8,9,10]. The FDE methods can reduce the computational complexity dramatically, holding the same performance compared with the time-domain equalization (TDE). What’s more, when it refers to the specific updating methods of weighting matrix, the least mean squares (LMS) and recursive least squares (RLS) algorithms are two most widely adopted and studied methods [11,12,13]. Compared with LMS algorithm, faster convergence and better performance can be accomplished by applying RLS at the cost of higher complexity [14,15,16].

It should also be noticed that linear coupling between different modes can be easily influenced by a random temperature fluctuation or mechanical perturbation [17], so the transmission matrix of fiber channels is actually time-varying with a certain speed related to the perturbation rate. To achieve timely signal recovery, the equalizer on the receiver needs to re-estimate the transmission matrix more frequently at the maximum speed of the fiber channel evolution [18]. Moreover, if the speed of the fiber channel fluctuation is too fast to track for the equalizer, the quality of the recovery would be degraded, leading to decrease in transmission capacity [19]. So, the TV effect will not only increase the burden on a receiver but also degrade the quality of channel equalization. Thus, aside from the computational complexity and performance, the tracking capability or adaptation time of the equalizers also becomes important for SDM systems, which naturally means RLS-based MIMO algorithm with faster convergence is much valuable as a possible solution for TV effects.

Long-haul SDM systems are also impaired by MDL arising from the imperfections or mode-dependent difference of inline components such as fibers, amplifiers and multiplexers, who are inducing modal loss/gain disparities [20,21,22,23]. It is a fundamental limitation for a desirable data-rate or transmission distance. MDL causes the difference in optical signal to noise ratio (OSNR) degradations among fiber modes, which finally reduce the system capacity [24]. Thus, to suppress MDL effectively, specific optical devices such as mode scramblers and strongly-coupled fibers are utilized to mix different modes and balance the differences between them [25,26]. MDL-induced penalty is also found can be mitigated by advanced DSP techniques such as receiver-side maximum-likelihood (ML) detection and interference cancellation (IC) as well as transmitter-side space-time coding (STC) techniques.

Another fundamental limitation for SDM systems is Kerr-effect-caused nonlinear signal distortions. Because of the inevitable mode overlap, nonlinear effects will not only happen between different frequencies in the same eigen mode (intramodal nonlinearity), but also between different eigen modes (intermodal nonlinearity) [27,28,29]. Intermodal nonlinear coupling will further degrade the signal quality. The digital back-propagation (DBP) algorithm is the most common DSP method to compensate the fiber nonlinearities in single mode fiber. And fortunately, the extension of DBP based single-mode split-step Fourier method (SSFM) has recently been proposed to solve the multimode propagation problem in FMFs [30,31,32].

Recently, artificial intelligence (AI) technologies have also attracted great interest in optical fiber communication, for its high nonlinear representation ability. Many neural network models are provided by deep learning to handle various optical communication problems. We will also review the recent applications of AI in optical communication.

The outline of this review article is organized as follows: Section 2 sets forth the principles of MIMO equalizers, the main challenges and applications of MIMO equalizers. Section 3 introduces the significance and principle of RLS techniques in MIMO equalizers, and the tracking performance in MDM TV channels based on frequency-domain variable forgetting factor RLS algorithm (FD-VFF-RLS). In Section 4, the mainstream DSP solutions for mitigating MDL effects are reviewed. Then, Section 5 explores the novel DSP schemes to compensate nonlinear effects in MDM systems and enlightens the advanced AI techniques that could be introduced in MDM systems. A brief conclusion and some perspectives will be provided in the last section.

2. Linear DSP MIMO Technique for MDM Systems

An optical MIMO system with $N_T$ transmitting channels and $N_R$ receiving channels can be linearly modeled as Equation (1).

$$y=Hx+n$$

(1)

where $x$, $y$, $n$, and $H$ denote the $N_T \times 1$ transmitted symbol vector, the $N_R \times 1$ received symbol vector, the $N_R \times 1$ zero-mean complex noise vector, and the $N_R \times N_T$ channel transmission matrix, respectively. To reconstruct the signal $\widehat{x}$ from the received symbols $y$, MIMO processing must be used to achieve the inverse transmission matrix which can be expressed as Equation (2):

$$\widehat{x}=Wy$$

(2)

where $W$ is the weight matrix of MIMO equalizer. Zero-forcing and minimum mean square error method are usually utilized to achieve $W$ if transmission matrix $H$ is already known [33,34] Generally, there is always no priori information of MIMO channel, so adaptive filters are widely used to solve out the weight matrix in iteration process. So far, two main linear equalization schemes have been investigated in detail: adaptive TDE [35,36] and FDE [37,38].

Taking LMS algorithm as an example, LMS-FDE has significant advantage on the computational complexity compared with LMS-TDE. The computational complexity of MIMO equalizers increases with the number of finite impulse response (FIR) filters length. The transmitted signal suffers a large amount of DMGD constantly accumulating with the transmission distance, which finally leads to a large time spread or delay. Thus, a longer FIR filter will be required, which means a corresponding higher computational complexity. In order to relax the complexity of the equalizers, FDE is well adopted by utilizing fast Fourier transform (FFT) to achieve complex convolutions in a block-by-block way [37] as shown in Figure 1 [10,39].

The computational complexity for the equalization algorithm could be measured by the number of complex multiplications per mode per symbol. The complexity of TDE increases linearly with the filter length, while the FDE scales logarithmically with the taps number. The complexity except for carrier recovery for these two types of algorithms can be expressed as Equations (3) and (4) [40]:

$$C_{TDE}=3m\Delta \tau L{R}_s$$

(3)

$$C_{FDE}=\left(4+4m\right){\log }_2(2\Delta \tau L{R}_s)+8m$$

(4)

where $\Delta \tau$ is the fiber DMGD, $L$ is the fiber link length, $R_{\mathrm{s}}$ is symbol rate, $m$ is the number of channels for transmission. The computational complexities as a function of number of taps are shown in Figure 2, in which we assumed m equals 10. We could see that FDE reduce the complexity dramatically compared with TDE. For long MDM transmission link, the advantage of FDE will be more significant because the required taps number will be higher.

As the complexity of LMS-TDE and LMS-FDE has been compared above, we could also discuss complexity of the RLS algorithm by the way to prove that the complexity of RLS-FDE is not so sophisticated as it seems to be for per symbol per mode.

RLS algorithm is also a main receiver-side equalization candidate for SDM systems, which utilizes iterative updating to minimize the exponentially weighted cost function, whose convergence speed is nearly independent on the input vectors. In the simulation of [9], the RLS algorithm was demonstrated through a cyclic prefix (CP) for each data block in MDM systems. The CP could reduce the complexity per symbol at the cost of throughput spectrum efficiency. We also previously proposed an overlap-save based FD-RLS algorithm for MDM systems with a fixed forgetting factor. It proved that RLS has the ability to achieve higher convergence speed, faster adaptation, better performance at the cost of moderately higher computational complexity compared with LMS algorithm [41].

The higher convergence speed of RLS leads to higher throughput efficiency. Generally, for a conventional MIMO-FDE, training sequence is required for initial weight matrix updating to ensure the decision-directed (DD) equalization after training symbols for continuous tracking. Therefore, slower convergence speed will result in a longer training sequence and then lower spectral efficiency. While RLS algorithm owns faster convergence speed compared with LMS algorithm, less training sequences are required so that higher spectrum efficiency would be achieved, especially in the long-haul MDM systems.

What’s more, as mentioned above, less training sequences are required for RLS algorithm to perform update processing. The computational complexity per mode per symbol of RLS algorithm is comparable with LMS, which is shown in

Table 1. Complexity comparison between the FD-RLS and FD-LMS.

Operation	Number of Complex Multiplications
	LMS	RLS
Compute output	$4m$	$4m$
Update filter coef.	$4m$	$4m$
FFT/IFFT	$(4+4m){\log }_2{N}_f$	$(4+4m){\log }_2{N}_f$
Update Kalman vector	-	$8+4m$
Update correlation matrix	-	$12m$

. Table 1 is the complexity comparison between the FD-RLS and the conventional FD-LMS algorithms, in which $m$ is the mode number and $N_f$ is the FIR filter length for each even/odd block equaling $2L{R}_{\mathrm{s}}\Delta \tau$. Let us consider a 15 spatial modes transmission system [42] where $m = 30$ and $N_f = 1024$, the complexity ratio $C_{\mathrm{RLS}}/C_{\mathrm{LMS}}$ coulde be derived to be around 1.33. The complexity ratio for different channels number $m$ is shown in Figure 3, which indicates that the complexity of RLS-FDE is comparable with, or rather slightly higher than LMS-FDE. Moreover, considering that RLS-FDE requires less training symbols as compared to that for the LMS algorithm and RLS can be switch into DD-LMS algorithm after initial weight matrix equalization, the computational complexity per training symbol of RLS algorithm is much lower than LMS.

3. RLS MIMO Algorithms and Its Application on Equalization of TV Effects

The procedure of an overlap-based FD-RLS algorithm is shown schematically in Figure 4 for a MIMO MDM system with $m$ degrees of freedom. Here, $y_i\left(n\right)$ and $x_i\left(n\right)$ represent the time-domain input signal vectors and output signal vectors of mode $i$ with $1\le i\le m$. After parallelization from serial sequences, two consecutive blocks of TD input vectors are concatenated into FFT with an overlap rate of 0.5 to obtain the FD input signal vector $Y\left(k\right)$ and the FD output signal vector $X\left(k\right)$ would be equalized in frequency domain through the overlap-save method. The prosses of this algorithm is similar to that of the FD-LMS algorithm shown in Figure 1 [37] except the correlation matrix updating process contained in the dotted box. The RLS algorithm is intended to minimize its exponentially weighted cost function with iterations in frequency domain, which utilizes the assumption of mutual independence between different frequency components. The iterative matrix updating can be formulated mathematically as Equations (5)–(7):

$$K(k)\leftarrow \frac{{\lambda }^{-1}R(k)Y(k)}{1+{\lambda }^{-1}Y{(k)}^{H}R(k)Y(k)}$$

(5)

$$W(k)\leftarrow W(k)+K(k)E{(k)}^H$$

(6)

$$R(k)\leftarrow {\lambda }^{-1}R(k)-{\lambda }^{-1}K(k)Y{(k)}^{H}R(k)$$

(7)

In the Equations (5)– (7), $R\left(k\right)$ is an m × m inverse matrix of the correlation of input signals at the k-th frequency component, which can be initialized as a unit matrix multiplied by a constant inversely proportional to the signal to noise ratio (SNR) of the input data; $\lambda$ is a forgetting factor satisfying $0\ll \lambda <1$ and the superscript $\mathrm{H}$ denotes the conjugate transpose operator. Kalman vectors, $K$, which is the key parameter to update the filter coefficients, always is computed based on the frequency domain signal input vector $Y\left(k\right)$ and the inverse correlation matrix $R\left(k\right)$ of the previous block. Besides, the even and odd tributaries should be updated separately because they use different input vectors and naturally different correlation matrix. Generally, RLS could exploit more information from the input signals by the Kalman vector compared to typical LMS algorithm based on stochastic gradient descent [43].

It is noteworthy that variable $\lambda$ could be time-varying to further improve the performance of RLS. Since a fixed $\lambda$ leads to a trade-off between a faster convergence speed and a lower misadjustment error, this compromise could be balanced well with a variable $\lambda$, which algorithm is referred to VFF-RLS. Lots of work around VFF-RLS has been done for different signal processing applications [44,45,46]. However, calculating the specific value of $\lambda$ is always operated by division or extraction of a root, which leads to rather higher computational complexity compared that with fixed forgetting factor. To avoid higher complexity while still holding the advantages of VFF, we proposed a FD-RLS schemes with an exponential VFF in Equation (8) [47]:

$$\lambda (n)={\lambda }_{\max }-\gamma \times {10}^{-n\tau }$$

(8)

where $n$ represents the FD-block index. $\lambda$ decreases with the increasing of $n$. So, at the beginning of the equalization, we set $\lambda$ to be a lower value for faster convergence. When the filter matrix converges to near steady-state solution through enough iterations, $\lambda$ reaches ${\lambda }_{max}$ to ensure low misadjustment. Our proposed VFF-RLS-FDE algorithm was utilized to a 6 × 6 MIMO-MDM transmission data. The experimental results showed that the algorithm achieve better performance than conventional LMS and the RLS has a fixed $\lambda$ equaling 0.9. It can maintain faster convergence speed as shown in Figure 5a, which plots the mean square error (MSE) curves in a 600-km MDM transmission. The part of beginning convergence is zoomed in in the inset, where the green line of dashes stands for the target MSE level. For better comparison under the same standard, the desirable MSE is fixed to $1.9 \times {10}^{-5}$, which represents an MSE level that could be achieved by both the VFF-RLS/RLS and LMS algorithms even through 1000-km MDM transmission.

In order to further highlight the improvement of the convergence speed of VFF-RLS/RLS, we defined the ratio of convergence speed (RCS) as the ratio of the number of training symbols needed by LMS algorithm to that by VFF-RLS or RLS algorithm for converging to achieve the same MSE. Figure 5b shows the RCS values at different transmission distances for conventional RLS and VFF-RLS, where the solid and dotted lines are cubic polynomial fits for RLS and VFF-RLS, respectively. Faster convergence speed was obtained for VFF-RLS/RLS at longer transmission distance. The advantage in convergence speed could be explained by that lower SNR caused by longer transmission distance requires more robust performance to reach the target MSE, which condition could be satisfied by VFF RLS or RLS. In particular, after transmission excessing 800-km, compared to conventional LMS algorithm, the convergence speed is improved over 90% by the VFF-RLS and RLS algorithm. Most notably, the RCS of VFF-RLS-FDE is 15.3% higher than conventional RLS-FDE at 600 km, which means that the VFF-RLS algorithm can also obtain a significant RCS improvement over the conventional RLS algorithm.

In addition to the computational complexity, performance and convergence speed, the tracking speed of the equalization algorithms also attracts much attention for long-haul and high-dimension MDM transmission, due to the TV effects in FMF. On the one hand, the evolution rate of transmission matrix of FMFs scales with the number of spatial modes [18]. Compared with SMFs, the evolution speed of two-mode fiber is one order of magnitude faster. What’s more, the rotation speed of Stokes vector of four-core coupled-core fiber on the Poincare sphere may exceed 100 krad/s [48]. On the other hand, the evolution rate of transmission matrix also scales with fiber length [18]. Therefore, for long-haul MDM transmission, which are easier to be disturbed by perturbations, dealing with TV MDM channels and developing faster-tracking MIMO equalizers are more necessary. In addition, there are no effective optical devices to control or compensate the time varying modal dynamics of MDM system. Therefore, people will rely more on DSP techniques to deal with TV MDM channels. Since RLS algorithm has been proposed theoretically for the MDM system to achieve a fast convergence speed and tracking capability, we had further extended the VFF-RLS algorithm to the equalization of TV channels in FMF links to track the constantly changing transfer matrix [49], which is FD error-dependent VFF-RLS algorithm (ED-VFF-RLS).

The forgetting factor of our previously proposed ED-VFF-RLS algorithm could be decided automatically by error signals. With regard to algorithm schemes, the procedure of ED-VFF-RLS is similar with the above conventional FD-RLS algorithm except for the forgetting factor updating process. Figure 6 shows schematically the procedure of the ED-VFF-RLS algorithm of an $m\times m$ MDM system. The forgetting factor $\lambda$ is decided by the estimation of the power of noise signal and priori error signal. Specifically, $\lambda$ would be recalculated at each frequency-domain block as Equation (9) [44]:

$$\lambda (n)=\left\{\begin{array}{cc}{\lambda }_{\max }& if\hspace{1em}{\widehat{\sigma }}_{\mathrm{e}}(n)\le \gamma {\widehat{\sigma }}_v(n)\\ \min \left\{\frac{{\widehat{\sigma }}_q(n){\widehat{\sigma }}_v(n)}{\xi +\left|{\widehat{\sigma }}_e(n)-{\widehat{\sigma }}_v(n)\right|},{\lambda }_{\max }\right\}& if\hspace{1em}{\widehat{\sigma }}_{\mathrm{e}}(n)\ge \gamma {\widehat{\sigma }}_v(n)\end{array}\right.$$

(9)

where ${\lambda }_{\max }\cong 1$, $1\le \gamma \le 2$, $\xi$ is a small positive number preventing division by zero and $q\left(n\right)=Y_{\mathrm{ev},\mathrm{od}}^{\mathrm{H}}\left(n\right)P_{\mathrm{ev},\mathrm{od}}^{\mathrm{H}}\left(n\right)Y_{\mathrm{ev},\mathrm{od}}\left(n\right)$. In addition, ${\sigma }_q^2\left(n\right)=E\left[q^2\left(n\right)\right], \mathrm{while}$ ${\widehat{\sigma }}_e^2\left(n\right)$ and ${\widehat{\sigma }}_v^2\left(n\right)$ is the mean power of the priori error signal and the noise signal, respectively. For convenience of calculation, an exponential window [50] was adopted to computed the three mean powers in Equation (9) with ${\widehat{\sigma }}_e^2\left(n\right)=\alpha {\widehat{\sigma }}_e^2\left(n-1\right)+\left(1-\alpha \right)E\left[e^2\left(n\right)\right]$, ${\widehat{\sigma }}_q^2\left(n\right)=\alpha {\widehat{\sigma }}_q^2\left(n-1\right)+\left(1-\alpha \right)E\left[q^2\left(n\right)\right]$ and ${\widehat{\sigma }}_v^2\left(n\right)=\beta {\widehat{\sigma }}_v^2\left(n-1\right)+\left(1-\beta \right)E\left[e^2\left(n\right)\right]$, where the weight factors α and β meet the condition of $\alpha <\beta <1$.

We use two sets of parameters to represent the different trade-off between misalignment error and tracking capability, respectively. And Figure 7a,b have compared the tracking capability of ED-VFF-RLS with that of conventional LMS under the two sets of parameters. First of all, Figure 7a shows that the proposed ED-VFF-RLS has the nearly same performance as LMS under the first set of parameters when the angular frequency of rotation matrix is low. When angular frequency is further increased to over 100 krad/s, the Q² factors of the ED-VFF-RLS algorithm have more than 2.2-dB improvement, in comparison with LMS algorithm adopting a smaller unnormalized step-size. It can be observed that the consistency of FD algorithms agrees well with TD algorithms. While in Figure 7b, the ED-VFF-RLS algorithm with smaller $\alpha$ factor displays nearly identical tracking capability as the conventional LMS algorithm using a larger unnormalized step-size, but the former’s Q² factor is improved around 0.5-dB as compared to LMS. It is worth noting that at the cost of a minor Q² penalty in static environment, the performance of equalizers could be significantly promoted in fast time-varying situations.

4. DSP Techniques on Compensation of MDL Effects in FMF Links

Transmitted signals in SDM links are also impaired by different gain or loss between different spatial modes, which phenomenon could be collectively referred to as MDL effect. MDL mainly arises from the imperfections of inline components such as splices and micro-bends of fibers, amplifiers and multiplexers.

MDL causes different OSNR degradations between different modes, which finally reduce the system capacity. For an optical MIMO system modeled as Equation (1), if there is not any prior information of the SDM links at transmitter, the capacity, $C$, of a MIMO-SDM system with equal transmitted power between different modes is given by Equation (10) [51,52]:

$$C={\displaystyle \sum }_{i=1}^{N_R}lo{g}_2\left[1+\frac{P}{N_T{N}_0}{\lambda }_i\right]$$

(10)

where $P$ is the total power of all transmitted signal, $N_0$ is the noise power of each received channel, and ${\lambda }_i$ is the eigenvalues of the $N_R \times N_{\mathrm{R}}$ positive semidefinite Hermitian matrix $H{H}^{†}$ that satisfy ${\lambda }_i \ge 0 \left(i \in \left\{1, 2,\dots , N_{\mathrm{R}}\right\}\right)$.

Equation (1) indicates that the available MIMO system capacity is the summation of the capacity of respective parallel $N_R$ single channels, each of which has a power gain or loss of ${\lambda }_i$ [52]. Since MDL could be defined as the ratio between the maximum eigenvalue ($\max \left({\lambda }_i\right)$) and the minimum eigenvalue ($\min \left({\lambda }_i\right)$) of $H{H}^{†}$, the capacity of MIMO system is naturally reduced due to the presence of MDL.

On the other hand, in terms of DSP, unlike DMGD which could be compensated totally by MIMO detection, MDL has a serious damage on linear MIMO equalization performance as a result of the non-unitary transfer matrix induced by MDL. Much effort on DSP techniques has been explored to mitigate the MDL-induced capacity loss or to suppress MDL-impaired non-unitary channels.

4.1. Maximum-Likelihood (ML) Detector

A receiver-side ML detection scheme was firstly reported in [53] to compensate MDL in FMFs. The ML detector is designed to maximize the probability $P(x | y)$ for all transmitted hyper-symbols. In an additive white Gaussian noise (AWGN) channel model, the typical form of ML detection principle [54] is given by Equation (11):

$${\widehat{x}}_{ML}=\arg \underset{x\in {\mathbb{A}}^{N_T}}{\min }{‖y-Hx‖}^2$$

(11)

where $\mathbb{A}$ is the set of value of ideal constellation symbols, $x$ is the transmitted symbols vector, $y$ is the received symbols vector, and ${\widehat{x}}_{ML}$ is the estimation of transmitted symbols of ML detector. The ML detection seeks for the transmit symbols vector multiplied by transmission matrix $H$ which has the minimum Euclidean distance from the received signal. The search is conducted over all possible symbol combinations from all spatial channels and possible modulation symbols.

As shown in Figure 8, ML detection is the optimal receiver detector among all the simulation results compared with other linear equalizers such as zero-forcing (ZF) and minimum-mean-square-error (MMSE) equalizer [53]. The advantages of ML detection make it a powerful tool to reduce MDL in combination with strong-coupling fibers or mode scramblers [33,53].

However, its complexity grows exponentially with both the number of modes and the order of the modulation constellation with regard to Figure 9, forcing us to consider the compromise between its performance to and the extremely high computational complexity. Thus, for MDL reducing, few DSP methods were proposed to achieve near-ML performance without being impacted by the huge computational complexity, such as the improved reduced-search (IRS) ML detection [55,56], the sphere decoding (SD) [56] and complex lattice reduction aided detection (CLR) [57]. Each algorithm outperforms the linear equalizer MMSE to improve the tolerance of MDL for MDM system with acceptable computational complexity gain.

4.2. Interference Cancellation (IC)

Linear MIMO equalizers are also attractive to mitigate MDL effect because of its relative low complexity, adaptive schemes and high compatibility. The performance of these equalizers could be further enhanced by introducing more prior information of the channels which indicates that the SNR of each mode can be considered to utilize. Inspired by this idea, the interference from higher-SNR channels to the lower-SNR ones can be cancelled by equalizing stronger modes in advance. Each channel is equalized one by one from the highest- to the lowest-SNR modes by estimating the interference from other channels, which equalizer is called successive interference cancellation (SIC) schemes.

In order to realize longer-distance SDM transmission in [58,59], an unreplicated successive interference cancelling (U-SIC) equalizer that mitigates MDL-induced intermodal interference were proposed and successfully demonstrated. Through the corresponding experiments, an MDL tolerance improvement of 3.3 dB and an OSNR gain of >4.5 dB were demonstrated even under high DMGD up to 39 ns. As shown in Figure 10, as compared to the conventional SIC algorithm, channel estimation is no more required for the proposed U-SIC.

Figure 11 shows the advantages of U-SIC scheme on the improvement of Q factor and lower requirement of OSNR to achieve hard-decision (HD) forward error correction (FEC) (FEC) limit for bit error ratio (BER) in comparison with conventional linear MIMO algorithms. What’s more, in [60] the combination of U-SIC technique and mode scramblers enable the demonstration of the 2500-km long-haul MDM transmission with more than 2-dB Q-factor gain.

For higher performance in MDL-impaired MDM systems, a novel unreplicated parallel interference cancellation (U-PIC) scheme was proposed in [61] by increasing the number of iterations compared with U-SIC. By iterative utilization of the soft information of transmitted symbols from decoders, the U-PIC equalizer successfully cancel the MDL-induced intermodal interference among the channels with similar SNR. Experiment demonstrated that the U-PIC algorithm could achieve better performance than both the conventional linear MIMO equalizers and the above U-SIC algorithm.

4.3. Space-Time Coding Technique (STC)

A few studies have also shown that STC technique in transmitter can effectively suppress MDL effects in SDM systems, which was originally proposed to suppress spatial channel fading in wireless MIMO communications [62]. Instead of directly multiplexing different $M$ modes to transmit independent symbols stream $S_{M\times 1}$. MIMO scheme could be beneficial from the space and time degrees of freedom at once by making multiple copies of an input symbol over different channels at different time slots. Naturally, through various channel propagation, these copies could be exploited to have a better estimation of the transmitted symbols at receiver side and the MDL effect would be further mitigated. The schematic structure of SDM MIMO system applying STC technique is shown in Figure 12, where $X_{M\times T}$ is a space-time transmitted codeword. $Y_{M\times T}$ is the codeword at receiver side where $M$ denotes the number of channels and $T$ is the length of the code. $N_{M\times T}$ denotes a white Gaussian noise.

The effectiveness of STC for MDL suppression in SDM systems has already been demonstrated by various simulations and subsequent experiments. In [63] and [64], it was shown that the combination of redundancy-free STC with inline mode scramblers and ML detector can thoroughly eliminate the SNR penalties caused by the MDL at most 10 dB. Moreover, two reduced-complexity STC detectors achieving a near-ML performance were also proposed: a sub-optimal decoding scheme and a multiblock STC approach in [64]. What’s more, two more simple schemes were proposed for STC applications to decrease coding complexity in SDM systems. In [65], Hadamard transform (HT) was utilized in STC technique to disperse the power of each symbol over all modes for identical channel equalization performance. Then, a dense 627-km SDM transmission experiment were demonstrated to prove the HT’s advantage of improving MDL tolerance, where the transmission distance was increased by 20%. While in [66], a mode-interleaving method was also presented to simplify STC schemes, which method could improve the transmission reach by 40% through simulation, as well as saving $\mathrm{O}\left(M^2\right)$ computation complexity compared with conventional space-time block coding (STBC) method. The numerical conclusions of the benefits of STC were soon confirmed in [67] through experimental measurements, where lower BER at the same OSNR condition was achieved and the Q-factor in FMF transmission systems was improved by 3.4 dB in an absence of 10-dB MDL as shown in Figure 13a,b, respectively.

Since STC with mode scramblers was shown to be a powerful tool to mitigate MDL effect, more efforts were adopted to explore possible techniques for further improved performance of the SDM systems that apply STC technique. In [68], an upper bound of the error probability of the optical channel induced by MDL was derived, this upper bound leads to a design criterion of space-time codes which is able to totally mitigate MDL in SDM systems. In [69] and [70], the combination of STC and FEC was studied to suppress the MDL effect. Numerical simulation results showed that, through guidance of the aforementioned upper bound of error probability, the obtained gain of the combination is equal to the summation of both coding gain.

Moreover, the DSP techniques at the receiver for MDM systems were also widely studied to better suite with STC schemes. In [71] and [72], a STBC-aided RLS algorithm was developed for MDL mitigation in MDM systems to achieve quasi-RLS convergence speed as well as quasi-LMS complexity, with which assistance a 7.8 dB OSNR tolerance was obtained for polarization-division multiplexed (PDM) 64QAM system. In [73], a receiver-side linear decoding schemes was proposed for the orthogonal STBC-aided MDM system to achieve near-ML performance with reducing decoding complexity, which grows linearly with the order of constellations. In [74], a novel hybrid detection with combination of the IRS method and the decision feedback equalizer (DFE), was realized for the threaded algebraic space-time codes (TAST) assisted system, where the proposed detector achieved near-ML performance with a lower computational complexity.

We have concluded the above three DSP solutions for suppressing or mitigating MDL effect in

Table 2. Comparison between different digital solutions for mitigating MDL effect.

Classification	Receiver-Side		Transmitter-Side
Name	ML	IC	STC
Complexity	High	Low	Moderate (excluding equalizer)
Advantages	Optimal equalizer	Suboptimal equalizer	Easily compatible with other solutions; Coder technique
Disadvantages	High complexity	Worse performance	Loss of multiplexing gain; Required for extra coder and equalizer

to help readers understand their advantages and disadvantages. As we can, see ML detector and IC techniques are receiver-side equalizers while STC is a transmitter-side coder technique which means it is easy to cooperate with other receiver-side solutions.

5. Novel DSP Techniques for MIMO Systems

With the fast improvement of data processing capacity, many novel DSP algorithms have been proposed in the last few years. This section focuses on the recent emerged DSP techniques for MIMO systems, including the extended DBP algorithm to handle nonlinear effects in FMFs, AI techniques to achieve MIMO equalization, and so on. In comparison with the conventional methods, these novel DSP algorithms have the potential to replace them for better performance.

5.1. Digital Back-Propagation Algorithm (DBP)

It is well known to us that the DBP algorithm is the most common DSP method to compensate nonlinearities in single mode fiber. However, as mode coupling exists in FMFs, traditional DBP cannot be successfully used. The extension of single-mode split-step Fourier method (SSFM) based DBP has recently been proposed to solve the multimode propagation problem over FMFs [30,31]. As illustrated in Figure 14, a symmetric SSFM is implemented, where the nonlinear effects are considered in the middle of the segment instead of the segment boundary. By utilizing this structure, the generalized nonlinear Schrödinger equation (GNLSE) can be simplified according to the strength of the linear mode coupling.

Following the strategy above, two original Manakov-like equations with nonlinear coefficients are derived for FMFs in two extreme coupling regimes. In the weak coupling (WC) regime, we only need to consider the average value over birefringence fluctuations, which leads to a reduced intramodal degeneracy factor of 8/9 and a reduced intermodal degeneracy factor of 4/3 [75]. In the strong coupling (SC) regime, all propagation modes are included on average.

A 19 channels wavelength-division multiplexed (WDM) MDM system is simulated to study the performance as shown in Figure 15 [31]. There are six LP modes with dual-polarization in each channel, as 14 Gbaud polarization-multiplexed 16QAM, the overall data rate of 12.8 Tb/s is transmitted over 12 spans of 20 km 6-mode FMF. In the receiver side, DBP is implemented by utilizing a fixed step size of 100 m, subsequently, chromatic dispersion (CD) compensation, time synchronization, channel estimation is performed, and finally Q-factor is calculated.

Figure 16 shows the Q-factor performance with launched power in three different case. In Figure 16a, both WC-DBP and SC-DBP have a Q-factor improvement at low XT, which means that the Manakov approximations are reasonable. But it can be found that only SC-DBP can perform well at high XT shown in Figure 16b, which can be explained by the nonlinear coefficients in WC-DBP is larger than the actual channel leading to overcompensation. As shown in Figure 16c, when GD management with high DMD is utilized, WC-DBP can still perform well in low XT.

Recently, DBP is realized in a deep neural network architecture [32]. The architecture of this deep neural network (DNN) is shown in Figure 17, where the inputs are the received signal samples and the outputs are the estimated symbols. The weight $W_k$ represent linear effects in transmission, such as mode dependent loss and chromatic dispersion. And ${\sigma }_k$ is the nonlinear operator which represents the nonlinear phase derotation. A polarization rotation matrix $R_k$ is appended at the k-th stage of the DBP operation and can be optimized from training data without any knowledge of the principle states of polarization (PSP). Thus, all the linear and nonlinear parameters in traditional DBP can be optimized in the DNN.

The PDM-WDM transmission system is experimentally constructed to investigate the performance of this DNN [32]. The experiment setup is shown in Figure 18, where a 5 channel 50-GHz-spaced WDM system with 28 Gbaud 16-QAM modulation format is utilized. Ten fiber spans with different length are cascaded. At the receiver side, the signal is sampled and then processed in offline.

The Q factors of the center channel is shown in Figure 19a, it can be found that the DNN based DBP outperforms traditional DBP even in 1 step per span (StPS). Figure 19b shows the “M”-shaped amplitude responses of the filter, which exhibits some high-pass feature at the later staged in DNN. This phenomenon can be attributed to that the “M”-shaped filter tries to compensate an additional undesired term with a “∩” shaped spectrum. The optimized nonlinear coefficients with a “U” shape are shown in Figure 19c, which suggests that the nonlinear-phase derotation is larger in the middle stages than the beginning and end of the DBP. Figure 19d shows the optimized rotation angles for PSP, which agree with the theoretical expectation.

5.2. Artificial Intelligence (AI) Technology for MIMO Equalization

Recently, AI technologies have also attracted great interest in optical fiber communication, for its high excellent ability. Deep learning, as a main branch of AI, provides many neural network models to handle various optical communication problems. For example, artificial neural network (ANN) is widely utilized in nonlinear equalization [76,77,78] and optical performance monitoring [79,80], convolutional neural network (CNN) has the ability to get high accuracy of output images in MMF [81] and enabled high-spatial-density SDM framework [82]. And DNN, which is more complex than ANN, shows its excellent performance in end-to-end learning [83]. What’s more, the applications of deep learning in DSP bring a revolution in MIMO transmission systems. In this part, several neural networks are introduced to achieve MIMO equalization, and better performance can be obtained compared with the traditional methods.

As mentioned above, mode coupling is inevitable in MDM system which causes channel crosstalk and DMD. Thanks to deep learning, ANN can be used as a MIMO equalizer to mitigate these channel impairments [84]. A two mode MDM transmission system with an ANN equalizer is illustrated in Figure 20a, the ANN equalizer is placed at the receiver side and consists of three layers as depicted in Figure 20b. Meanwhile, a novel feed-forward and back propagation algorithm is employed to update the weight, in which the back propagation retrieved the feed-forward output as an input to the hidden layer.

Channel impulse response is analyzed to characterize the ANN performance. Figure 21a shows the input channel impulse response for two channels, in which both channels are coupled with each other significantly. Figure 21b,c are the target and output channel impulse response, respectively. There is a high similarity between these two pictures where the impulse response of both channels is separated clearly. Hence, this ANN equalizer has a good performance in compensating mode coupling and DMD.

Based on the fact that deep learning neural networks are highly nonlinear and have an ability to form arbitrarily nonlinear decision boundaries, a supervised deep learning neural network (DLNN) is employed in MIMO detection [85]. A schematic diagram is shown in Figure 22, where an MDM optical transmission system with N input modes and N output modes is adopted. At the receiver side, the received signals are fed to the DLNN to implement MIMO detection.

Figure 23 shows the detailed structure of the proposed DLNN. In this scheme, N complex signals are decomposed into the real parts and imaginary parts. And then, these 2N real signals are fed into a feed-forward neural network with M layers. For N × N MIMO system, 2N input neurons and 2N output neurons are required in the DLNN, which means that the complexity of DLNN increases with the polynomial time complexity of the MIMO dimension.

The performance of DLNN, traditional ZF detector and advanced Semi-Definite Relaxation Row-by-Row (SDR-RBR) is compared in a 2 × 2 MDM system. Figure 24 shows the BER curves versus channel condition number. It can be seen that DLNN has a better performance than the other two detectors with Q-factor greater than 15 dB.

It is generally known that the nonlinear Shannon limit has become the basic barrier of a channel maximum capacity in communication. To mitigate various nonlinear effects, DSP can be utilized. Recently, deep-learning-based MIMO equalization has been adopted to tackle the interplay of FWM interference and parametric amplified noise in coherent optical communication [86]. Meanwhile, inter-carrier crosstalk effects even without frequency stochastic variations can also be compensated by this novel MIMO neural network, which has hitherto been considered impossible.

In orthogonal frequency division Multiplexing (OFDM) system, the frequency dependent deterministic distortion leads to the center subcarriers suffering from higher nonlinear distortion than edge subcarriers, which leads to a lower signal-to-interference-plus-noise ratio (SINR). Consequently, as shown in Figure 25, a more complex ANN is employed in center subcarriers than edge subcarriers. Following this strategy, the SINR difference between each subcarrier is suppressed, resulting in a more robust equalization performance.

Figure 26 illustrates the structure of this mentioned equalizer. The hidden layer is adaptively adjusted by a set of electronic frequency sub-carriers which represent the third dimension of different frequencies. The complexity of this adaptive neural network depends on the hidden layers counts which is 3 or 4 and the modulation format level which is 4 for QPSK or 16 for 16-QAM. Therefore, compared with traditional methods, this 3D MIMO neural network will not increase the computational complexity.

In order to explore the performance of the MIMO deep learning equalization algorithm, as shown in

Table 3. Different cases of block-size for middle subcarriers and number of hidden layers.

Cases	Block-Size for Middle Subcarriers	Hidden Layers
1	50	3
2	100	3
3	150	3
4	2 × 54 (middle groups), 2 × 51 (edge groups)	4

, four different cases with different number of central subcarrier blocks were used for comparison. Figure 27a shows the result in a 3200-km QPSK WDM-CO-OFDM system, it can be seen that the performance under case 2 is better than the other three cases. And compared with the commercial single-carrier/polarization QPSK system for a 3200 km multi-channel configuration, the MIMO deep learning under case 2 brings a record 3.8-dB Q factor improvement. Such performance enhancement is maintained for a 2000-km single channel 16-QAM CO-OFDM system as is shown in Figure 27b.

In case 2, MIMO deep learning equalizer with traditional ANN and deterministic DBP in different transmission systems is compared: (a) WDM QPSK CO-OFDM at 3200 km and (b) Single channel 16-QAM CO-OFDM at 2000 km. It can be found from Figure 28a that compared with the ANN and DBP methods, at the optimal launched optical power (LOP) of −5 dBm, the Q-factor is improved by 1.4 dB and 2.1 dB, respectively. This is due to the fact that the DBP cannot accurately know the absolute position and relative spacing of the channels and subcarrier frequencies. From Figure 28b, it can be found that compared with linear equalization and ANN, the maximum allowable LOP of this MIMO equalizer is increased by 4 dB, and at the optimal LOP of 6 dBm, the Q factor 2.1 dB is improved in comparison to DBP.

In

Table 4. Applications of neural network in MIMO system.

Structures of NN	Applications
ANN	Compensating mode coupling and DMD
DLNN	MIMO detection
3D-ANN	MIMO equalization in the presence of FWM

, different structures of neural network were summarized in terms of their respective applications in DSP techniques of MDM systems, which you could find that each type has its own speciality in mitigation of certain impairments in MDM links.

6. Conclusions

In this paper, we presented an overview of recent progress on advanced DSP techniques for MDM systems. Methodologies of digital approaches were reviewed that can mitigate the impact of MD, XT, MDL and nonlinearity on the system performance in long-haul MIMO-MDM transmissions. What’s more, AI technologies were also mentioned for its advantage to handle various optical communication problems, and we believe that machine learning would bring more possibilities on equalization, monitoring and characterization of MDM systems. In particular, the TV channels for MDM transmission also attract more attention in recent years, so our DSP techniques of FD-VFF RLS for MIMO-MDM systems were firstly reviewed to enlighten more works on tracking TV effect in FMF links.