Machine Learning-Assisted Mitigation of Optical Multipath Interference in PAM4 IM-DD Transmission Systems

Abstract

This paper aims to mitigate multipath interference (MPI) in intensity modulation with direct detection (IM-DD) systems using machine learning techniques, specifically for four-level pulse amplitude modulation (PAM4) systems. We propose a machine learning-assisted MPI mitigation scheme, called KNN-aided SVM+RF-M. In this scheme, KNN-aided SVM serves as a soft decision algorithm that adapts the decision threshold to signal amplitude fluctuations, improving the decision accuracy for MPI-affected PAM4 signals. By replacing the original hard decision in the RF-M algorithm with KNN-aided SVM, we mitigate the error transfer problem inherent in RF-M. MPI mitigation is then achieved through MPI estimation and noise value cancellation methods applied to signals after soft decision processing. Our proposed scheme is validated in a 28 GBaud PAM4-DD transmission system, and the simulation results show that our proposed scheme can improve SIR tolerance by 2 dB and receiver sensitivity by about 1 dB at the 7% HD-FEC threshold compared to the original RF-M scheme.

1. Introduction

The demand for communication bandwidth is rapidly increasing, driven by the emergence of new network applications such as virtual reality and augmented reality. This surge in demand, particularly for data center links, poses significant challenges for the design and deployment of data centers. Optical transmission has emerged as a more promising and feasible solution for data center networks, offering high bandwidth, low latency, and low power consumption [1]. Currently, intensity modulation with direct detection (IM-DD) systems are the predominant choice for optical transmission in data centers, favored for their low power consumption, cost effectiveness, and simple configuration [2]. Among the various advanced modulation formats used in IM-DD systems, pulse amplitude modulation (PAM) typically outperforms discrete multitone (DMT) and carrier-less amplitude modulation (CAP) owing to its simplicity [3]. Specifically, four-level pulse amplitude modulation (PAM4) has been recommended for 400 GbE applications [4,5]. Even for next-generation 800 GbE links, PAM4 is regarded as a promising solution, making it the preferred choice for IM-DD transmission systems [6,7].

One of the key factors limiting the performance of IM-DD systems is multipath interference (MPI). MPI primarily arises from multiple reflections of optical signals at the end faces of fiber connectors and transceivers. Specifically, during the transmission of optical signals through fibers, loose physical connections and dirty connector end faces can lead to Fresnel reflection and Rayleigh scattering. This results in the reflection of optical signals traveling in the opposite direction. When the transmitted signal and the reflected light—subject to random delays—arrive at the receiving end simultaneously, interference occurs, resulting in transmission degradation known as MPI impairment. At the receiver of IM-DD systems, after the optical signal is detected by the photodiode (PD), MPI impairment is converted from laser phase noise to intensity noise, which increases the bit error rate (BER). Furthermore, when numerous contaminated fiber connectors are present in the system, the intensity of the reflected signal—and, consequently, MPI noise—increases, leading to more pronounced degradation of system performance. Additionally, cost-effective large linewidth lasers, such as distributed feedback lasers (DFBs), which are commonly used in IM-DD systems, further exacerbate MPI noise. PAM4-format signals are particularly vulnerable to MPI noise, and the impact of MPI on higher-order PAM signals is even more severe. Therefore, it is essential to explore mitigation strategies for MPI in IM-DD systems. The implementation of sophisticated digital signal processing (DSP) at both the transmitter and receiver is considered necessary to address transmission impairments in IM-DD systems [8]. Consequently, DSP algorithms for MPI mitigation are recognized as a viable solution.

In recent years, there have been several innovative studies focused on mitigating MPI. Based on the low-frequency characteristics of MPI, it was proposed in [9] that adding a high-pass filter (HPF) at the receiver could effectively filter out MPI noise. However, the effectiveness of this approach depended on the cut-off frequency set by the HPF, and it could introduce additional power costs for large linewidth lasers, where its effectiveness was diminished. To enhance the performance of the HPF scheme, another MPI mitigation strategy was introduced in [10], which employed DC-balanced line coding at the transmitter along with the HPF at the receiver. This hybrid approach flattened the low-frequency components of the signal through coding and subsequently filtered out the low-frequency MPI noise. Additionally, ref. [11] proposed an adaptive decision threshold (ADT) to improve the BER performance of the system without any MPI noise suppression, utilizing an adaptive dynamic update of the decision threshold. In our previous work, we also introduced two algorithms within DSP modules, DA-M and RF-M, which effectively mitigated MPI [12]. The DA-M scheme mitigated interference by eliminating the instantaneous intensity shifts caused by amplitude fluctuations in the signals, while the RF-M scheme reconstructed the MPI noise model before removing MPI noise from the distorted signals. However, both algorithms encountered error transfer issues and required optimization.

Currently, machine learning (ML) has emerged as a significant solution to various challenging problems owing to its advantages, including high accuracy, adaptability, and execution efficiency. ML technology has been widely adopted in short-range optical communication scenarios [13], such as optical performance monitoring (OPM), modulation format identification (MFI), signal processing, and indoor optical wireless communication (OWC). The application of ML techniques in IM-DD systems is considered an effective alternative to classical DSP methods for improving system performance. ML techniques have been employed to address severe impairments caused by issues such as inter-symbol interference (ISI) arising from dispersion and nonlinear signal detection by PD. These techniques have been applied in various areas, including linear and nonlinear equalization, dispersion compensation, and nonlinear pre-distortion [14].

Furthermore, DSP solutions leveraging ML techniques can enhance signal classification capabilities. For instance, ref. [15] presented a scheme utilizing simplified photonic reservoir computing (RC) to classify nonlinearly distorted signals, demonstrating experimentally a BER improvement by two orders of magnitude compared to direct classification of transmitted signals, corresponding to a 75% increase in communication range. In [16], complete binary tree support vector machines (CBT-SVMs) were proposed, achieving significant BER performance improvements when applied to PAM4 and PAM8 signals. Additionally, ref. [17] introduced weighted K-nearest neighbor (W-KNN), which integrated signal classification and equalization, achieving a power budget of up to 35 dB in a 50 Gbps PAM4 25 km passive optical network (PON) simulation system. In summary, applying machine learning techniques in IM-DD systems presents an effective solution.

In this paper, we propose a machine learning-assisted MPI mitigation scheme, called KNN-aided SVM+RF-M. Our proposed scheme is the first to introduce machine learning-assisted signal decision as a means to mitigate MPI. The amplitude of signals impaired by MPI exhibits irregular fluctuations in the signal level diagram. However, by leveraging the capability of a support vector machine (SVM) to identify the fluctuating patterns of such MPI-impaired signals, it becomes feasible to dynamically adjust the signal decision thresholds in accordance with amplitude variations—a process referred to as soft decision.

The KNN-aided SVM+RF-M scheme consists of two main processes: (1) The signal decision algorithm, KNN-aided SVM, enhances the decision accuracy for MPI-affected PAM4 signals. This algorithm comprises two stages of signal classification. The first stage employs a novel multi-category classification classifier that we propose, named 2ary-SVM, to initially classify PAM4 signals. The SVM model leverages its ability to recognize MPI damage patterns to provide a soft signal decision, particularly through three decision thresholds that can adapt to fluctuations in signal amplitude. The second stage incorporates W-KNN for secondary classification after the first stage, which is used to correct misclassified sample labels from the first stage’s results, thereby further improving the classification accuracy of the 2ary-SVM algorithm. (2) Fluctuation mitigation is based on the RF-M algorithm. In this process, the initial hard decision of the original RF-M algorithm is replaced by the KNN-aided SVM algorithm, followed by MPI estimation and noise value elimination on the results obtained after the soft decision, which facilitates fluctuation suppression. Our proposed mitigation scheme is validated in a 28 GBaud PAM4-DD transmission system. The simulation results demonstrate that, compared to the original RF-M scheme, our proposed scheme improves SIR tolerance by 2 dB and improves receiver sensitivity by about 1 dB, considering the 7% HD-FEC threshold.

2. Principle of the KNN-Aided SVM+RF-M Scheme

2.1. KNN-Aided SVM Algorithm

As illustrated in Figure 1, the PAM4 signal amplitude affected by MPI exhibits irregular fluctuations, leading to a high probability of misclassification when using traditional hard decision methods. Previous studies highlighted the effectiveness of ML algorithms for addressing such non-linearly separable problems, prompting us to select SVM for signal classification. The SVM model, trained on the amplitude of PAM4 signals, effectively adapts to amplitude fluctuations by recognizing the patterns of MPI-induced damage. This method, termed soft decision, allows the SVM model to output support vectors that represent the amplitude fluctuation patterns of the damaged signals, as illustrated in Figure 1b.

The fundamental SVM classifier is a binary classifier that seeks to find the maximal margin hyperplane to define the decision function. For linearly inseparable data, SVM employs a kernel function to map the data into a higher-dimensional space where they become linearly separable. As depicted in Figure 1a, PAM signals affected by MPI are linearly inseparable in two dimensions, but SVM transforms them into a linearly separable form in three dimensions.

The SVM training process is formulated as a quadratic programming problem, represented by the training dataset $\mathrm{T}=\left\{\right(x_1^r,y_1^r),(x_1^r,y_1^r),...,(x_n^r,y_n^r\left)\right\}$, where $x_i^r$ is the training sample and $y_i^r$ is the corresponding label, with $y_i^r\in \left\{+1,-1\right\},1\le \mathrm{i}\le \mathrm{n}$. The testing dataset $\mathrm{E}=\left\{\right(x_1^e,y_1^e),(x_2^e,y_2^e),...,(x_m^e,y_m^e\left)\right\}$ consists of m samples to be tested. The optimization objective is given by:

$$\begin{array}{c} \underset{\alpha }{\min }{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^n}{\alpha }_i{\alpha }_j{y}_i^r{y}_j^{r}K\left(x_i^r·x_j^r\right)-{\displaystyle \sum _{i=1}^n}{\alpha }_i, \mathrm{s}.\mathrm{t}. {\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{n}}}{\alpha }_i{y}_i^r=0, {\alpha }_i\ge 0\end{array}$$

(1)

where $\alpha ={({\alpha }_1,{\alpha }_2,{\alpha }_3,\dots {\alpha }_n)}^r$ are the Lagrange multipliers, and $K\left(x_i^r·x_j^r\right)$ is the kernel function. Solving this constrained optimization problem yields the decision function $\mathrm{f}\left(x\right)$:

$$\begin{array}{c}\mathrm{f}\left(x_j^e\right)=\left({\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{n}}}{\mathsf{\alpha }}_{\mathrm{i}}^{*}{y}_i\mathrm{K}\left(x_j^e,x_i^r\right)\right)+b^{*}\end{array}$$

(2)

where $x_j^e$ is the test sample, $1\le \mathrm{j}\le \mathrm{m}$, and ${{\mathsf{\alpha }}_{\mathrm{i}}^{*} \mathrm{a}\mathrm{n}\mathrm{d} b}^{*}$ are the parameters of the optimal hyperplane.

The fundamental SVM classifier cannot directly perform a four-level decision for PAM4 signals; therefore, it must be extended to a multi-class classifier. Figure 2 illustrates the proposed classification strategy, which decomposes the four-level decision task into two binary classification tasks using two SVM classifiers.

By assigning a label value to each bit of a PAM4 symbol, a pair of labels can represent a category of PAM4, and then the label pairs are mapped to the corresponding four category labels according to a specific rule. With this mapping rule, using two SVM models for signal classification, SVM1 determines the positive and negative signs of the level values, while SVM2 classifies the high and low level values [18].

Based on the classification strategy, we propose a modified SVM classifier, referred to as 2ary-SVM. The structure of 2ary-SVM is depicted in Figure 3.

The decision function of 2ary-SVM for classifying a test sample is formulated as follows:

$$\begin{array}{c} {\mathrm{f}}_{\mathrm{k}}\left(x_j^e\right)={\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{n}}}{\mathsf{\alpha }}_{\mathrm{i}}{\mathrm{y}}_{\mathrm{i}}^{\mathrm{k}}\mathrm{K}\left(x_j^e,x_i^r\right)+\mathrm{b}, \mathrm{k}=\mathrm{1,2}, 1\le \mathrm{j}\le \mathrm{m}\end{array}$$

(3)

where ${\mathsf{\alpha }}_{\mathrm{i}}$ is the Lagrange parameter of training dataset T, and $K\left(x_j^e,x_i^r\right)$ is the kernel function. In this case, the Gaussian radial bias kernel function is adopted. Subsequently, the corresponding support vectors (training samples with ${\mathsf{\alpha }}_{\mathrm{i}}\ne 0$) are output. The two trained SVM models output the labels of samples $y_j^{e1}$ and $y_j^{e2}$, respectively. These two labels form a label pair for a test sample, denoted as ${(y}_j^{e1},y_j^{e2})$. This label pair is then mapped to the four category labels corresponding to the four levels, with the specific mapping determined by the following equation:

$$\begin{array}{c} y_j^e=1+{\displaystyle \sum _{k=1}^2}\left(1+y_j^{ek}\right)·2^{1-k} \mathrm{k}=\mathrm{1,2},1\le j\le m\end{array}$$

(4)

For instance, if the label pair output by the two SVMs for a test sample is (+1, −1), the category label for this test sample can be computed as $y^e=3,$ according to Equation (4).

Despite the effectiveness of 2ary-SVM, the generalization capability of the SVM model may decline with increasing amplitude fluctuations in the signals. To address this issue, we further propose augmenting 2ary-SVM with the W-KNN algorithm owing to its simplicity and flexibility. The W-KNN algorithm avoids the complexities associated with extensive training processes and does not require large datasets, as is often necessary when training neural networks. Moreover, it mitigates the misleading effects of the standard KNN algorithm, which treats all dimensions equally [17]. Consequently, we propose the two-stage decision algorithm, KNN-aided SVM. The structure of the KNN-aided SVM algorithm is illustrated in Figure 4.

The KNN-aided SVM algorithm consists of two stages. In the first stage, PAM4 signals are classified using 2ary-SVM. The second stage utilizes the W-KNN algorithm to refine the classification for samples that are easily misclassified and near the SVM decision boundary. Here, W-KNN acts as a binary classifier for specific samples rather than as a four-class classifier for all samples. The final predicted labels are demapped into PAM4 symbols to reconstruct the original signal sequence.

During the W-KNN process, the decision value ${\mathrm{f}}_{\mathrm{k}}\left({\mathrm{x}}_{\mathrm{j}}^{\mathrm{e}}\right)$ is derived from the SVM model, defining the decision boundary where ${\mathrm{f}}_{\mathrm{k}}\left({\mathrm{x}}_{\mathrm{j}}^{\mathrm{e}}\right)=\pm 1$. Test samples for W-KNN are selected based on the condition $\left|{\mathrm{f}}_{\mathrm{k}}\left({\mathrm{x}}_{\mathrm{j}}^{\mathrm{e}}\right)\right|\le 1$, focusing on those near the decision boundary. The samples within a narrow segment near the levels of ±1 and ±3 are less likely to be misclassified and are thus suitable for the training samples of W-KNN.

The W-KNN algorithm applies distance weighting, assigning higher weights to training samples closer to the tested sample. The Euclidean distance $D_{ji}$ is calculated for all training samples, creating an array $D_j={(D_{j1},D_{j2},\dots ,D_{jn})}^T$, which is then sorted to find the smallest K distances. The K value is set to 3. Weights for these samples are calculated using $W=1/{\left(D_k\right)}^2$. The labels of training samples with the same labels are summed, and the label with the highest total weight becomes the predicted label for the tested sample. In this case, W-KNN is utilized as a binary classifier. After the classification in the KNN stage, the labels of samples misclassified in the SVM stage are updated.

2.2. Fluctuation Mitigation Based on RF-M

The original RF-M algorithm, designed for MPI mitigation, first divides the signals into four level intervals using hard decision and then performs MPI estimation and noise cancellation within each interval individually. MPI noise is estimated by determining the offset level value relative to the hard decision value, allowing the amplitude fluctuations of MPI-affected signals to be smoothed by removing the estimated noise. However, the RF-M algorithm has inherent limitations, and its initial reliance on hard decisions increases the likelihood of misclassification. As a result, the symbols obtained after subsequent calculations to eliminate MPI noise may still be incorrect, leading to an error propagation problem. To address this problem, we finally propose the KNN-aided SVM+RF-M scheme, and the two processes of the scheme are illustrated in Figure 5.

The first process is the soft decision of the PAM4 impaired signals. After classification using KNN-aided SVM, the predicted label $y_i$ for each PAM4 symbol is obtained. The second process involves fluctuation suppression using RF-M. The soft-decided PAM4 signal sequence is divided into blocks of length M, chosen from a range that minimizes the BER of the system. Each M-length block is further divided into smaller blocks of length N. The three decision thresholds in each block adapt to amplitude fluctuation. We modified the three hard decision thresholds obtained by averaging for each block in the original RF-M, aiming to first classify the signals to the correct level as accurately as possible through KNN-aided SVM. Fluctuation suppression was then applied to the smaller N-length blocks. We obtained the MPI estimation by using the average value to subtract the decision value. The interfered signals were then subtracted from the product of the MPI estimation and weighting factor to obtain samples with reduced interference. The principle of fluctuation suppression can be expressed as follows:

$$\begin{array}{c} \left\{\begin{array}{c}{MPI}_i={\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=1}^{n/2}}\left[x_{i+j}+x_{i+j-1}\right]-D_i\\ x_i=x_i-{\mu }_i{MPI}_i\end{array} \right.\end{array}$$

(5)

where $D_i$ is the decision level value of $x_i$, and ${\mu }_i$ is the weighting factor. According to our previous work in [12], it was found that the lowest BER was obtained when ${\mu }_i$ was equal to 1; so here the weighting factor was fixed at 1. After the above calculation, we obtained the smoothed signals.

3. Results and Discussion

3.1. Simulation Setup

To evaluate the effectiveness of the proposed algorithms, a PAM4-DD system was constructed using VPI Transmission Maker Optical Systems 8.7 and MATLAB R2023b. The simulation setup is illustrated in Figure 6.

At the transmitter, pseudorandom binary sequences (PRBSs) were generated and subsequently mapped into PAM4 format signals with a symbol rate of 28 GBaud. The generated PAM4 signals were then up-sampled by a factor of 2 and filtered using a root-raised cosine (RRC) filter with a roll-off factor of 1, which represented the worst-case system performance. Our focus was on algorithms aimed at mitigating MPI, specifically selecting systems that exhibit significant MPI effects to facilitate the observation of the algorithm’s effectiveness. Consequently, we configured the system’s roll-off factor to 1.

The transmission section was divided into two branches: one for transmitting signals and the other for simulating MPI interference generated by a set of reflected signals with an added delay component. The optical delay was approximately a few microseconds. A variable optical attenuator (VOA) in the upper branch was employed to adjust the intensity of the reflected signals. One performance metric, SIR, defined as the power ratio between the transmitted signal and the reflected signal, served as a measure to evaluate the impact of MPI noise. A higher SIR correlated with reduced MPI noise. Another VOA (VOA2) was used to adjust the received optical power (ROP), which was set to −5 dBm in the simulation.

At the receiver, the PAM4 signals with MPI noise were filtered using a low-pass filter and then sampled at 2 Sa/s. A feed-forward equalizer (FFE) with 55 taps was employed to compensate for the signal impairment, and the tap values were derived from a training sequence using the least mean square (LMS) algorithm. Next, MPI suppression was performed using our proposed KNN-aided SVM+RF-M scheme. Finally, the PAM4 signals were demapped and the BER is calculated.

3.2. Simulation Results

We first verified the decision results of the SVM stage in the KNN-aided SVM algorithm. Figure 7a,b show the classification results of 2ary-SVM. Moreover, in order to analyze the optimization effect of the soft decision, we also introduced the traditional direct decision for comparison. The classification results using direct decision are displayed in Figure 7c. The hard decision thresholds caused many PAM4 symbols to be incorrectly classified into the wrong level intervals, resulting in a BER of 2.1 × 10⁻³. By contrast, the BER achieved through signal decision using 2ary-SVM was 1.5 × 10⁻³, significantly lower than that of the direct decision method.

In Figure 7a,b, the blue data points represent negative examples with true label values of −1, while the red data points indicate positive examples with true label values of +1. The classification results aligned with the strategy introduced in the principles section of 2ary-SVM. Specifically, SVM1 successfully distinguished between positive and negative levels, while SVM2 differentiated between high and low levels, thereby validating the feasibility of 2ary-SVM. The black line represents the decision threshold, corresponding to the decision function ${\mathrm{f}}_{\mathrm{k}}\left(x^e\right)=0$. It is evident that the decision threshold adapted to the amplitude fluctuations of the signals affected by MPI, transitioning the signal evaluation from a hard decision to a soft decision approach. The soft decision method facilitated the allocation of as many symbols as possible to their correct level intervals, thereby enhancing accuracy compared to direct decision making. As shown in Figure 7a, the BER was 7.3 × 10⁻⁴ when using SVM1 to classify the positive and negative levels, while the BER was 2.3 × 10⁻³ when employing SVM2 to distinguish between high and low levels. The lower BER performance of SVM2, in comparison to SVM1, could be attributed to the more pronounced amplitude fluctuations caused by MPI at the high level, which negatively impacted the generalization ability of SVM2.

Then, we further validated the improvement of 2ary-SVM following the secondary decision made during the KNN stage. Figure 8a illustrates the decision results obtained after applying 2ary-SVM. It was seen that the SVM models yielded suboptimal classification results when SIR was set at 20 dB. This can be attributed to the severe MPI, which caused significant fluctuations in signal amplitude, making it challenging for the SVM model to identify the MPI damage patterns. Consequently, the decision threshold produced by the model struggled to adapt to these amplitude variations, leading to more rigid classification outcomes and, ultimately, poorer BER performance.

In Figure 8b, we present the results obtained after the secondary decision in the KNN stage. It is evident that some symbols have been corrected to their appropriate levels, particularly those near ±2. The BER decreased to 5.7 × 10⁻³, demonstrating a notable improvement over the SVM stage. These results convincingly illustrate that the KNN-aided SVM algorithm effectively mitigated MPI by accurately judging impaired signals.

Nevertheless, relying solely on soft decision for MPI suppression remained insufficient. Therefore, we explored the application of the fluctuation suppression principle of RF-M to enhance mitigation outcomes further. As shown in Figure 8c, the signal fluctuations were reduced following the removal of noise by RF-M, resulting in a BER that fell below the 7% HD-FEC threshold. It was evident that the quality of the impaired signal progressively improved across the three stages. This demonstrated the efficacy of our proposed KNN-aided SVM+RF-M scheme in suppressing MPI.

Figure 9 illustrates the variation in BER performance of signals affected by MPI noise at different SIR values. As shown in Figure 9a, both 2ary-SVM and KNN-aided SVM enhanced BER performance under optical back-to-back (BTB) conditions. KNN-aided SVM demonstrated superior optimization performance compared to 2ary-SVM.

The two blue curves in Figure 9a,b are the same case. In Figure 9a, the curve “direct decision” corresponds to the case where the signal decision uses hard decision thresholds of 0, +2, and −2 after FFE equalization, followed by PAM4 demapping and calculating the BER. In Figure 9b, the curve “without MPI mitigation” refers to the BER obtained by applying the same signal processing as above, and it also represents the case where no MPI mitigation algorithm is used. These two curves are presented in separate figures to facilitate a clearer discussion of each scenario individually.

When the linewidth was set at 10 × 10⁶ Hz using the direct decision method, the BER of MPI-impacted signals fell below the 7% HD-FEC threshold of 3.8 × 10⁻³ when the SIR reached 25 dB. Compared to the traditional direct decision, 2ary-SVM improved SIR tolerance by 1 dB, while KNN-aided SVM further enhanced SIR tolerance by 2 dB. The improved BER performance of KNN-aided SVM confirmed the effectiveness of the W-KNN-aided decision approach. The application of W-KNN for the secondary decision can effectively address the poor model generalization ability of SVM in scenarios with excessive MPI noise. In Figure 9b, we can see that our proposed scheme improved the SIR tolerance by 2 dB compared to the original RF-M scheme. The results indicated that KNN-aided KNN-aided SVM+RF-M for MPI mitigation was effective. By optimizing the signal decision method in RF-M with KNN-aided SVM, the error propagation issue inherent in the original RF-M was alleviated, resulting in enhanced BER performance.

To comprehensively evaluate the performance of the proposed scheme, the BER was analyzed by varying the ROP values while maintaining a fixed SIR of 27 dB. As illustrated in Figure 10, without any MPI mitigation method, the BER remained below the 7% HD-FEC threshold at an ROP of −12 dBm. When employing 2ary-SVM, the BER fell below the 7% HD-FEC threshold at an ROP of −13 dBm, while KNN-aided SVM achieved a BER below 3.8 × 10⁻³ at an ROP of −14 dBm. The SIR tolerance was improved by 1 dB and 2 dB in both stages of suppression, respectively, as compared to no suppression.

Furthermore, it can be seen that using only soft decision yielded ROP sensitivity similar to that of the original RF-M. However, KNN-aided SVM+RF-M improved the receiver sensitivity by about 1 dB compared to the original RF-M scheme. The integration of soft decision-aided MPI mitigation compensated for the limitations of both approaches. This combination leveraged the RF-M’s effective fluctuation mitigation capabilities alongside the machine learning algorithm’s ability to recognize patterns of MPI impairment, resulting in better BER performance.

4. Conclusions

In this paper, we present a machine learning-assisted MPI mitigation scheme, known as KNN-aided SVM+RF-M, which is validated in a 28 GBaud PAM4-DD transmission system. The proposed scheme comprises two main components: (1) A signal decision algorithm, KNN-aided SVM, which improves decision accuracy for MPI-affected PAM4 signals through two stages of classification. The first stage utilizes a novel multi-category classifier, called 2ary-SVM, which effectively recognizes MPI damage patterns and provides soft signal decisions via adaptive decision thresholds. The second stage employs W-KNN for secondary classification to correct any misclassified labels from the first stage, thus enhancing classification accuracy. (2) Fluctuation mitigation is achieved by integrating the KNN-aided SVM algorithm with the original RF-M algorithm, replacing its initial hard decision with a soft decision process followed by MPI estimation and noise elimination. The simulation results demonstrate that our proposed scheme can improve SIR tolerance by 2 dB and improve the receiver sensitivity by about 1 dB at the 7% HD-FEC threshold compared to the original RF-M scheme.