Low-Complexity Pruned Convolutional Neural Network Based Nonlinear Equalizer in Coherent Optical Communication Systems

Abstract

Nonlinear impairments caused by devices and fiber transmission links in a coherent optical communication system can severely limit its transmission distance and achievable capacity. In this paper, we propose a low-complexity pruned-convolutional-neural-network-(CNN)-based nonlinear equalizer, to compensate nonlinear signal impairments for coherent optical communication systems. By increasing the size of the effective receptive field with an 11 × 11 large convolutional kernel, the performance of feature extraction for CNNs is enhanced and the structure of the CNN is simplified. And by performing the channel-level pruning algorithm, to prune the insignificant channels, the complexity of the CNN model is dramatically reduced. These operations could save the important component of the CNN model and reduce the model width and computation amount. The performance of the proposed CNN-based nonlinear equalizer was experimentally evaluated in a 120 Gbit/s 64-quadrature-amplitude-modulation (64-QAM) coherent optical communication system over 375 km of standard single-mode fiber (SSMF). The experimental results showed that, compared to a CNN-based nonlinear equalizer with a 6 × 6 normal convolutional kernel, the proposed CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel, after channel-level pruning, saved approximately 15.5% space complexity and 43.1% time complexity, without degrading the equalization performance. The proposed low-complexity pruned-CNN-based nonlinear equalizer has great potential for application in realistic devices and holds promising prospects for coherent optical communication systems.

1. Introduction

Breakthroughs and the continuous development of emerging technologies—such as cloud computing, artificial intelligence and the mobile internet—have propelled modern society into the “Big Data Era”. The increasing demand for super-large data storage, transmission and sharing services has further driven the explosive growth of network traffic. Therefore, modern communication networks need higher transmission rates, larger transmission capacity and better transmission quality, to ensure the growing demand of network traffic [1,2,3,4,5]. Improving the capacity of optical fiber communication systems has become an urgent issue for economic and social development [6,7,8,9,10].

Coherent optical communication technology—which combines a high-order modulation format, coherent detection technology and digital signal processing technology—can achieve high spectral efficiency, long distance and high-capacity signal transmission, and is an important technology for coping with the traffic crisis in modern communication networks. However, in current high-speed coherent optical communication systems, nonlinear impairment caused by devices and fiber transmission links is the most important factor limiting the high-capacity and long-distance transmission of optical signals in higher-order modulation formats [11,12,13]. Therefore, it is of great significance to explore nonlinear equalization techniques for coherent optical communication systems.

Over the last few years, with the development of machine learning, several machine learning techniques have been explored for dealing with nonlinear equalization in optical communication systems [14,15,16,17,18,19,20,21]. Neural network algorithms have achieved excellent performance in the nonlinear equalization of optical communication systems, attributable to their powerful nonlinear mapping capabilities for inputs and outputs. Examples include the artificial neural network (ANN) [22,23], the deep neural network (DNN) [24], the convolutional neural network (CNN) [25], the recurrent neural network (RNN) [26,27,28,29], the soft deep neural network (SDNN) [30], the sparsity learning deep neural network [31], the complex-valued neural network [32], the probabilistic neural network (PNN) [33], the echo state network [34] and Bayesian neural networks (BNN) [35]. On this basis, the neural-network-aided perturbation-theory-based fiber nonlinearity compensation technique has been widely investigated and has demonstrated its effectiveness in estimating complex nonlinear distortion fields with perturbation triplets as the input features [36]. In our previous work [37], we constructed a dual-channel feature map based on perturbation triplets as the input features, and a multilayer CNN with a 6 × 6 normal convolutional kernel was used, to process the input dual-channel feature map for nonlinear equalization. Significant complexity reductions were obtained, compared to perturbative nonlinearity compensation using a fully connected neural network. However, there was scope for further optimizing the structure of the CNN, to reduce the complexity without affecting the nonlinear equalization performances.

In this paper, in order to further reduce the complexity and improve the performance of nonlinear equalization, we propose a low-complexity pruned-CNN-based nonlinear equalizer, to compensate nonlinear signal impairments for coherent optical communication systems. Our experimental verification used an 11 × 11 large convolutional kernel, to increase the size of the effective receptive field, so that the performance of feature extraction for CNNs could be enhanced and the structure of the CNN could be simplified. A channel-level pruning algorithm was utilized, to prune the insignificant channels, whereby the complexity of the CNN could be dramatically reduced. The experimental verification of the proposed pruned-CNN-based nonlinear equalizer was demonstrated in a 120 Gb/s 64 QAM coherent optical communication system with a transmission distance of 375 km. The experimental results showed that the proposed CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel outperformed the CNN-based nonlinear equalizer with a 6 × 6 normal convolutional kernel. And, by further using the channel-level pruning algorithm, the space complexity and time complexity of the proposed CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel could be further reduced. The proposed low-complexity pruned-CNN-based nonlinear equalizer showed great potential for coherent optical communication systems.

2. Principles

In this section, we present the feature map construction, applying the perturbation theory, we propose our CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel, we introduce the channel-level pruning algorithm and we analyze the corresponding complexity.

2.1. Feature Map Construction

For a polarization multiplexing coherent optical communication system, the nonlinear Schrödinger equation that describes the evolution of the optical field is as follows [38]:

$$\frac{\partial u_x(z,t)}{\partial z}+j\frac{{\beta }_2}{2}\frac{{\partial }^2{u}_x(z,t)}{\partial t^2}=j\frac{8}{9}\gamma \left[{\left|u_x(z,t)\right|}^2+|+u_y(z,t){|}^2\right]u_x(z,t);$$

(1)

$$\frac{\partial u_y(z,t)}{\partial z}+j\frac{{\beta }_2}{2}\frac{{\partial }^2{u}_y(z,t)}{\partial t^2}=j\frac{8}{9}\gamma \left[{\left|u_y(z,t)\right|}^2+|+u_x(z,t){|}^2\right]u_y(z,t),$$

(2)

where $u_x(z,t)$ and $u_y(z,t)$ are the optical fields of X and Y polarization, respectively, ${\beta }_2$ is the group velocity dispersion and $\gamma$ is the nonlinear coefficient. In the first-order perturbation theory, the solutions of Equations (1) and (2) are composed of the linear term $u_{0,x}(z,t)$ or $u_{0,y}(z,t)$ and the nonlinear perturbation term $\mathsf{\Delta }{u}_x(z,t)$ or $\mathsf{\Delta }{u}_y(z,t)$:

$$u_x(z,t)=u_{0,x}(z,t)+\mathsf{\Delta }{u}_x(z,t);$$

(3)

$$u_y(z,t)=u_{0,y}(z,t)+\mathsf{\Delta }{u}_y(z,t).$$

(4)

Assuming that the pulse spreading caused by fiber dispersion is much larger than the symbol duration, the nonlinear perturbation terms for the symbol at $t=0$ can be approximated as [39]:

$$\mathsf{\Delta }{u}_x=P_0^{3/2}\sum _{m,n}\left(X_n{X}_{m+n}^{*}{X}_m+Y_n{Y}_{m+n}^{*}{X}_m\right)C_{m,n};$$

(5)

$$\mathsf{\Delta }{u}_y=P_0^{3/2}\sum _{m,n}\left(Y_n{Y}_{m+n}^{*}{Y}_m+X_n{X}_{m+n}^{*}{Y}_m\right)C_{m,n},$$

(6)

where $P_0$ is the launched optical power, $X_n$ and $Y_n$ are symbol sequences for the X and Y polarization, m and n are symbol indices, with respect to the symbol of interest $X_0$ and $Y_0$, $C_{m,n}$ are the nonlinear perturbation coefficients. The $X_n{X}_{m+n}^{*}{X}_m+Y_n{Y}_{m+n}^{*}{X}_m$ term and the $Y_n{Y}_{m+n}^{*}{Y}_m+X_n{X}_{m+n}^{*}{Y}_m$ term are defined as the intrachannel crossphase modulation (IXPM) and intrachannel fourwave mixing (IFWM) triplets. The triplets do not depend on the link parameters and can be calculated directly from the received symbol sequences.

In this paper, we chose to apply the feature map construction method we proposed in [37], where the feature units (FUs) of received complex M-QAM symbols are defined as the triplets $X_n{X}_{m+n}^{*}{X}_m+Y_n{Y}_{m+n}^{*}{X}_m$ or $Y_n{Y}_{m+n}^{*}{Y}_m+X_n{X}_{m+n}^{*}{Y}_m$. The schematic diagram of the feature map is shown in Figure 1. Each FU of a received complex M-QAM symbol is a complex unit, corresponding to the triplet term of n and m values. Many different FUs form a feature map with two channels, Channel 1 and Channel 2, which represent the real and imaginary parts of the feature map, respectively. The superscript and subscript of FU denote the values of n and m, respectively. S denotes the maximum value of n and m. The channel size is $\left(2S+1\right)$, the side length of the square feature map.

2.2. CNN-Based Nonlinear Equalizer with an 11 × 11 Large Convolutional Kernel

For this paper, for a trade-off between the nonlinear equalization performance and the compensation complexity, for the feature map, the value of S was set to 5. Thus, the size of the corresponding input feature map for our proposed CNN-based nonlinear equalizer was 11 × 11 × 2. The input feature map not only contained the inherent connection between the real part and the imaginary part of the triplet terms, but also preserved the position relationships between different FUs.

The architecture of the proposed CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel is shown in Figure 2. The proposed CNN-based nonlinear equalizer contains four layers: an input layer; a convolution layer; a fully connected layer; and an output layer. In the input layer, the 11 × 11 × 2 feature map of a received complex M-QAM symbol is input into the convolution layer. In the convolution layer, n convolutional channels with 11 × 11 large kernel size are utilized, to capture the global input feature maps information, and the output feature map is 1 × 1 × n. Batch normalization [40] is adopted after the convolution layer, as a standard approach to achieving fast convergence and better generalization performance. The output feature map is fully connected to the fully connected layer. The number of nodes in the fully connected layer is the same as the number of classes of the M-QAM signals. The Softmax operation calculates the probabilities that the corresponding received complex M-QAM symbol maps to each class. Then, the output layer outputs the corresponding predicted class of the corresponding received complex M-QAM symbol, with the maximum probability.

An 11 × 11 large convolutional kernel is utilized to capture the global input feature maps information. As shown in Figure 3, compared to the normal 6 × 6 convolutional kernel utilized in our previous work [37], the large 11 × 11 convolutional kernel has the following advantages: (1) it has a larger effective receptive field, so that the performance of feature extraction for the convolutional layer can be significantly enhanced; (2) to obtain 1 × 1 output maps, only one convolution layer with an 11 × 11 large convolutional kernel is required, whereas with a 6 × 6 normal convolutional kernel, multiple layers of convolution are required; therefore, the structure of the CNN can be greatly simplified; (3) the effective receptive field of an 11 × 11 large convolutional kernel is large enough to cover the entire input feature map, so that the correlation between each FU in the input feature map can be better captured. Thus, the CNN model with an 11 × 11 large convolutional kernel establishes a closer connection between the input feature map and the output result.

2.3. The Channel-Level Pruning Algorithm

In order to further reduce the CNN model size and decrease the computational complexity of the CNN model, without compromising the performance of nonlinear equalization, a simple, yet effective, network training scheme [41] is applied to our proposed CNN-based nonlinear equalizer. This simple training scheme can achieve channel-level sparsity in the CNN model by leveraging the scaling factors of each channel, to effectively identify and prune unimportant channels in the CNN model.

The idea of this channel-level pruning method is introducing a scaling factor $\gamma$ for each channel, then jointly training the network weights and these scaling factors with sparsity regularization, pruning the channels with small factors and fine-tuning the pruned network, as shown in Figure 4. Scaling factors $\gamma$ act as the agents for channel selection: as they are jointly optimized with the network weights, the network can automatically identity insignificant channels, which can be safely removed without greatly affecting the performance. When we prune the channels of CNN models trained with channel-level sparsity, the channels of CNN models will be sorted from largest to smallest, according to the scaling factors $\gamma$, and then the bottom $\alpha n$ channels with smaller scaling factors $\gamma$ will be pruned. The pruned percentage is denoted by $\alpha$, which is a manually set percentile among all channels.

The training objective in this training scheme is as follows:

$$L=\sum _{\left(X,y\right)}l\left(f\left(X,W\right),y\right)+\lambda \sum _{\gamma \in \mathsf{\Gamma }}g\left(\gamma \right),$$

(7)

where $\left(X,y\right)$ denotes the input and target of the training dataset, W denotes the trainable weights of the CNN model, the first sum term corresponds to the normal training loss of CNN (cross entropy loss, in this paper), $g\left(·\right)$ is a sparsity-induced penalty on the scaling factors ($g\left(\gamma \right)=\left|\gamma \right|$, L1-norm, in this paper) and $\lambda$ balances the two terms. Subgradient descent is adopted as the optimization method for the non-smooth L1 penalty term.

Pruning a channel essentially corresponds to removing all the incoming and outgoing connections of that channel: by doing so, a more compact network with less parameters and computing operations can be obtained. A fine-tuning process on the pruned CNN model can partly compensate the performance loss caused by channel pruning.

2.4. Complexity Analysis

We analyze the complexity of the proposed CNN-based nonlinear equalizer before and after pruning. We focus on analyzing the complexity in two aspects: the number of parameters of the CNN-based nonlinear equalizer, and the number of multiplications required for the CNN-based nonlinear equalizer to equalize per M-QAM symbol.

(1) The number of parameters of the CNN-based nonlinear equalizer before pruning mainly comprises the parameters of the convolution layer and the parameters of the fully connected layer. The number of parameters for the convolution layer is the number of weight parameters of the convolutional kernels, $N_{P\_C\_unp}=C_{in}\times C_{out}\times K$. The number of parameters for the fully connected layer is $N_{P\_F\_unp}=C_{out}\times M$. Thus, the number of parameters of the CNN-based nonlinear equalizer before pruning is $N_{P\_unp}=C_{in}\times C_{out}\times K+C_{out}\times M$. $C_{in}$ and $C_{out}$ denote the numbers of the input channels and output channels, respectively, while K denotes the convolution kernel size and M denotes the number of nodes in the fully connected layer.

(2) The number of parameters of the CNN-based nonlinear equalizer after pruning mainly includes the parameters of the convolution layer and the parameters of the fully connected layer. The number of parameters for the convolution layer is the number of weight parameters of the convolutional kernels, $N_{P\_C\_p}=C_{in}\times C_{out}\times \alpha \times K+C_{out}$. The number of parameters for the fully connected layer is $N_{P\_F\_p}=C_{out}\times \alpha \times M$. Thus, the number of parameters of the CNN-based nonlinear equalizer after pruning is $N_{P\_p}=C_{in}\times C_{out}\times \alpha \times K+C_{out}+C_{out}\times \alpha \times M$. The pruned percentage is denoted by $\alpha$.

(3) The number of multiplications required for the CNN-based nonlinear equalizer before pruning, to equalize per M-QAM symbol, mainly comprises the multiplications of the convolution layer and the multiplications of the fully connected layer. The number of multiplications for the convolution layer is $N_{M\_C\_unp}=C_{in}\times C_{out}\times K$. The number of multiplications for the fully connected layer is $N_{M\_F\_unp}=C_{out}\times M$. Thus, the number of multiplications of the CNN-based nonlinear equalizer before pruning is $N_{M\_unp}=C_{in}\times C_{out}\times K+C_{out}\times M$.

(4) The number of multiplications required for the CNN-based nonlinear equalizer after pruning to equalize per M-QAM symbol mainly includes the multiplications of the convolution layer and the multiplications of the fully connected layer. The number of multiplications for the convolution layer $N_{M\_C\_p}=C_{in}\times C_{out}\times \alpha \times K$. The number of multiplications for the fully connected layer is $N_{M\_F\_p}=C_{out}\times \alpha \times M$. Thus, the number of multiplications of the CNN-based nonlinear equalizer before pruning is $N_{M\_p}=C_{in}\times C_{out}\times \alpha \times K+C_{out}\times \alpha \times M$.

3. Experimental Setup

The experimental setup for a 120 Gb/s 64-QAM coherent optical communication system with a transmission distance of 375 km is depicted in Figure 5.

At the transmitter, the nominal linewidth of the external cavity laser (ECL) is 100 kHz. Symbol sequences of the 64-QAM signals generated by the MATLAB program are uploaded to an arbitrary waveform generator (AWG) with a sampling rate of 25 GSa/s. Two electric amplifiers (EA) are used to amplify two analog signals. Then, the two amplified analog signals are sent into the inphase/quadrature (I/Q) modulator. Next, a modulated optical signal is generated by the I/Q modulator. The polarization multiplexing of the signal is completed by a polarization-division multiplexing (PDM) module. The PDM module is composed of a polarization-maintaining optical coupler (PM-OC), an optical delay line, a polarization controller (PC) and a polarization beam combiner (PBC). Then, the optical signal is amplified by an Erbium-doped fiber amplifier (EDFA), and the optical signal power can be adjusted by a variable optical attenuator (VOA).

The transmission link consists of 5-span G.652D single-mode fiber (SMF), with a length of 75 km. An EDFA is used to compensate the fiber loss at the end of each span.

At the receiver, the local oscillator (LO) for coherent detection is an ECL with 100 kHz linewidth. An LO, two polarization beam splitters (PBSs), two 90-degree optical hybrids and four balanced photodetectors (BPDs) form an optical polarization and phase-diversity coherent receiver front end. The X- and Y-polarization components of the received optical signal and the local oscillator are separately combined and detected by two identical phase-diversity receivers. A 90-degree optical hybrid and two BPDs form a phase-diversity receiver. Then, the signals are digitized by a 4-channel digital phosphor oscilloscope (DPO) with a 100 GSa/s sampling rate.

The offline DSP consists of a low-pass filter, amplitude normalization, chromatic dispersion compensation (CDC), clock recovery, resampling, the Gram–Schmidt orthogonalizing process (GSOP), constant-modulus-algorithm-(CMA) equalization, frequency offset estimation (FOE), carrier phase estimation (CPE) based on blind phase search, a CNN-based nonlinear equalizer (CNN-based NLE) and 64-QAM demapping.

In our experiment, the measured launched optical power ranged from −4 dBm to 5 dBm. The datasets of each launched optical power contained approximately 2${}^{20}$ symbols. The whole dataset was divided into training data (80%) and testing data (20%). The CNN models were built, trained and evaluated in Pytorch 1.6.0. Furthermore, in the training process, structural reparameterization [42] was utilized, in order for the CNN model to achieve better feature representation.

4. Results and Discussion

The number of output channels of the convolution layer is an important parameter influencing the equalization performance and computational complexity of the proposed CNN-based nonlinear equalizer. Figure 6a presents the BER performance of the proposed CNN-based nonlinear equalizer shown in Figure 2, with different numbers of output channels of the convolution layer. The launched optical power was 1 dBm. As shown in the figure, when the number of output channels exceeded 160, the proposed CNN-based nonlinear equalizer achieved BER below the HD-FEC limit of 3.8 × 10${}^{-3}$; when the number of output channels exceeded 310, the proposed CNN-based nonlinear equalizer achieved BER below 1.0 × 10${}^{-3}$. The more output channels in the convolutional layer, the more convolutional kernels there are, and the stronger the ability to globally extract information from the input feature map simultaneously. However, too many output channels of the convolution layer would result in a redundant CNN model structure and excessive computational complexity.

The BER performance of the proposed CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels was similar to that of a CNN-based nonlinear equalizer with a 6 × 6 normal convolutional kernel [37], when the launched optical power was 1 dBm. Figure 6b presents the BER performance of the proposed pruned-CNN-based nonlinear equalizer of 270 output channels with different pruned percentages. It can be seen that, when the pruned percentage was 15%, which means that 15% of the channels of the CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel were pruned, the BER results changed from 1.23 × 10${}^{-3}$ to 1.28 × 10${}^{-3}$, a degree of BER performance which is acceptable.

Figure 7 presents the BER performance of a CNN-based nonlinear equalizer with a 6 × 6 normal convolutional kernel [37], the proposed unpruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels and a 15%-pruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels. The BER results represented by the blue line are the BER performance of the minimum-Euclidean-distance-(MED)-based decision. The BER results represented by the orange line (A) are the BER performance of the CNN-based nonlinear equalizer with a 6 × 6 normal convolutional kernel [37]. The BER results represented by the yellow line (B) are the BER performance of the proposed unpruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels. The BER results represented by the purple line (C) are the BER performance of the 15%-pruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels. As shown in the figure, both the proposed unpruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels and the 15%-pruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels outperformed the CNN-based nonlinear equalizer with a 6 × 6 normal convolutional kernel.

The computational complexity of the CNN-based nonlinear equalizer with a 6 × 6 normal convolutional kernel [37] (orange columns (A)), the proposed unpruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels (yellow columns (B)) and the 15%-pruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels (purple columns (C)), with similar BER performances (1.284 × 10${}^{-3}$, 1.239 × 10${}^{-3}$, 1.283 × 10${}^{-3}$, respectively, when the launched optical power was 1 dBm) are presented in Figure 8. Space complexity represents the number of parameters for the CNN-based nonlinear equalizer, and time complexity represents the number of multiplications required for the CNN-based nonlinear equalizer to equalize per symbol. Lower space complexity means less memory space is taken up, and lower time complexity means less running time is required.

It can be seen that, compared to the CNN-based nonlinear equalizer with a 6 × 6 normal convolutional kernel, the proposed unpruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels saved about 33% time complexity. Furthermore, the 15%-pruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel and 270 output channels saved about 15.5% space complexity and 43.1% time complexity. The proposed pruned-CNN-based nonlinear equalizer requires less memory occupation and less running time, has great potential for application in realistic devices and is suitably matched for coherent optical communication systems with higher rate, longer distance and higher capacity requirements.

5. Conclusions

In this paper, we propose a low-complexity pruned-CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel. An 11 × 11 large convolutional kernel can better capture the global input feature maps information, and the insignificant channels of the CNN model are pruned through training with channel-level sparsity. Then, the important component of the CNN model can be saved, and the model size and computation amount can be reduced. An experimental 120 Gb/s 64-QAM coherent optical communication system with a transmission distance of 375 km was built, to verify the performance of the proposed low-complexity pruned-CNN-based nonlinear equalizer. The experimental results showed that, compared to a CNN-based nonlinear equalizer with a 6 × 6 normal convolutional kernel, the proposed CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel saved about 33% time complexity, while its achieved BERs were similar. Furthermore, the complexity of the proposed CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel after channel-level pruning was further reduced: about 15.5% space complexity and 43.1% time complexity could be saved without degrading the equalization performance. This could ensure the application of the proposed low-complexity pruned-CNN-based nonlinear equalizer in realistic devices with less memory occupation and less running time. The proposed low-complexity pruned-CNN-based nonlinear equalizer has great potential for coherent optical communication systems, and promises to have a positive impact on the further development of modern communication networks.