Multicore Photonic Complex-Valued Neural Network with Transformation Layer

Abstract

Photonic neural network chips have been widely studied because of their low power consumption, high speed and large bandwidth. Using amplitude and phase to encode, photonic chips can accelerate complex-valued neural network computations. In this article, a photonic complex-valued neural network (PCNN) chip is designed. The scale of the single-core PCNN chip is limited because of optical losses, and the multicore architecture of the chip is used to improve computing capability. Further, for improving the performance of the PCNN, we propose the transformation layer, which can be implemented by the designed photonic chip to transform real-valued encoding to complex-valued encoding, which has richer information. Compared with real-valued input, the transformation layer can effectively improve the classification accuracy from 93.14% to 97.51% of a 64-dimensional input on the MNIST test set. Finally, we analyze the multicore computation of the PCNN. Compared with the single-core architecture, the multicore architecture can improve the classification accuracy by implementing larger neural networks and has better phase noise robustness. The proposed architecture and algorithms are beneficial to promote the accelerated computing of photonic chips for complex-valued neural networks and are promising for use in many applications, such as image recognition and signal processing.

1. Introduction

In recent years, deep learning [1] has developed rapidly and is widely used in many fields, such as in the optimization of organic solar cells [2] and the design of subwavelength photonic devices [3]. The development of deep learning is inseparable from the support of computing chips, such as the central processing unit (CPU), graphics processing unit (GPU) and tensor processing unit (TPU). As Moore’s Law is slowing down, electronic chips have encountered more and more serious problems, such as high power consumption. Compared with electronic chips, photonic neural network chips have the advantages of low power consumption, high computation speed, large bandwidth and high parallelism and have been widely studied in recent years. There are many different schemes for photonic neural network chips. Shen et al. demonstrated a Mach–Zehnder interferometer (MZI)-based photonic processor and used it for vowel recognition [4]. Tait et al. reported silicon photonic neural networks based on microring weight banks [5]. Xu et al. demonstrated an optical coherent dot-product chip and accomplished the image reconstruction task [6]. On et al. proposed the photonic matrix multiplication accelerator based on wavelength division multiplexing and coherent detection [7].

Unlike real-valued neural networks, complex-valued neural networks [8] use complex numbers for data processing and have many applications in image processing, audio signal processing and radar signal processing. In electronic chips, complex numbers need to be encoded with two real numbers, which will increase parameter and computation complexity. Light has multiple dimensions, such as wavelength, amplitude, phase and polarization. The calculation of complex-valued neural networks can be efficiently implemented by using the amplitude and phase of light to encode signals [9]. Compared with real-valued neural networks, complex-valued neural networks can achieve higher classification accuracies in image recognition tasks [10]. Images are usually encoded with real values. However, complex-valued images based on the Fourier transform can effectively improve classification accuracy of the complex-valued neural network [11]. Fourier transform processing is also used for photonic neural networks [12]. Lenses can efficiently realize the Fourier transform, but the integration of lenses and photonic chips is very complex [13]. On the other hand, chip-level Fourier transform structures are usually different from those of photonic neural network chips [14].

In this article, the photonic complex-valued neural network (PCNN) chip was designed including input, weights and output parts. The photonic chip can realize complex number computations by changing the phase and amplitude of light. Though photonics continues developing, the scale of photonic integration chips is still relatively small because of optical losses and device noise, especially compared with the scale of neural networks, which have millions of parameters. Therefore, the multicore PCNN architecture is used to improve the computing parallelism and computing capability of photonic chips. For improving classification accuracy of the PCNN, the transformation layer is proposed that can convert real-valued images into complex-valued images. On the MNIST test set, the classification accuracy of the 64-dimensional transformation layer is 97.51%, which is almost the same as the accuracy of the Fourier transform-based complex-valued inputs and 4.37% higher than the accuracy of real-valued inputs. The PCNN chip has plenty of phase shifters, and phase noise will decrease the performance of PCNNs. We analyze the effect of phase noise on the single-core architecture and the multicore architecture. The larger single-core PCNN has worse phase noise robustness, and the multicore PCNN with more cores and a smaller single core has better phase noise robustness. The main contributions of this article are as follows:

• Using the multicore PCNN architecture to improve computing capability;
• Proposing the transformation layer, which can be implemented by the designed PCNN chip for improving performance of the PCNN;
• Analyzing the effect of phase noise on the multicore PCNN.

The remainder of this article is organized as follows: In Section 2, we introduce architectures of the PCNN chip and the multicore PCNN. In Section 3, we describe the transformation layer, calculate optical losses of the single-core PCNN chip and analyze phase noise robustness of the PCNN. Section 4 compares the multicore PCNN with other photonic neural networks, and Section 5 concludes the article.

2. Multicore Photonic Complex-Valued Neural Network

2.1. Photonic Complex-Valued Neural Network Chip

The architecture of the PCNN chip is shown in Figure 1a. The chip is composed of three parts, which are neural network input, neural network weights and calculation output. The MZI, 2 × 2 MZI and balanced detectors are basic units of the PCNN chip. Detailed variables of different parts are shown in

Table 1. Variables of the PCNN chip.

Part of PCNN Chip	Variable	Description
Neural network weights	$M$	Complex-valued matrix
	$U,V^{*}$	Unitary matrix
	$\Sigma$	Diagonal matrix
	$T_{\mathrm{MZI}}$	Unitary transformation of 2 × 2 MZI
	$\theta$	Inner phase shift value of 2 × 2 MZI
	$\phi$	outer phase shift value of 2 × 2 MZI
Calculation output	$E_s$$(E_l$)	Signal light(Reference light)
	$A_s$$(A_l$)	Amplitude of signal light(reference light)
	${\varphi }_s$$({\varphi }_l$)	Phases of signal light(reference light)
	$w$	Frequency of signal light and reference light
	$\mathsf{\Delta }I$	Output photocurrent of balanced detectors

.

The input light is split into reference light and multiple signal light. Signal light and reference light transmit into the input part and output part, respectively. The input part uses MZIs and phase shifters to encode the input of the neural network, including amplitude encoding and phase encoding, as shown in Figure 1b. The weights part can realize an arbitrary complex-valued matrix through singular value decomposition,

$$M=U\Sigma V^{*}.$$

(1)

The complex-valued matrix $M$ is decomposed into the product of two unitary matrices $U$, $V^{*}$ and a diagonal matrix $\Sigma$. The unitary matrix can be realized by a cascaded array of 2 × 2 MZIs and phase shifters. Figure 1a shows a rectangular mesh [15] of 2 × 2 MZIs. Figure 1c is a 2 × 2 MZI including two phase shifters. The inner phase shifter (phase shift value $\theta$) changes the amplitude of light, and the outer phase shifter (phase shift value $\phi$) changes the phase of light. The 2 × 2 MZI can realize unitary transformation $T_{\mathrm{MZI}}$,

$$T_{\mathrm{MZI}}=i{e}^{\frac{i\theta }{2}}\left[\begin{array}{cc}{e}^{i\phi }\sin \left(\frac{\theta }{2}\right)& \cos \left(\frac{\theta }{2}\right)\\ e^{i\phi }\cos \left(\frac{\theta }{2}\right)& -\sin \left(\frac{\theta }{2}\right)\end{array}\right].$$

(2)

The diagonal matrix can be realized by a column of MZIs. The balanced detectors in Figure 1d can detect amplitude and phase of the light. The signal light $E_s$ from weights part is

$$E_s\left(t\right)=A_s{e}^{i\left(wt+{\varphi }_s\right)},$$

(3)

and the reference light $E_l$ from beam splitters is

$$E_l\left(t\right)=A_l{e}^{i\left(wt+{\varphi }_l\right)},$$

(4)

where $A_s, A_l$ are amplitudes of the light, ${\varphi }_s, {\varphi }_l$ are phases of the light, and $w$ is the frequency of the light. The output photocurrent $\mathsf{\Delta }{I}_1$ of balanced detectors is

$$\mathsf{\Delta }{I}_1=I_1-I_2={\left|E_s+E_l\right|}^2-{\left|E_s-E_l\right|}^2=2A_s{A}_l\cos \left(\mathsf{\Delta }\varphi \right),$$

(5)

$$\mathsf{\Delta }\varphi ={\varphi }_s-{\varphi }_l.$$

(6)

After adding π/2 phase shift to the reference light, the output photocurrent $\mathsf{\Delta }{I}_2$ is

$$\mathsf{\Delta }{I}_2=I_3-I_4={\left|E_s+i{E}_l\right|}^2-{\left|E_s-i{E}_l\right|}^2=2A_s{A}_l\sin \left(\mathsf{\Delta }\varphi \right).$$

(7)

Then the phase and the amplitude of light can be calculated [9]. Balanced detectors array is used to realize the output of complex numbers. The designed PCNN chip can parallel process multichannel data.

2.2. Multicore Architecture of Photonic Chip

In recent years, the scale of photonic chips has continued to increase. Photonic neural network chips based on MZIs have grown from 4 × 4 [4] to 64 × 64 [16], which has thousands of photonic devices. However, due to various factors, such as optical losses and device noise, the scale of photonic chips is relatively small, and the computing capability of single-core photonic neural network chips is limited. The multicore architecture of photonic chips can be used to improve computation capability. As shown in Figure 2a, the multicore architecture consists of multiple photonic and electronic chips. PCNN chips realize the accelerated computing of matrices of complex-valued neural networks. The PCNN chip is connected to a digital-to-analog converter (DAC) and an analog-to-digital converter (ADC), which realize the conversion of digital signals and analog signals, i.e., conversion of optical signals and electrical signals. The activation units realize nonlinear functions calculations in the electrical domain, and memories store input data, weights data and output data. PCNN chips and electronic chips can be fabricated in different complimentary metal-oxide semiconductor (CMOS) process nodes and integrated by 2.5D or 3D integration as shown in Figure 2b. PCNN chips and electronic chips are packaged on an interposer in 2.5D integration, and they are vertically packaged in 3D integration [17].

2.3. Multicore Photonic Complex-Valued Neural Network

As shown in Figure 3a, the PCNN includes an input layer, one or more hidden layers and an output layer. For the i-th hidden layer,

$$h_{yi}=f_i\left(W_i\times h_{xi}\right),$$

(8)

$h_{xi}$, $h_{yi}$, $W_i$ and $f_i$ are input, output, weights and nonlinear activation functions of the hidden layer, respectively. Unlike traditional neural networks, matrices computation of the PCNN is realized by adjusting values of phase shifters ($\theta , \phi$) in Figure 1a. The schematic diagram of the multicore matrix calculation is shown in Figure 3b. Weights matrix $W$ of the neural network can be split into multiple small matrices, $W_{11}$, $W_{12}$, $W_{21}$ and $W_{22}$, which are calculated by different PCNN chips. Correspondingly, the input is split, and the output of multiple chips can be obtained.

3. Results

3.1. Photonic Complex-Valued Neural Network with Transformation Layer

We used Tensorflow and Neurophox [18] to simulate the PCNN. In Figure 4a, a two-layer PCNN is constructed. The nonlinear activation function is CReLU, the loss function is the mean square error, and the optimizer is the Adam optimizer [19]. The neural network is trained on the MNIST dataset. The MNIST dataset contains ten handwritten numbers from 0 to 9. It is split into the training set and test set, which have 60,000 and 10,000 images, respectively. Dataset preprocessing has an important influence on the performance of neural networks. Recently, the Fourier transform-based image preprocessing shown in Figure 4b has been used for photonic neural networks. Figure 4c shows the classification accuracy of a PCNN with a 64-dimensional input on the MNIST training set. After trained for 100 epochs, the PCNN based on Fourier transform preprocessing can achieve 98.01% accuracy, while the PCNN trained on the original MNSIT dataset can only achieve 94.35% accuracy. The classification accuracy on the MNIST test set is shown in Figure 4d. The test accuracy of the Fourier transform preprocessing-based PCNN is 4.37% higher than that of the PCNN with the original MNIST test set.

Using photonic devices to implement the Fourier transform is fast and very energy efficient. However, the architecture of the photonic Fourier transform is quite different from that of a designed PCNN chip. A transformation layer that can transform real-valued encoding to complex-valued encoding is proposed. Furthermore, the transformation layer can be implemented by the designed PCNN chip through singular value decomposition. Figure 5a is the schematic diagram of the transformation layer. It is a single-layer neural network without activation functions. The input is real-valued images of the MNIST dataset, and the target is the Fourier transform-based complex-valued images of the MNIST dataset. The loss function of the single-layer neural network is the complex-valued mean square error, and the optimizer is Adam.

Figure 5b shows the loss function curve of the transformation layer on the MNIST training set. After 200 epochs, the loss function value is less than 10⁻⁴, and the loss function value on the test set is 1.45 × 10⁻⁵. Figure 5c shows the amplitude and phase images of the Fourier transform, the transformation layer and difference between them. The maximum value of the amplitude difference image is less than 0.01% of the amplitude image, and the maximum value of the phase difference image is less than 0.4% of the phase image. The transformation layer can achieve a precise fit. The classification accuracies of the PCNN with a transformation layer on the MNIST training set and test set are shown in Figure 4c,d. The PCNN with a transformation layer can achieve 97.51% test accuracy, which is almost equivalent to that of the Fourier transform and much better than that of the original MNIST dataset. Therefore, the transformation layer of optical domain can be used to replace the Fourier transform in the applications of PCNNs.

3.2. Multicore Architecture Analysis

The N × N PCNN with the transformation layer, which has an N × N complex-valued weights matrix, is trained on the MNIST and Fashion-MNIST datasets. The Fashion-MNIST dataset is similar to the MNIST dataset, and it has ten types of clothing. The classification accuracies of different sizes of single-core N × N PCNNs are shown in Figure 6a,b. Larger PCNNs have more parameters and can achieve higher classification accuracies. The accuracies of the 128 × 128 PCNN on the MNIST and Fashion-MNIST test sets are 98.12% and 88.95%, respectively, which is much better than 90.33% and 79.45% of the 16 × 16 PCNN.

However, the integration of photonic circuits is limited due to the optical losses of photonic devices. For the PCNN chip shown in Figure 1a, the losses of splitters of input part and the MZI arrays of weights parts are most influential [20] and are analyzed.

$$L_{splitter}=I{L}_{splitter}+E{L}_{splitter}.$$

(9)

$$\begin{array}{cc}{L}_{MZIs array}& =\left(2N+1\right)\times L_{MZI}\hfill \\ & =\left(2N+1\right)\times \left(2\times E{L}_{weights splitter}+L_{wg}\times Lengt{h}_{MZI}\right).\hfill \end{array}$$

(10)

$L_{splitter}$ is the loss of splitters of the input part including insertion loss $I{L}_{splitter}$ and excess loss $E{L}_{splitter}$. The $I{L}_{splitter}$ of a PCNN chip with N-dimensional input is 10log₁₀N. The loss of the MZI arrays of weights parts, $L_{MZIs array}$, depends on the number of MZIs through which light travels and the loss of a MZI $L_{MZI}$. The weights part of an N × N PCNN chip has two rectangular meshes of MZIs and a column of MZIs. The optical depth [15] of rectangular meshes of MZIs and a column of MZIs is N and 1, respectively. Every 2 × 2 MZI in the weights parts consists of two thermo-optic phase shifters and two splitters. The loss of the phase shifter is negligible [21], and the losses of the 2 × 2 splitters and waveguide are considered. For simplicity, the length of a 2 × 2 MZI is the sum of the lengths of two phase shifters. The device parameters are given in

Table 2. Device parameters used for loss analysis of the PCNN chip.

Device Parameter	Variable	Value	Refs
Input splitter insertion loss	$I{L}_{splitter}$	10log10N
Input splitter excess loss	$E{L}_{splitter}$	0.1 dB	[22]
Waveguide loss	$L_{wg}$	0.43 dB/cm	[23]
MZI length	$Lengt{h}_{MZI}$	640 μm	[21]
Weights splitter excess loss	$E{L}_{weights splitter}$	0.1 dB	[24]
Laser power	$P_{laser}$	10 dBm
Photodetector responsivity	$R$	0.835 A/W	[23]
Photodetector dark current	$I_d$	2.58 nA

, and the loss of the PCNN chip is shown in Figure 7. Furthermore, the photodetector (PD) current is calculated when the laser power is 10 dBm. The detailed parameters of PD are shown in Table 2. It can be seen from Figure 7 that the photocurrent of a 128 × 128 PCNN chip is less than the dark current of PD. If the chip scale exceeds the limitation of optical losses, the single-core PCNN chip can no longer complete signal detection and neural network calculations.

It is more effective to improve computing capability and classification accuracy by the multicore architecture because the scale of a single-core PCNN chip is limited.

Table 3. Classification accuracies of different sizes of single-core and multicore PCNNs on the MNIST and Fashion-MNIST test sets.

Size of Photonic Complex-Valued Neural Network		Classification Accuracy
		MNIST	Fashion-MNIST
Single-core	32 × 32	95.76%	82.60%
	64 × 64	97.51%	87.01%
	128 × 128	98.12%	88.95%
Multicore	4-core, 64 × 64/core	98.07%	88.96%
	16-core, 32 × 32/core	97.95%	88.96%

shows the classification accuracies of different sizes of single-core and multicore PCNNs with the transformation layer on the MNIST and Fashion-MNIST test sets. The classification accuracy of the multicore N × N PCNN is higher than the single-core N × N PCNN. For example, the classification accuracy of a 4-core 64 × 64 (per core) PCNN is 88.96% on the Fashion-MNIST test set, while the classification accuracy of a single-core 64 × 64 PCNN is 87.01%. PCNNs of similar sizes can achieve approximate accuracy, such as single-core 128 × 128, 4-core 64 × 64 and 16-core 32 × 32 PCNNs, but 4-core 64 × 64 and 16-core 32 × 32 PCNNs are more practical.

The noise of photonic chips will decrease the performance of photonic neural networks [25]. The designed PCNN chip, especially the weights part, contains a lot of phase shifters, which are susceptible to thermal noise. The phase noise is modelled as Gaussian noise $G\left(0, {\sigma }^2\right)$ with a mean 0 and a standard deviation $\sigma$ [26]. Figure 8a,b show the classification accuracies of single-core PCNNs on the MNIST and Fashion-MNIST test sets, respectively. The standard deviation σ of phase noise ranges from 0 to 0.1 rad. For every standard deviation, phase noise was sampled ten times, and the mean and standard deviations of the classification accuracy were calculated. From Figure 8a,b it can be seen that the larger single-core PCNN is more seriously affected by phase noise. When the σ is 0.08 rad, the 32 × 32 PCNN has the highest mean accuracy of 36.33% on the MNIST test set, and the 128 × 128 PCNN is 15.73%. We further analyzed the effect of phase noise on the multicore PCNN. Figure 8c,d show multicore PCNNs with phase noise that are tested on the MNIST and Fashion-MNIST test sets. Compared with the single-core PCNN, the multicore PCNN has better phase noise robustness. With the same phase noise, the multicore PCNN with more cores and a smaller single core has a better performance. For example, when the standard deviation of phase noise is 0.06 rad, the mean classification accuracies of 1-core, 4-core and 16-core on the MNIST test set are 39.48%, 62.71% and 77.26%, respectively.

4. Discussion

Table 4. Comparison of the multicore PCNN with photonic neural networks in the literature. PUNN: photonic unitary neural network.

	This Work			[9]		[18]
Neural Network	PCNN		Multicore PCNN	PCNN		PUNN
Size	64 × 64		4-core, 64 × 64/core	8 × 8		36 × 36
Dataset	MNIST		MNIST	MNIST		MNIST
Preprocessing	Original dataset	Transformation layer	Transformation layer	Real encoding	Complex encoding	Fourier transform
Classification Accuracy	93.14%	97.51%	98.07%	91.00%	93.50%	96.60%

shows a comparison of the multicore PCNN with other photonic neural networks in terms of neural network types, neural network sizes, datasets, preprocessing and classification accuracies. Weights matrices of the PCNN are arbitrary complex-valued matrices, and those of the photonic unitary neural network (PUNN) are unitary matrices. The unitary matrix is a complex-valued square matrix, and its inverse is also its conjugate transpose. All photonic neural networks are tested on the MNIST test set.

Compared with the real-valued input of the original MNIST dataset, complex-valued input has richer input information and effectively improves the classification accuracy of the PCNN. Complex-valued images based on the Fourier transform are used in [9] and [18]. The Fourier transform can be implemented by Fourier optics, but components of Fourier optics are different from those of PCNNs. The transformation layer proposed in this work can also improve the performance of the PCNN and has better compatibility with the designed PCNN chip.

Larger photonic neural networks can achieve higher classification accuracies. Accuracies of the 8 × 8 PCNN, the 36 × 36 PUNN and the 64 × 64 PCNN are 93.50%, 96.60% and 97.51%, respectively. However, larger photonic neural networks have worse phase noise robustness as reported in [18] and shown in Figure 8. The photonic chips, which are used for calculating photonic neural networks in Table 4, have many MZIs. The MZIs are arranged in a rectangular or triangular mesh, and noise of phase shifters is also cascaded. The larger single-core photonic neural network requires more photonic devices to perform calculations and is less robust against phase noise. Unlike the single-core architecture analyzed above, the phase noise of the multicore architecture consists of cascaded noise of the single core and the accumulated noise of multiple cores. The multicore PCNN is more robust against phase noise as shown in Figure 8. Moreover, the optical losses of every photonic core in the multicore architecture are relatively low. The multicore architecture can be used for more complex calculations, and the multicore PCNNs have the best performance (98.07% accuracy) compared with other photonic neural networks in Table 4.

5. Conclusions

In summary, a PCNN chip was designed, and the PCNN with a transformation layer, which effectively improves image classification accuracy, was proposed. A single-core PCNN chip measuring 128 × 128 or larger cannot work because of optical losses. Therefore, the multicore architecture is used for more powerful computing capability. PCNNs were tested on the MNIST test set and Fashion-MNIST test set. Furthermore, the effect of phase noise on PCNNs was analyzed. The multicore PCNN has higher classification accuracy and better phase noise robustness than the single-core PCNN. The multicore PCNN is promising for use in image processing, signal processing and other applications.