Design of Diffractive Neural Networks for Solving Different Classification Problems at Different Wavelengths

Abstract

We consider the problem of designing a diffractive neural network (DNN) consisting of a set of sequentially placed phase diffractive optical elements (DOEs) and intended for the optical solution of several given classification problems at different operating wavelengths, so that each classification problem is solved at the corresponding wavelength. The problem of calculating the DNN is formulated as the problem of minimizing a functional that depends on the functions of the diffractive microrelief height of the DOEs constituting the DNN and represents the error in solving the given classification problems at the operating wavelengths. We obtain explicit and compact expressions for the derivatives of this functional, and using them, we formulate a gradient method for the DNN calculation. Using this method, we design DNNs for solving the following three classification problems at three different wavelengths: the problem of classifying handwritten digits from the MNIST database, the problem of classifying fashion products from the Fashion MNIST database, and the problem of classifying ten handwritten letters from the EMNIST database. The presented simulation results of the designed DNNs demonstrate the high performance of the proposed method.

1. Introduction

In recent years, the design of photonic structures for optical computing and optical information processing has attracted significant interest. These structures are considered as a promising platform for the further development of computing systems and are intended for creating an alternative to electronic components or supplementing them [1,2,3,4]. Optical neural networks [5,6,7,8,9], and, in particular, diffractive neural networks (DNNs), comprising a cascade of sequentially placed phase diffractive optical elements (DOEs) [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26], are considered as one of the most promising and rapidly developing areas in the field of optical information processing. It should be noted that DOEs (both single and cascaded) have a long history and are widely used for solving a large class of problems of steering laser radiation [27,28,29,30,31,32,33]. At the same time, the use of cascaded DOEs for the optical solution of machine learning problems was first demonstrated only in 2018 in ref. [10]. In this work, the authors pointed out several analogies between a cascade of DOEs and “conventional” artificial neural networks and introduced the term “diffractive deep neural network”. The possibility of the optical solution of classification problems using cascaded DOEs was theoretically and experimentally demonstrated in ref. [10]. Subsequent works considered the use of DNNs (cascaded DOEs) for solving various classification problems [11,12,13,14,15,17,25,26], object and video recognition [13,15], salient object detection [11], implementing multispectral imaging [22], and performing matrix multiplication, as well as implementing other linear operators [12,18,20,21]. The main method for designing DNNs is the stochastic gradient descent method, as well as “improved” first-order methods based on it [34]. These methods have become widely used and have shown their high efficiency in beam shaping problems traditionally solved using DOEs [35,36].

In most works, DNNs are calculated to work with radiation of a single operating wavelength. At the same time, the problem of calculating DNNs designed to work with radiation of various wavelengths is of great scientific and practical interest. In the following text, we will refer to such DNNs as spectral DNNs (or cascaded spectral DOEs). Spectral DNNs can be used to process spectral data, carry out parallel computations by simultaneously solving several machine learning problems at different wavelengths, change their functionality (i.e., the problem being solved) depending on the wavelength of the incident radiation, etc. In particular, references [23,24] considered the calculation of DNNs for spectral filtering and spectral analysis of the incident radiation. In references [21,22], spectral DNNs were considered for the optical implementation of various linear transformations at different wavelengths (each transformation being carried out at its “own” wavelength), as well as for multispectral imaging. One of the main problems, which can be efficiently solved using DNNs, is the problem of optical image classification. However, to the best of our knowledge, the relevant problem of calculating spectral DNNs for solving several different classification problems at different wavelengths has not yet been studied. In particular, although the solution of classification problems using the radiation of several different wavelengths was considered in recent works [25,26], several wavelengths were used only to improve the quality of the solution of a single fixed classification problem. Thus, the solution of several different classification problems at different wavelengths has not been considered in [25,26] (as well as in the other existing works).

In this work, we consider the design of spectral DNNs (cascaded spectral DOEs) for solving several different classification problems at several different wavelengths. We formulate the problem of calculating a spectral DNN as the problem of minimizing a functional representing the error of solving the given classification problems at the operating wavelengths. This functional depends on the functions defining the diffractive microrelief height of the DOEs constituting the DNN. Explicit and compact expressions are obtained for the derivatives of the error functional, and on this basis, a gradient method for the DNN design is presented. Using the proposed gradient method, we calculate several examples of spectral DNNs for solving the following three problems: classification of handwritten digits from the MNIST database at a wavelength of 457 nm, classification of fashion products from the Fashion MNIST database at 532 nm, and classification of ten handwritten letters from A to J (lowercase and uppercase) from the EMNIST database at 633 nm. The presented numerical simulation results demonstrate good classification accuracies provided by the designed spectral DNNs.

2. Design of Spectral DNNs for Solving Several Classification Problems

Let us consider the problem of calculating a spectral DNN (a cascaded DOE) intended for solving several classification problems $P_q,q=1,\dots ,Q$ at different wavelengths ${\lambda }_q$, $q=1,\dots ,Q$, so that each classification problem $P_q$ is solved at the corresponding wavelength ${\lambda }_q$. We assume that the cascaded DOE consists of n phase DOEs located in the planes $z=f_1,\dots ,z=f_n$$(0<f_1<\cdots <f_n)$ and defined by the functions of diffractive microrelief height $h_1\left({\mathbf{u}}_1\right),\dots ,h_n\left({\mathbf{u}}_n\right)$, where ${\mathbf{u}}_j=(u_j,v_j)$ are Cartesian coordinates in the planes $z=f_j$ (Figure 1).

Let us first describe the required operation of the DNN at a certain single wavelength ${\lambda }_q$. We assume that in the input plane $z=0$, amplitude images of objects from $N_q$ different classes corresponding to the classification problem $P_q$ are sequentially generated. Each generated image is illuminated by a plane wave with wavelength ${\lambda }_q$. Let us denote $w_{0,q,j}\left({\mathbf{u}}_0\right)$ as the complex amplitude of the light field generated in this way in the input plane. In the following, the subscript of a certain complex amplitude of the field $w_{m,q,j}\left({\mathbf{u}}_m\right)$ contains the index m of the plane in which this amplitude is defined, the wavelength index q (which is also the index of the corresponding classification problem), and the class number j of the input image.

The light field $w_{0,q,j}\left({\mathbf{u}}_0\right)$ generated at $z=0$ then propagates through the cascaded DOE to the output plane $z=f_{n+1}$. We assume that the light propagation in the free space (between the planes in which the DOEs are located) is described by the Fresnel–Kirchhoff diffraction integral, and that the transmission of the light field through a DOE can be described in the thin optical element approximation as the multiplication of the beam complex amplitude by the complex transmission function (CTF) of this DOE. The CTF of the m-th DOE is wavelength-dependent, and for the wavelength ${\lambda }_q$, it has the following form:

$$T_{m,q}\left({\mathbf{u}}_m\right)=exp\left\{\mathrm{i}{\phi }_{m,q}\left({\mathbf{u}}_m\right)\right\}=exp\left\{\mathrm{i}{\displaystyle \frac{2\pi }{{\lambda }_q}}[n\left({\lambda }_q\right)-1]h_m\left({\mathbf{u}}_m\right)\right\},$$

(1)

where ${\phi }_{m,q}\left({\mathbf{u}}_m\right)$ is the phase function of the DOE (the phase shift introduced by the DOE) at the wavelength ${\lambda }_q$, and $n\left({\lambda }_q\right)$ is the refractive index of the DOE material. Under these assumptions, the propagation of the input beam $w_{0,q,j}\left({\mathbf{u}}_0\right)$ from the input plane $z=0$ through the cascaded DOE to the output plane $z=f_{n+1}$ is described by the following recurrent formula:

$$\begin{array}{cc}\hfill w_{1,q,j}\left({\mathbf{u}}_1\right)=C_{q,1}\int \int & w_{0,q,j}\left({\mathbf{u}}_0\right)exp\left\{\mathrm{i}{\displaystyle \frac{\pi }{{\lambda }_q{d}_1}}{({\mathbf{u}}_1-{\mathbf{u}}_0)}^2\right\}{\mathrm{d}}^2{\mathbf{u}}_0,\hfill \\ \hfill w_{m,q,j}\left({\mathbf{u}}_m\right)=C_{q,m}\int \int & w_{m-1,q,j}\left({\mathbf{u}}_{m-1}\right)T_{m-1,q}\left({\mathbf{u}}_{m-1}\right)·exp\left\{\mathrm{i}{\displaystyle \frac{\pi }{{\lambda }_q{d}_m}}{({\mathbf{u}}_m-{\mathbf{u}}_{m-1})}^2\right\}{\mathrm{d}}^2{\mathbf{u}}_{m-1},\hfill \\ & m=2,\dots ,n+1,\hfill \end{array}$$

(2)

where $w_{m,q,j}\left({\mathbf{u}}_m\right),m=1,\dots ,n$ are the complex amplitudes of the fields incident on the corresponding (m-th) DOEs having the CTFs $T_{m,q}\left({\mathbf{u}}_m\right)$, $C_{q,m}={\left(\mathrm{i}{\lambda }_q{d}_m\right)}^{-1}exp\{\mathrm{i}2\pi d_m/{\lambda }_q\}$, and $d_m=f_m-f_{m-1}$ are the distances between the adjacent planes.

We assume that in the output plane $z=f_{n+1}$, $N_q$ spatially separated target regions $G_{q,k},k=1,\dots ,N_q$ are defined, which correspond to $N_q$ different classes of the problem $P_q$ (see Figure 1). At each input image, a certain “energy” distribution $E_{q,k},k=1,\dots ,N_q$ is generated in these regions, which corresponds to the integrals of the generated intensity distribution $I_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)={\left|w_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)\right|}^2$ over the following regions:

$$E_{q,k}=\int \int I_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right){\chi }_{q,k}\left({\mathbf{u}}_{n+1}\right){\mathrm{d}}^2{\mathbf{u}}_{n+1},k=1,\dots ,N_q,$$

(3)

where ${\chi }_{q,k}\left({\mathbf{u}}_{n+1}\right)$ is the indicator function of the region $G_{q,k}$. For solving the classification problem $P_q$, it is necessary for the cascaded DOE to generate such an intensity distribution in the output plane for the “input signal” of the j-th class $w_{0,q,j}\left({\mathbf{u}}_0\right)$, so that the maximum of the generated energies $E_{q,k},k=1,\dots ,N_q$ is reached in the corresponding target region $G_{q,j}$ [10,12].

Above, we described the required operation of the DNN at a single wavelength ${\lambda }_q$. The problem of designing a DNN for solving several different classification problems $P_q,q=1,\dots ,Q$ at different wavelengths ${\lambda }_q$ can also be formulated as the problem of calculating the functions of the diffractive microrelief height $h_1\left({\mathbf{u}}_1\right),\dots ,h_n\left({\mathbf{u}}_n\right)$ of the cascaded DOE. In this case, these functions have to be found in such a way, so that at each wavelength ${\lambda }_q$, with an input signal being an image of a certain object of the problem $P_q$ solved at this wavelength, the DNN provides the maximum energy in the target region corresponding to the class of the input image.

3. Gradient Method for Designing Spectral DNNs

For solving the described problem of calculating a spectral DNN, we will use a stochastic gradient descent method as it is commonly applied for training artificial neural networks. Let us first present a general description of the method. We assume that for the calculation (training) of the DNN (cascaded DOE), a training set $S=S_1\cup \cdots \cup S_Q$ is used, which consists of training subsets $S_q$ for the considered classification problems $P_q,q=1,\dots ,Q$. Each training set $S_q$ contains a number of input distributions (complex amplitudes of the fields) generated from the images of the objects of the problem $P_q$ at the wavelength ${\lambda }_q$. At each step of the method, a set of distributions (referred to as a batch) is randomly chosen from the training set S. For this batch, we calculate the derivatives of a certain error functional $\epsilon (h_1,\dots ,h_n)$, which depends on the functions of the diffractive microrelief height and evaluates the DNN performance. Then, a step in the direction of the anti-gradient is performed, which gives the updated microrelief heights. Since the mathematical expectations of the derivatives calculated over a batch are proportional to the derivatives of the functional calculated for the whole training set, such an approach corresponds to the stochastic gradient descent method. Let us note that in contrast to the majority of the existing works on the spectral DNN design [21,22,23], below, we will present a detailed derivation of explicit expressions for the derivatives of the error functional.

Without losing generality, we will assume that the batch corresponds to the following set of input distributions: $w_{0,q,j}\left({\mathbf{u}}_0\right),q=1,\dots ,Q,j=1,\dots ,N_q$. Thus, we assume that the batch contains $N_1+N_2+\dots +N_Q$ input distributions, and for each $q\in \{1,\dots ,Q\}$, it includes $N_q$ images of the objects of different classes from the training set $S_q$ generated at the corresponding wavelength ${\lambda }_q$. In order to describe the calculations carried out for the batch, let us write the error functional in an explicit form. Let the classification error of an incident beam $w_{0,q,j}\left({\mathbf{u}}_0\right)$ representing an object from the j-th class from the problem $P_q$ be described by a certain error functional ${\epsilon }_{q,j}(h_1,\dots ,h_n)$. Since the classification is carried out by analyzing the energies $E_{q,k}$ in the regions $G_{q,k}$ [see Equation (3)], the functional ${\epsilon }_{q,j}(h_1,\dots ,h_n)$ in the general case has the following form:

$${\epsilon }_{q,j}(h_1,\dots ,h_n)=D_{q,j}(E_{q,1},\dots ,E_{q,N_q}),$$

(4)

where $D_{q,j}$ is a certain function describing the deviation of the generated energy distribution (3) from the required distribution, in which the energy is concentrated in the required j-th target region. Then, the error functional for a batch containing the distributions $w_{0,q,j}\left({\mathbf{u}}_0\right)$, $q=1,\dots ,Q,j=1,\dots ,N_q$ can be represented as a sum of the presented functionals:

$$\epsilon (h_1,\dots ,h_n)=\sum _{q=1}^Q\sum _{j=1}^{N_q}{\epsilon }_{q,j}(h_1,\dots ,h_n).$$

(5)

For the functional (5), it is easy to find the Fréchet derivatives $\delta \epsilon /\delta h_m$. Indeed, since the functional (5) is equal to the sum of functionals, its derivatives have the following form:

$${\displaystyle \frac{\delta \epsilon (h_1,\dots ,h_n)}{\delta h_m}}=\sum _{q=1}^Q\sum _{j=1}^{N_q}{\displaystyle \frac{\delta {\epsilon }_{q,j}(h_1,\dots ,h_n)}{\delta h_m}},m=1,\dots ,n.$$

(6)

Let us consider the calculation of the derivative $\delta {\epsilon }_{q,j}/\delta h_m$ in Equation (6) with respect to the function $h_m$. To do this, let us first denote the increment of this functional caused by an increment $\Delta h_m$ of the microrelief height function $h_m$ as follows:

$${\Delta }_m{\epsilon }_{q,j}(h_1,\dots ,h_n)={\epsilon }_{q,j}(h_1,\dots ,h_m+\Delta h_m,\dots ,h_n)-{\epsilon }_{q,j}(h_1,\dots ,h_m,\dots ,h_n)$$

(7)

According to Equations (3) and (4), this increment reads as follows:

$$\begin{array}{cc}\hfill & {\Delta }_m{\epsilon }_{q,j}(h_1,\dots ,h_n)=\sum _{k=1}^{N_q}{\displaystyle \frac{\partial D_{q,j}}{\partial E_{q,k}}}\left({\Delta }_m{E}_{q,k}\right)\hfill \\ & =\sum _{k=1}^{N_q}{\displaystyle \frac{\partial D_{q,j}}{\partial E_{q,k}}}\int \int \left[{\Delta }_m{I}_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)\right]·{\chi }_{q,k}\left({\mathbf{u}}_{n+1}\right){\mathrm{d}}^2{\mathbf{u}}_{n+1}\hfill \\ & =\sum _{k=1}^{N_q}{\displaystyle \frac{\partial D_{q,j}}{\partial E_{q,k}}}\int \int {\Delta }_m\left[w_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)w_{n+1,q,j}^{*}\left({\mathbf{u}}_{n+1}\right)\right]·{\chi }_{q,k}\left({\mathbf{u}}_{n+1}\right){\mathrm{d}}^2{\mathbf{u}}_{n+1}\hfill \\ & =2Re\int \int \left[{\Delta }_m{w}_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)\right]·F_{n+1,q,j}^{*}\left({\mathbf{u}}_{n+1}\right){\mathrm{d}}^2{\mathbf{u}}_{n+1},\hfill \end{array}$$

(8)

where ${\Delta }_m{E}_{q,k}$, ${\Delta }_m{I}_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)$, and ${\Delta }_m{w}_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)$ are the increments of the energy, intensity distribution, and complex amplitude, respectively, caused by an increment of the height $\Delta h_m$, and

$$F_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)=w_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)·\sum _{k=1}^{N_q}{\chi }_{q,k}\left({\mathbf{u}}_{n+1}\right){\displaystyle \frac{\partial D_{q,j}}{\partial E_{q,k}}}.$$

(9)

By denoting the scalar product of complex functions with angled brackets, we arrive at the following:

$${\Delta }_m{\epsilon }_{q,j}(h_1,\dots ,h_n)=2Re〈{\Delta }_m{w}_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right),F_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)〉.$$

(10)

One can easily show that the operator describing the forward propagation of the light field through a set of phase DOEs [see Equation (2)], as well as the operator of the backpropagation of the field, are unitary and conserve the scalar product [16]. Using this conservation property, we can represent the increment of the error functional (10) as follows:

$${\Delta }_m{\epsilon }_{q,j}({\phi }_1,\dots ,{\phi }_n)=2Re〈{Pr}_{f_{n+1}\to f_m^{+}}\left({\Delta }_m{w}_{n+1,q,j}\right),{Pr}_{f_{n+1}\to f_m^{+}}\left(F_{n+1,q,j}\right)〉,$$

(11)

where ${Pr}_{f_{n+1}\to f_m^{+}}$ is the backpropagation operator of the field from the output plane $z=f_{n+1}$ to the plane $z=f_m^{+}$ located immediately after the plane of the m-th DOE $z=f_m$. Note that the backpropagation of the field in the free space is described by the same Fresnel–Kirchhoff integral, where the propagation distance is taken with a minus sign, in contrast to the forward propagation. The “backward propagation” of the beam through a phase DOE is described by the multiplication of the complex amplitude of the beam by the complex conjugate of the CTF of the DOE. Thus, at $m=n$, the field $F_{m,q,j}\left({\mathbf{u}}_{n+1}\right)={Pr}_{f_{n+1}\to f_m^{+}}\left(F_{n+1,q,j}\right)$ has the following form:

$$F_{n,q,j}\left({\mathbf{u}}_n\right)=C_{q,n+1}^{*}\int \int F_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)exp\left\{\mathrm{i}\pi {\displaystyle \frac{{({\mathbf{u}}_n-{\mathbf{u}}_{n+1})}^2}{{\lambda }_q·(-d_{n+1})}}\right\}{\mathrm{d}}^2{\mathbf{u}}_{n+1}.$$

(12)

Then, at $m<n$, the field $F_{m,q,j}\left({\mathbf{u}}_{n+1}\right)$ is calculated recursively using the following formula:

$$\begin{array}{cc}\hfill F_{l-1,q,j}\left({\mathbf{u}}_{l-1}\right)=C_{q,l}^{*}\int \int F_{l,q,j}\left({\mathbf{u}}_l\right)T_{q,j}^{*}\left({\mathbf{u}}_l\right)exp& \left\{\mathrm{i}\pi {\displaystyle \frac{{({\mathbf{u}}_{l-1}-{\mathbf{u}}_l)}^2}{{\lambda }_q·(-d_l)}}\right\}{\mathrm{d}}^2{\mathbf{u}}_l,\hfill \\ & l=n,\dots ,m+1.\hfill \end{array}$$

(13)

Let us note that since ${Pr}_{f_{n+1}\to f_m^{+}}\left({\Delta }_m{w}_{n+1,q,j}\right)={\Delta }_m\left(w_{m,q,j}{T}_{m,q}\right)$, where $w_{m,q,j}\left({\mathbf{u}}_m\right)T_{m,q}\left({\mathbf{u}}_m\right)$ is the complex amplitude of the field immediately after the plane of the m-th DOE upon the forward propagation, the increment (11) can be transformed as follows:

$$\begin{array}{cc}\hfill {\Delta }_m{\epsilon }_{q,j}(h_1,\dots ,h_n)& =2Re〈{\Delta }_m\left(w_{m,q,j}{T}_{m,q}\right),F_{m,q,j}〉\hfill \\ & =2Re\int \int w_{m,q,j}\left({\mathbf{u}}_m\right)\Delta T_{m,q}\left({\mathbf{u}}_m\right)F_{m,q,j}^{*}\left({\mathbf{u}}_m\right){\mathrm{d}}^2{\mathbf{u}}_m.\hfill \end{array}$$

(14)

Since

$$\begin{array}{cc}\hfill \Delta T_{m,q}& =exp\left\{\mathrm{i}{\gamma }_q(h_m+\Delta h_m)\right\}-exp\left\{\mathrm{i}{\gamma }_q{h}_m\right\}\hfill \\ & =T_{m,q}\mathrm{i}{\gamma }_q\Delta h_m+\mathrm{o}(\Delta h_m),\hfill \end{array}$$

(15)

where ${\gamma }_q=2\pi [n\left({\lambda }_q\right)-1]/{\lambda }_q$, then the principal linear part of the increment (14) can be written as the following scalar product:

$$\begin{array}{cc}\hfill {\delta }_m{\epsilon }_{q,j}(h_1,\dots ,h_n)& =-2{\gamma }_q\int \int \Delta h_m\left({\mathbf{u}}_m\right)Im\left[w_{m,q,j}\left({\mathbf{u}}_m\right)T_{m,q}\left({\mathbf{u}}_m\right)F_{m,q,j}^{*}\left({\mathbf{u}}_m\right)\right]{\mathrm{d}}^2{\mathbf{u}}_m\hfill \\ & =-2{\gamma }_q〈\Delta h_m,Im\left[w_{m,q,j}{T}_{m,q}{F}_{m,q,j}^{*}\right]〉.\hfill \end{array}$$

(16)

According to Equation (16), the Fréchet derivative of the functional (4) has the following form:

$${\displaystyle \frac{\delta {\epsilon }_{q,j}(h_1,\dots ,h_n)}{\delta h_m}}=-2{\gamma }_{q}Im\left[w_{m,q,j}\left({\mathbf{u}}_m\right)T_{m,q}\left({\mathbf{u}}_m\right)F_{m,q,j}^{*}\left({\mathbf{u}}_m\right)\right].$$

(17)

Thus, the calculation of the gradient of the functional for a batch can be performed using Equations (6) and (17). It is worth noting that in the existing works on the design of spectral DNNs (see, e.g., refs. [21,22,23]), explicit expressions for the gradient of the error functionals are not presented, and their calculation is performed numerically using the standard PyTorch and TensorFlow frameworks. In this regard, we consider the obtained Equations (6) and (17) for the derivatives of the error functional as a new and important theoretical results.

Above, the functional ${\epsilon }_{q,j}(h_1,\dots ,h_n)$ describing the classification error of an object of the j-th class in the problem $P_q$ was written in a general form (4), where $D_{q,j}(E_{q,1},\dots ,E_{q,N_q})$ is a certain error function depending on the energy distribution (3) generated at the functions $h_1,\dots ,h_n$. Let us consider a particular example of the functional. For correct classification of an input image of the j-th class, it is necessary for the energy $E_{q,j}$ in the corresponding region $G_{q,j}$ to have a “large” value $E_{\max }$ and for the energies in the other regions to be close to zero. Accordingly, as an error functional for recognizing an input distribution of the j-th class, one can, for example, use the following quadratic functional [19]:

$${\epsilon }_{q,j}(h_1,\dots ,h_n)=\sum _{k=1}^{N_q}{\left(E_{q,k}-E_{\max }{\delta }_{k,j}\right)}^2,$$

(18)

where ${\delta }_{k,j}$ is the Kronecker delta. The derivatives of the functional (18) are calculated using the general Formula (17), where, according to Equation (9), the function $F_{m,q,j}\left({\mathbf{u}}_m\right)$ is calculated through the backpropagation of the field:

$$F_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)=2w_{n+1,q,j}\left({\mathbf{u}}_{n+1}\right)\sum _{k=1}^{N_q}{\chi }_{q,k}\left({\mathbf{u}}_{n+1}\right)·(E_{q,k}-E_{\max }{\delta }_{k,j}).$$

(19)

Let us note that in the design of a cascaded DOE, the functions of the diffractive microrelief height $h_1\left({\mathbf{u}}_1\right),\dots ,h_n\left({\mathbf{u}}_n\right)$ are usually assumed to be bounded and take values from a certain interval $[0,h_{\max }]$, where $h_{\max }$ is the maximum microrelief height (the $h_{\max }$ value is defined by the technology used for the DOE fabrication). The presence of constraints $0⩽h_m\left({\mathbf{u}}_1\right)⩽h_{\max },i=1,\dots ,n$ makes the problem of designing a cascaded DOE a conditional optimization problem. To take these constraints into account, it is necessary to introduce the following projection operator on the set of bounded height functions into the iterative calculation process:

$$\mathrm{P}\left(h\right)=\left\{\begin{array}{cc}0,\hfill & h<0,\hfill \\ h,\hfill & h\in [0,h_{\max }),\hfill \\ h_{\max },\hfill & h⩾h_{\max }.\hfill \end{array}\right.$$

(20)

In particular, the introduction of this operator to the gradient method for designing cascaded DOEs leads to the gradient projection method, in which the height functions are updated as follows:

$$h_m^k\left({\mathbf{u}}_m\right)=\mathrm{P}\left[h_m^{k-1}\left({\mathbf{u}}_m\right)-t{\displaystyle \frac{\delta \epsilon }{\delta h_m}}\left({\mathbf{u}}_m\right)\right],m=1,\dots ,n,$$

(21)

where the superscript k denotes the iteration number and t is the step of the gradient method. Note that instead of the simplest version of the gradient method of Equation (21), one can utilize its various extensions, e.g., the widely used Adam method [34].

4. Design Examples of Spectral DNNs

Let us consider the calculation of a spectral DNN for solving three different classification problems $P_q,q=1,2,3$ at the following three operating wavelengths: ${\lambda }_1=457\mathrm{nm}$, ${\lambda }_2=532\mathrm{nm}$, and ${\lambda }_3=633\mathrm{nm}$, which correspond to the solid-state lasers commonly used in optical design. Let us choose the following problems to solve: the problem of classifying handwritten digits from the MNIST dataset at the wavelength ${\lambda }_1=457\mathrm{nm}$ (problem $P_1$), the problem of classifying fashion products from the Fashion MNIST dataset at ${\lambda }_2=532\mathrm{nm}$ (problem $P_2$), and, finally, the problem of classifying handwritten letters from A to J (lowercase and uppercase) from the EMNIST dataset at ${\lambda }_3=633\mathrm{nm}$ (problem $P_3$). Note that each of the chosen classification problems contains the objects of ten classes, i.e., $N_1=N_2=N_3=10$. Let us also note that it is these classification problems that are solved (by separately designed networks operating at a single wavelength) in the vast majority of the existing works on DNN design.

For the DNN design, let us use the following parameters. We assume the input images for the classification problems $P_q,q=1,2,3$ in the input plane to be defined on a $56\times 56$ square grid with a step size of $d=10\mathsf{\mu }\mathrm{m}$. The “interlayer” distances between the input plane to the first DOE, between the DOEs, and from the last DOE to the output plane are the same and equal $\Delta f=160\mathrm{mm}$. The microrelief height functions in the DOE planes are defined on $512\times 512$ square grids with a step size (pixel size) of $10\mathsf{\mu }\mathrm{m}$. In this case, the side length of the DOE aperture amounts to 5.12 mm. We set the maximum height of the diffractive microrelief to be $h_{\max }=6\mathsf{\mu }\mathrm{m}$. Note that DOEs with such height and pixel size have a moderate aspect ratio of $h_{\max }/d=0.6$ and thus can be fabricated using the standard direct laser writing technique [37,38]. Let us also note that the chosen parameters are in agreement with the results of ref. [17]. In that work, it was shown that a DNN operating at a wavelength of $\lambda$ behaves like a fully connected neural network and can achieve a good performance if its Fresnel number, defined as $d^2/(\lambda \Delta f)$, lies in the range of $[{10}^{-4},{10}^{-2}]$. For our design parameters, the Fresnel numbers are of about ${10}^{-3}$ for all three operating wavelengths and thus belong to this “optimal” range. In addition, we verified that at the chosen parameters, the diffraction pattern from the input image formed on the first DOE approximately covered the DOE aperture, i.e., it was neither “concentrated” in its central part (which would make the peripheral parts of the DOE not operational) nor noticeably exceeded the DOE boundaries (which would result in a loss of information). For the sake of simplicity, as the refractive indices of the DOE material, we will use the same value $n\left({\lambda }_1\right)=n\left({\lambda }_2\right)=n\left({\lambda }_3\right)=1.46$, which, nevertheless, is quite close to the refractive index of fused silica at the operating wavelengths.

4.1. Sequential Solution of the Classification Problems

Let us first assume that the images of the objects from different classification problems $P_q,q=1,2,3$ are generated in the input plane $z=0$ sequentially, and that each image from the problem $P_q$ is illuminated by a normally incident plane wave (propagating along the z axis) with wavelength ${\lambda }_q$. Since each of the considered problems $P_q$ contains the objects of 10 classes, which, as we assumed, are generated in the input plane $z=0$ in a sequential way, it is sufficient to use a single set of 10 target regions $G_k,k=1,\dots ,10$ in the output plane for all three problems. In this case, the DNN will change the classification problem being solved by changing the wavelength ${\lambda }_q$ of the incident radiation. The target regions, in which energy maxima have to be generated for different classes, are shown in Figure 2a and have a square shape with sides of $0.25\mathrm{mm}$. The classes numbered as $0,\dots ,9$ in Figure 2a correspond to the digits $0,\dots ,9$ in problem $P_1$, different fashion products (T-shirt/top, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, ankle boot) in problem $P_2$, and the letters from A to J in problem $P_3$.

First, using the developed gradient method [Equations (5), (6), (17)–(21)], a DNN consisting of a single DOE was calculated. Let us note that the calculations of the derivatives of the error functionals were performed numerically using the angular spectrum method [39,40]. For the DOE calculation, we used a training set S containing 60,000 images of handwritten digits from the MNIST dataset, 60,000 images of fashion products from the Fashion MNIST dataset, and 48,000 images of handwritten letters from the EMNIST dataset. As the initial function of the microrelief height, a realization of white noise with a uniform distribution of values in the $[0,h_{\max }]$ range was used. Note that in addition to the random initial functions of the microrelief height, we also used constant initial functions, which led to close but slightly inferior results in terms of the DNN performance. The training, which was carried out until reasonable convergence of the value of the error functional, took approximately 4 h using an NVIDIA RTX 3060 12 Gb graphics card utilized for the computations. The obtained microrelief height function of the designed DOE is shown in Figure 3a.

After the training, the performance of the calculated DOE was evaluated using a test set containing 10,000 images for each of the problems $P_1$ and $P_2$ and 8000 images for problem $P_3$ (the images from the test set were not included in the training set). The obtained values of the classification accuracies of the objects from different classes (such values are often referred to as recall) for the three considered classification problems are shown with circles in Figure 4a, which are connected with solid lines as a guide to the eye. The overall classification accuracy (i.e., the ratio of the quantity of correctly recognized objects to the size of the test set) amounts to 96.41% for problem $P_1$, 84.11% for problem $P_2$, and 90.87% for problem $P_3$. The full confusion matrices for the three considered classification problems describing the obtained results in more detail are given in the supplementary materials (see Figure S1). Let us note that a relatively low classification accuracy for the objects of the 6-th class of problem $P_2$ (shirt) in Figure 4a is caused by the fact that these objects are visually close to the objects of the classes 0, 2, and 4 (T-shirt/top, pullover, coat) (see Figure S1). This effect is also present for the latter classes, albeit in this case, it is not as pronounced (see Figure 4a and Figure S1). Note that this feature is in agreement with the results of other works in which the FMNIST classification problem was considered [10,17].

In addition to the classification accuracy, another important parameter is the energy distribution in the target regions generated by the DOE. Let us define ${\overline{E}}_{q,j\to k}$ as the average energy calculated for the test set, which is directed to the k-th target region for the input objects of the j-th class from the problem $P_q$. These average energy values are shown in Figure S1 in the supplementary material in the form of the so-called energy distribution matrices. From a practical point of view, an important characteristic is the contrast value, which shows how much the energy in the region of the class under consideration exceeds the energy in the regions corresponding to the other classes. Let us introduce the contrast for the objects of the j-th class in problem $P_q$ as follows:

$${\mathrm{CR}}_{q,j}={\displaystyle \frac{{\overline{E}}_{q,j\to j}-\underset{k\ne j}{max}{\overline{E}}_{q,j\to k}}{{\overline{E}}_{q,j\to j}+\underset{k\ne j}{max}{\overline{E}}_{q,j\to k}}}.$$

(22)

In the opinion of the authors, for robust identification of “true maxima” of the energy in the experimental implementation of the DNN, it is necessary for the theoretical values of ${\mathrm{CR}}_{q,j}$ to exceed at least 0.1. The obtained contrast values for the three considered problems are shown in Figure 4b. The minimum contrast values ${\mathrm{CR}}_{\min ,q}={min}_j{\mathrm{CR}}_{q,j}$ for problems $P_q,q=1,2,3$ amount to 0.17, 0.10, and 0.13, respectively, and are not less than the chosen “critical” value of 0.1.

It is worth benchmarking the performance of the designed spectral single-DOE DNN solving three classification problems at three different wavelengths against separate DOEs, each of which solves a single classification problem $P_q$ at the corresponding operating wavelength ${\lambda }_q$. These DOEs were calculated using the gradient method using the parameters given above. For the calculated DOEs (not shown here for the sake of brevity), the values of the overall accuracy and minimum contrast obtained using the corresponding test set amount to 96.88% and 0.19 (problem $P_1$), 86.64% and 0.11 (problem $P_2$), and 93.3% and 0.13 (problem $P_3$). As one would expect, the spectral DOE [Figure 3a], which enables solving all three classification problems, provides lower classification accuracies compared to “reference” DOEs designed separately for each of the problems. At the same time, the decrease in accuracy is relatively small, and for the considered problems $P_q,q=1,2,3$, amounts to 0.47%, 2.53%, and 2.36%, respectively. The decrease in the minimum contrast for the three classification problems is also rather small.

It is also interesting to compare the performance of the calculated spectral DOE of Figure 3a with the performance of a DOE solving the same three classification problems, but at a single operating wavelength. This DOE was calculated using the gradient method for the wavelength ${\lambda }_1=457\mathrm{nm}$ using the parameters given above. For this DOE (not presented for brevity), the overall accuracy and minimum contrast amount to 92.69% and 0.12 (problem $P_1$), 81.96% and 0.07 (problem $P_2$), and 84.9% and 0.10 (problem $P_3$). One can see that the single DOE solving three classification problems at the same wavelength exhibits inferior performance compared to the spectral DOE. The decrease in the overall classification accuracy occurring when a single operating wavelength is used instead of three different wavelengths amounts to 3.72% (problem $P_1$), 2.15% (problem $P_2$), and 5.97% (problem $P_3$). The better performance of the spectral DOE operating at three different wavelengths can be explained by the fact that the phase shifts introduced by the DOE at different wavelengths are different [see Equation (1)]. In comparison with a DOE designed for a single working wavelength, this provides additional degrees of freedom during the optimization.

Having discussed the properties of a single spectral DOE, let us now move to a DNN comprising two DOEs. The microrelief height functions of the calculated DOEs are shown in Figure 3b. The obtained values of the classification accuracy in the three considered problems for this DNN are shown in Figure 4a with circles connected by dashed lines. The corresponding contrast plots are shown in Figure 4c. The resulting values of the overall classification accuracy and minimum contrast for the designed cascade of two DOEs equal 97.86% and 0.16 (problem $P_1$), 86.93% and 0.11 (problem $P_2$), and 93.07% and 0.12 (problem $P_3$). Full confusion matrices and energy distribution matrices for this structure are given in the supplementary materials (Figure S2). It is evident that the cascade of two DOEs, compared to the single DOE, provides a better performance. In particular, the increase in the overall classification accuracy amounts to 1.45% (problem $P_1$), 2.82% (problem $P_2$), and 2.2% (problem $P_3$) at virtually the same contrast. In order to illustrate the operation of a DNN consisting of two DOEs, in Figure 5, particular examples of input images from the classification problems $P_q,q=1,2,3$ are shown, as well as the corresponding energy distributions generated by the DNN in the output plane.

In addition, we also designed a spectral DNN containing three DOEs (microrelief height functions are not shown in the paper for the sake of brevity). For ease of comparison of the designed DNNs,

Table 1. Overall accuracy and minimum contrast provided by spectral DNNs consisting of one, two, and three DOEs solving three classification problems in sequential and parallel regimes.

Number of DOEs	Classification Problem	Wavelength $\mathit{\lambda }$ (nm)	Sequential Regime		Parallel Regime
			Overall Accuracy (%)	Minimum Contrast	Overall Accuracy (%)	Minimum Contrast
One	$P_1$: MNIST	457	96.41	0.17	96.25	0.18
	$P_2$: FMNIST	532	84.11	0.10	83.71	0.11
	$P_3$: EMNIST	633	90.87	0.13	90.56	0.14
Two	$P_1$: MNIST	457	97.86	0.16	97.38	0.19
	$P_2$: FMNIST	532	86.93	0.11	87.96	0.11
	$P_3$: EMNIST	633	93.07	0.12	92.93	0.16
Three	$P_1$: MNIST	457	97.89	0.20	97.41	0.21
	$P_2$: FMNIST	532	89.75	0.11	89.10	0.13
	$P_3$: EMNIST	633	93.22	0.19	92.95	0.17

presents the values of the overall classification accuracy and minimum contrast for the single DOE and the cascades of two and three DOEs. One can see that the values of the overall accuracy and minimum contrast for the cascade of three DOEs are 97.89% and 0.20 (problem $P_1$), 89.75% and 0.11 (problem $P_2$), and 93.22% and 0.19 (problem $P_3$). In comparison with the cascade of two DOEs, the cascade of three DOEs provides better values of the minimum contrast for problems $P_1$ and $P_3$ and a noticeably higher overall classification accuracy for problem $P_2$ (the classification accuracy increases by almost 3%). At the same time, the classification accuracy values for problems $P_1$ and $P_3$ remain almost unchanged. Let us also note that the addition of a fourth DOE to the cascade leads to only a marginal increase in the classification accuracy and the contrast.

To conclude this subsection, let us note that the achieved classification accuracy values are quite high, and even for the case of a single-DOE DNN, they exceed the values obtained in the other works applying DNNs operating at a single wavelength. For example, in refs. [10,12,14], the theoretical classification accuracies for the MNIST classification problem achieved by DNNs consisting of at least five DOEs and working at a single wavelength amounted to $91.75\%$, $92.28\%$, and $91.57\%$, respectively. These accuracy values are significantly lower than the value of $96.41\%$ achieved by the designed single-DOE spectral DNN for the classification problem $P_1$ (MNIST).

4.2. Parallel Solution of the Classification Problems

In the previous subsection, we assumed that the input fields corresponding to objects from different classification problems $P_q,q=1,2,3$ are generated in the input plane $z=0$ one after another, so that the DNN solves the corresponding classification problems in a sequential way. In this case, it was sufficient to use one set of 10 target regions $G_k,k=1,\dots ,10$ for all three classification problems [Figure 2a]. Let us now consider the case of parallel solution of the same classification problems $P_q,q=1,2,3$. We will assume that at each moment, three input fields with wavelengths ${\lambda }_q$ are simultaneously generated in the input plane. These fields correspond to certain objects from the considered problems of classifying handwritten digits (problem $P_1$), fashion products (problem $P_2$), and 10 handwritten letters (problem $P_3$). Since the problems $P_q$ have to be solved simultaneously, it is necessary to define three spatially separated sets of target regions $G_{q,k},k=1,\dots ,10$ corresponding to the problems being solved. The geometry of the target regions used in the present example is shown in Figure 2b.

Let the images of the objects from the classification problems $P_q$ in the input plane be defined on $56\times 56$ grids with a pixel size of $d=10\mathsf{\mu }\mathrm{m}$, the centers of which for different problems $P_q$ are shifted along the $u_0$ axis by different distances and are located at the points $s_1=(-2.56,0)\mathrm{mm}$ (problem $P_1$), $s_2=(0,0)$ (problem $P_2$), and $s_3=(2.56,0)\mathrm{mm}$ (problem $P_3$). These input images are schematically shown in Figure 1. We will assume that, in contrast to the previous case, the generated images are illuminated by obliquely incident plane waves with wavelengths ${\lambda }_q$ and the propagation directions “aimed” from the points $s_q$ at the center of the first DOE. As before, in the considered DNN examples, the distances between the adjacent planes involved in the DNN design problem are the same and equal 160 mm.

For the considered geometry of parallel solution of the classification problems using the developed gradient method given in Equations (5), (6), (17)–(21), spectral DNN were calculated, consisting of a single DOE and cascades of two and three DOEs. As an example, Figure 6 shows the microrelief height functions of the designed single DOE and cascade of two DOEs. In Figure 7, the corresponding plots of the classification accuracy and contrast are shown. The full confusion and energy distribution matrices are shown in the supplementary materials in Figures S3 and S4. For ease of comparison of the performance of the designed DNNs operating in the sequential and parallel regimes, the values of the overall classification accuracy and minimum contrast for the parallel case are shown in the right part of Table 1. By comparing Figure 4 and Figure 7 and the left and right parts of Table 1, one can see that the classification accuracy values in the sequential and parallel regimes are approximately the same. The rate of accuracy increase with increases in the number of DOEs constituting the DNN is also very similar for the sequential and parallel geometries.

5. Discussion and Conclusions

We presented an approach for designing spectral DNNs (cascaded spectral DOEs) intended for solving several given classification problems at several different wavelengths, with each classification problem being solved at its “own” wavelength of the incident radiation. In this approach, the problem of calculating the spectral DNN was formulated as the problem of minimizing a functional that depends on the functions of the diffraction microrelief height of the cascaded DOE, representing the error of solving the given classification problems at the design wavelengths. Explicit and compact expressions were obtained for the derivatives of the functional and were used for formulating a gradient method for the DNN calculation.

Using the proposed method, spectral DNNs were designed for solving the following three problems: the problem of classifying handwritten digits from the MNIST database at a wavelength of 457 nm (problem $P_1$), the problem of classifying fashion products from the Fashion MNIST database at a wavelength of 532 nm (problem $P_2$), and the problem of classifying ten handwritten letters from A to J (lowercase and uppercase) from the EMNIST database at a wavelength of 633 nm (problem $P_3$). DNNs were designed for two geometries, assuming sequential and parallel solution of different classification problems. In the first (sequential) geometry, the input beams are normally incident, and a single set of target regions is used for all the classification problems being solved. However, this configuration can also be applied to the case of parallel processing. In this case, similarly to ref. [21], it should be assumed that in the optical setup used to implement the solution of the classification problems, in addition to the DNN, there are additional optical elements that perform wavelength multiplexing of the incident beams in the input plane and wavelength demultiplexing of the resulting field distributions in the output plane. At the same time, the second (parallel) geometry does not require the use of additional multiplexing and demultiplexing devices due to the spatial separation of input and output fields with different wavelengths.

The presented numerical simulation results of the designed DNNs demonstrate the high performance of the proposed approach. In particular, in the parallel regime of solving the classification problems, a cascade of three DOEs provides the overall classification accuracy values of 97.41%, 89.1%, and 92.95% for the $P_1$ (MNIST), $P_2$ (Fashion MNIST), and $P_3$ (EMNIST) problems, respectively. It is important to note that these classification accuracy values (as well as the values achieved in the sequential regime) exceed the values obtained in the other works for DNNs designed for a single operating wavelength. For example, in refs. [10,12,14], the theoretical classification accuracies for the MNIST classification problem achieved by “single-wavelength” DNNs consisting of at least five DOEs amounted to $91.75\%$, $92.28\%$, and $91.57\%$, respectively. The classification accuracy values obtained in the seminal paper [10] for the Fashion MNIST classification problem were equal to $81.13\%$ and $86.60\%$ for DNNs consisting of five and ten DOEs, respectively. We do not present a comparison for the third classification problem (EMNIST), since, to the best of our knowledge, it has not been considered in the existing works dedicated to the DNN design.

An important problem, which is interesting from both theoretical and practical points of view, is the investigation of the achievable number of operating wavelengths (spectral channels) of the DNN and of the influence of this number on the DNN performance (namely, the classification accuracy and contrast). In this regard, it is worth mentioning recent work [21] in which the design of spectral DNNs was considered for the optical implementation of different linear transformations at different wavelengths. In the simulations, the authors claimed to implement more than 180 transformations at different wavelengths; however, their proof-of-concept experiment was carried out at two wavelengths only for very simple linear transformations (permutations of $3\times 3$ matrices), which highlights the complexity of this problem. Such an investigation for spectral DNNs solving different classification problems will be the subject of further research.