Handwritten Digits Recognition Based on a Parallel Optoelectronic Time-Delay Reservoir Computing System

Abstract

In this work, the performance of an optoelectronic time-delay reservoir computing system for performing a handwritten digit recognition task is numerically investigated, and a scheme to improve the recognition speed using multiple parallel reservoirs is proposed. By comparing four image injection methods based on a single time-delay reservoir, we find that when injecting the histograms of oriented gradient (HOG) features of the digit image, the accuracy rate (AR) is relatively high and is less affected by the offset phase. To improve the recognition speed, we construct a parallel time-delay reservoir system including multi-reservoirs, where each reservoir processes part of the HOG features of one image. Based on 6 parallel reservoirs with each reservoir possessing 100 virtual nodes, the AR can reach about 97.8%, and the reservoir processing speed can reach about 1 × 10⁶ digits per second. Meanwhile, the parallel reservoir system shows strong robustness to the parameter mismatch between multi-reservoirs.

1. Introduction

Nowadays, artificial intelligence (AI) technology plays an irreplaceable role in numerous fields. One of the important applications is static image recognition, such as handwritten digits recognition (HWDR) [1], vehicle perception, medical image analysis [2], and facial recognition [3]. The widespread application of AI benefits from the improvement of neuro-inspired algorithms and the rapid progress of digital computers in computing ability. However, the current efficiency of conventional digital computers when running neuro-inspired algorithms such as recurrent neural networks (RNNs) is still not satisfactory because these algorithms consume huge computing resources in training procedures [4]. In order to improve the efficiency of AI equipment, researchers have been working to find a neuro-inspired algorithm that possesses a simple training process, low energy consumption, and high performance. Reservoir computing (RC), which emerged in recent years, can meet these conditions and can be implemented in nonlinear systems. RC has been considered one of the promising neural algorithms to further promote the development of AI [5].

RC is a general term for the echo state network [6] and the liquid state machine [7]. A typical RC system can be divided into an input layer, a reservoir (a dynamic network composed of a large number of interconnected nonlinear nodes), and an output layer. RC uses randomly generated and fixed input weights and reservoir weights to reduce the training difficulty. Since there is no need to change the internal connections of the system, RC is much easier to implement in physical hardware than traditional neuromorphic computing schemes [8,9]. In 2011, the implementation of RC was greatly simplified, where a time-delay system was used as the reservoir, and the reservoir states (named as virtual node states) were constituted by the states of a single nonlinear node sampled multiple times within a delay cycle [10]. Since then, RC based on the time-delay system (TDRC) has been implemented in electrical circuits [11], optoelectronic systems [12,13,14,15], and all-optical systems [16,17,18]. Based on the serial input and output mechanism, TDRC shows excellent performance for processing time-related tasks such as time series prediction, waveform recognition, and digital speech recognition [19,20]. For instance, in 2017, Larger et al. implement a photonic reservoir computing system based on optoelectronic telecom devices, which achieves a million words per second classification speed for speech-recognition tasks [21]. In 2021, Zhong et al. obtained a chaotic time series synchronized with three dynamics systems based on an all-optical RC formed by SLs under optical feedback [22]. In the same year, Dai et al. numerically investigated the performance of optoelectronic RC for the classification of IQ-modulated signals [23]. In 2022, Han et al. proposed an approach for simultaneous modulation format identification and optical signal-to-noise monitoring in optical communication based on an optoelectronic RC [24]. However, when processing tasks such as image recognition, the performance of TDRC needs to be improved. For instance, the accuracy rate (AR) of recognizing handwritten digits in [25] using an optoelectronic TDRC is about 83%, which is significantly lower than the AR when using a convolutional neural network (CNN) [26] or an echo state network [27,28] based on parallel data processing mechanisms. Therefore, an in-depth investigation on improving the AR and recognition speed of static images based on TDRC is of some significance for promoting the application of TDRC in the AI field.

In this work, we numerically investigate the performance of an optoelectronic TDRC system for performing HWDR tasks. Based on a single time-delay reservoir, we first compare four different image input methods including the direct input of a full image, input of a trimmed image, input of 2×-downscaled features, and input histograms of oriented gradient (HOG) features. We found that under optimized parameters, the TDRC with 600 virtual nodes can achieve an AR of 97% with input HOG features. Then, we increase the number of parallel time-delay reservoirs used in the TDRC system to increase the data processing rate. Each reservoir is used for processing a part of the HOG features, and all the reservoir states are combined for training and testing. The results show that based on 6 reservoirs (each reservoir contains 100 nodes), the TDRC system can achieve an AR of 97.8% with a reservoir processing speed of 1 × 10⁶ digits per second.

2. System Model

Figure 1a,b shows the schematic diagram of the system setup and the operation procedure of the optoelectronic RC system for processing an HWDR task. The system uses a semiconductor laser as the light source to output continuous wave P₀. A Mach–Zehnder intensity modulator (MZM) is used as the nonlinear node to inject the task information into the reservoir. Under the continuous wave injection of P₀, the output optical intensity P(t) of MZM can be described as:

$$P(t)=P_0{\cos }^2(\frac{\pi }{2}\frac{m(t)}{V_{\pi }}+\varphi )$$

(1)

where m(t) is the modulation voltage applied to the MZM and V_π is the half-wave voltage of the MZM. φ denotes the offset phase adjusted by the bias voltage of the MZM. After passing through a variable attenuator (VA), the optical signal P(t) is converted into electrical signal P_out(t) using a photo detector (PD), which can be approximated as a linear transformation of P(t), and the amplitude of P_out(t) can be adjusted using the VA. The electrical signal P_out(t) passes through an amplifier, a high-pass filter (HPF), and a low-pass filter (LPF). Then, the output signal x(t) of the HPF is added with the input data γI(t) to modulate the MZM, where γ is an input scaling factor. As shown in Figure 1b, in the optoelectronic feedback loop, the optoelectronic conversion efficiency and electronic gain can be described as a feedback coefficient β, and the delay time of the feedback loop is τ. To simplify the analysis, the summed signal (γI(t) + x(t − τ)) applied to the MZM is expressed in a normalized unit as γI(t) + x(t − τ) ≡ πm(t)/2V_π. Therefore, the signal input to the LPF is described as βcos²[x(t − τ) + φ + γI(t)]. The LPF and HPF constitute a band-pass filter, where τ_L and τ_H describe the characteristic time of the LPF and HPF, respectively. Correspondingly, the cut-off frequency and cut-on frequency of LPF and HPF are 1/(2πτ_L) and 1/(2πτ_H), respectively. The transfer function in the frequency domain of the band-pass filter can be described as:

$$H(s)=\frac{s{\tau }_H}{(1+s{\tau }_L)(1+s{\tau }_H)}$$

(2)

where s is a complex Laplace variable. The detailed derivation process of the filter from the frequency domain to the time domain can refer to [29].

If without I(t), this system constitutes a common optoelectronic feedback loop, which has been widely investigated for generating a chaotic signal [30], forecasting high-dimensional chaotic dynamics [31], and performing chaos-based communication [32]. In recent years, this system has been successfully applied in the TDRC field for processing time series [33,34]. Under external information injection, the dynamics of this optoelectronic TDRC system can be expressed as [33]:

$${\tau }_L\frac{\mathrm{d}x(t)}{\mathrm{d}t}=-(1+\frac{{\tau }_L}{{\tau }_H})x(t)-v(t)+\beta {\cos }^2[x(t-\tau )+\varphi +\gamma I(t)]$$

(3)

$${\tau }_H\frac{\mathrm{d}v(t)}{\mathrm{d}t}=x(t)$$

(4)

where x(t) is the reservoir output signal. v(t) is a supplementary variable derived from the necessary filtering feature of the optoelectronic amplified process [21].

In this work, we use the mixed national institute of standards and technology (MNIST) database to evaluate the performance of this TDRC for performing an HWDR task. The MNIST database contains 70,000 images of handwritten digits, where 60,000 images form a training set, and another 10,000 images form a testing set [35]. All images in the MNIST database have been tailored to a standard size of 28 × 28 pixels, and the pixel values have been normalized to [0,1]. For performing this task, the image needs to be preprocessed and converted into a time series to meet the serial input mechanism of TDRC. In this work, we compare four relatively simple preprocessing methods, i.e., inputting the full image, inputting the trimmed image, extracting 2×-downscaled features, and extracting HOG features. For the case of inputting a full image, as shown in Figure 2a, the pixel matrix with dimensions of 28 × 28 is directly converted into a row vector u with 784 elements. For the case of inputting a trimmed image, we remove the unused border area of the image to form a smaller image with 20 × 16 pixels [36]. As shown in Figure 2b, the vector u of the trimmed image contains 320 elements. For the case of extracting 2×-downscaled features, the trimmed image (20 × 16 pixels) in Figure 2b is first divided into 80 small zones with each zone consisting of 2 × 2 pixels [37]. Then, as shown in Figure 2c, the mean values of each zone are extracted and formed into a vector u with 80 elements. For the case of extracting HOG features, we calculate the gradient g = (g_x² + g_y²)^1/2 of the images using two spatial filters of F_x = [–1, 0, 1] and F_y = [–1, 0, 1], where g_x and g_y are the horizontal and vertical gradient, respectively. The orientation within [0°, 180°] is obtained as α = arctan (g_y/g_x). As shown in Figure 2d, the image is divided into 16 cells (each possesses 7 × 7 pixels), and the gradient values of each cell are calculated according to their orientations with an interval of 20°. At last, a feature vector u consisting of 324 elements is obtained using block-normalizing over a region of 2 × 2 cells.

After preprocessing, the vector u containing m elements is multiplied by an input weight matrix W_in with the dimension of m × N to form a sequence I(t), where N is the number of the reservoir’s virtual nodes. In this work, the values of W_in are randomly extracted from {−1, 1}. In the readout layer, the reservoir states x(t) are collected by sampling the output of LPF as an interval of θ. Then, we train the output weight W_out using a ridge regression method, i.e., W_out = Y^targetX^T(XX^T + λI), where W_out is the readout weight and X is the states matrix. Y^target is the target matrix that is built according to the labels of the training set. We set Y^target with 10 rows corresponding to the 10 digits of 0~9, for which the elements of the row corresponding to the target number are set to 1, and the other elements are all set to 0. λ is the regularization coefficient, and I is the identity matrix. After the training procedure, for new input images, the RC system can output recognition results of Y^output = W_outX. In the case of correct training, Y^output should be a matrix similar to Y^target. Then, the elements of each row in Y^output are summed, and the row number of the largest sum is decided as the recognition result, which procedure is called winner-takes-all [38]. The performance of the system is evaluated using the AR, i.e., the ratio of the correct recognition number to the size of the test set.

3. Results

To simulate the dynamics of the optoelectronic RC, we solve Equations (3) and (4) using the fourth-order Runge–Kuta method, where the integration step is set to 1ns. The parameter values are τ_L = 10 ns, τ_H = 0.5 μs, virtual node interval θ = 0.01 μs, and τ = Nθ (N is the number of virtual nodes).

We first compare the recognition performance of the optoelectronic RC under four different input methods. For the methods of inputting a full image (u^1×784), a trimmed image (u^1×320), 2×-downscaled features (u^1×80), and HOG features (u^1×324), the number of virtual nodes N is set to 1500, 700, 200, and 700, respectively, which satisfies an input-to-neuron ratio about 1:2. Generally speaking, a relatively large N is helpful to improve the dimension of the reservoir state space (but a too large N will lead to saturation of the state space), i.e., a relatively small input-to-neuron ratio may achieve relatively good performance. In the test of comparing different input methods, for the sake of fairness, we set the same input-to-neuron ratio for the four input methods. Figure 3 shows the evolution maps of the AR in parameter space of feedback strength β and input scaling factor γ under offset phase ϕ = 0. The different colors represent different values of the AR, and the blank area represents AR ≤ 85%. We can find that for the four input methods, the values of the AR are symmetrically distributed on both sides of γ = 0.5, and the ARs close to γ = 0.5 are relatively low. We speculate that when γ = 0.5 and ϕ = 0, the non-linear transformation effect of the reservoir is relatively weak for the input data I(t) (normalized to [0, π]). Meanwhile, we can find that the relatively high ARs are located in the range of β ∈ [0.1, 0.6] where the reservoir can generate appropriate non-linear dynamics for this task. For the methods of inputting a full image (Figure 3a), trimmed image (Figure 3b), and 2×-downscaled features (Figure 3c), the maximum values of the AR are about 90% when γ is close to 0.9. For the method of inputting HOG features (Figure 3d), the maximum AR reaches 97%, which can be obtained for a wide range of γ. Obviously, under the method of HOG, the recognition performance is better, and the robustness to β and γ is stronger. We speculate that it is due to the gradients of an image, which are more useful than its pixels. Because the magnitude of gradients is large around edges and corners (regions of abrupt intensity changes), especially for handwritten digits. Moreover, the HOG operates on the local grid cells of the image, so it maintains good invariance for the geometrical and optical deformations of the image.

According to the results of Figure 3, we set β = 0.2 and γ = 0.92 to further investigate the influence of the offset phase ϕ of MZM on the AR. Figure 4 shows the AR as a function of ϕ for the cases using four different image preprocessing methods. For the methods of inputting a full image (Figure 4a), trimmed image (Figure 4b), and 2×-downscaled features (Figure 4c), when ϕ changes from 0 to 2π, the values of the AR fluctuate between 87% and 91% with a period of 0.5π. We can find that the performance of RC for these three preprocessing methods is similar under the same input-to-neuron ratio. However, RC possesses a relatively high data processing rate with the input of 2×-downscaled features because only 200 virtual nodes are required for achieving an AR of 91%. As shown in Figure 4d, RC presents the best performance with the input of HOG features, where the AR fluctuates slightly between 97% and 97.6%, and there is no obvious dependence on ϕ. From the above tests of Figure 3 and Figure 4, we can find that the preprocessing method of HOG can effectively extract the features of handwritten digits, and the reservoir with 700 virtual nodes can achieve a good recognition performance and strong robustness to the fluctuations in β, γ, and ϕ.

In the following sections, we take the method of inputting HOG features as an example to investigate how to improve the recognition speed based on multiple parallel optoelectronic reservoirs. Since the virtual node states of a TDRC are obtained using time multiplexing, in order to obtain a sufficiently high-dimensional state space, the input period cannot be too small, which limits the speed of TDRC with a single reservoir [39]. In order to increase the recognition speed, we can increase the number of real nonlinear nodes in the system to form a mutually coupled or parallel RC system and simultaneously extract the virtual node states from the feedback loop of each reservoir [40]. In this work, as shown in Figure 5, we transform the RC system of Figure 1 into a parallel processing RC system with k decoupled time-delay reservoirs, where the SL provides a continuous wave for k parallel reservoirs. The feature matrix preprocessed using the HOG method is divided into k parts by a column, which are injected into k reservoirs after masking with different masks. The masked information of I₁(t) to I_k(t) are injected into k reservoirs, which are transformed into x₁(t) to x_k(t), respectively. Within an input period, each x(t) is sampled as a vector x with a sampling interval of θ, where each vector x contains N elements. Therefore, x₁ = [x₁¹, x₁², …, x₁^N]^T, x₂ = [x₂¹, x₂², …, x₂^N]^T, and x_k = [x_k¹, x_k², …, x_k^N]^T constitute the columns of state matrices X. X is used for training and testing, which is similar to the method used in Figure 1. In this way, under the premise of obtaining the same number of total virtual nodes, the data injection period of the parallel RC system can be reduced by k times.

Figure 6a shows AR as a function of the number of total virtual nodes N. Considering the performance deviations caused by the randomly generated mask of W_in, we run RC 10 times for each value of N. Circles and stars in Figure 6 represent the mean values of 10 test results, and the vertical bar indicates the standard deviation of 10 tests. The blue line is for the case of using a single reservoir. We can find that as N increases from 100 to 800 with a step of 20, the AR of the single reservoir monotonously increases from 93.8% to 97.8%. The red line in Figure 6 is for the case of using k-reservoirs (k × 100 = N), where each reservoir possesses 100 virtual nodes and the k reservoirs use the same parameters of β = 0.2, γ = 0.92, and ϕ = 0.4π. It can be found that the AR increases with the increase in k. When k ≥ 7, AR reaches about 98%. Meanwhile, when k > 1, for the same N, the AR of multi-reservoirs is higher than that of the single reservoir. This means that under the premise of using the same number of total virtual nodes, the RC containing multiple reservoirs can further improve the recognition rate. We speculate that this is due to the different input weights used for multiple reservoirs, which makes the reservoir states more abundant than that in a single reservoir. Figure 6b shows the recognition speed as a function of N. For the system using a single reservoir (blue line), the recognition speed (=1/(Nθ)) decreases from 1 × 10⁶ digits per second to 1.25 × 10⁵ digits per second as N increases from 100 to 800. Obviously, the single reservoir with more virtual nodes can enable a better recognition performance but pays the price of reducing recognition speed. Meanwhile, the speed of the RC system using multi-reservoirs (red line) is always 1 × 10⁶ digits per second.

In the test of Figure 6, we defaulted that the multiple parallel reservoirs possess the same parameters. Next, we take the parallel RC system with six reservoirs (k = 6) as an example to analyze the impact of parameter mismatch between multiple reservoirs on the AR. In Figure 7, we select (β_r, γ_r, ϕ_r) = (0.2, 0.92, 0.4π) as the reference values and show the variation in the AR as a function of the parameter mismatch ratio. In this test, we run RC 20 times for each point of the parameter mismatch ratio, where the masks W_in are the same for each time, and the values of β, γ, ϕ are randomly generated within a certain parameter mismatch range. For example, when the parameter mismatch is ±5%, the value of β is taken randomly within (β_r(1 − 5%), β_r(1 + 5%)]. When the parameter mismatch is ±15%, β is taken randomly within (β_r(1 − 15%), β_r(1 − 5%)] ∪ (β_r(1 + 5%), β_r(1 + 15%)]. The red curve represents the mean AR of 20 tests, and the blue curve represents the upper and lower limits of the ARs. We can find that when the mismatch ratio is lower than 55%, the AR shows a slight fluctuation around 97.8%. When the mismatch ratio is higher than 55%, the AR gradually decreases, and the fluctuation range gradually expands. However, even if the mismatch ratio reaches ±95%, the ARs are still larger than 97%. The results indicate that this RC system with multiple parallel reservoirs shows strong robustness to the parameter mismatch between different reservoirs. In other words, when the system is implemented in hardware, it has the advantages of easy parameter adjustment and stable performance.

4. Discussion

In this work, we numerically investigate the method of improving the performance of HDWR based on an optoelectronic TDRC. The performance improvements include improvements in the AR and recognition speed. For increasing the AR, four image input methods including the input of a full image, a trimmed image, 2×-downscaled features, and HOG features are compared. In fact, there are many other image feature-extracting methods that can be used, such as projection histograms, distance profiles, and Gabor filters. In this work, we find that extracting HOG features requires fewer parameters to be adjusted and achieves a relatively high AR, where the maximum AR of about 98% is achieved. This AR is significantly higher than the value of 83% achieved in [25], which directly inputs a raw digit image into the reservoir. Moreover, the system in this work based on a relatively simple structure achieves an AR comparable to the AR of about 99% in [27], which is based on a conventional network-based RC.

For increasing the recognition speed, we adopt the scheme of using multiple parallel reservoirs. As for a single reservoir, there are two ways to increase the recognition rate (=1/Nθ): reducing the number N of virtual nodes or reducing the interval θ. However, to ensure good performance, neither N nor θ can be set too small. A too-small N will cause a low dimensional state space that decreases AR. A too-small θ will cause the loss of input information. For an optoelectronic RC, θ is often related to the inverse of the characteristic bandwidth of an optoelectronic oscillator. In this work, the characteristic bandwidth ([1/τ_H, 1/τ_L]) is about 100 MHz, which is obviously lower than the available 10 GHz bandwidth of the unfiltered optoelectronic feedback loop. This means that the recognition rate of a single reservoir can be further increased by increasing the characteristic bandwidth. The pursuit of the upper limit rate of a single reservoir will be carried out in our subsequent studies. The motivation for setting a relatively small characteristic bandwidth in this work is that our main purpose is to investigate the feasibility of improving the recognition speed based on parallel reservoirs. Moreover, setting a relatively small characteristic bandwidth makes it easier to implement this system in hardware based on our existing devices. Otherwise, the RC system with a large characteristic bandwidth (adopting small θ) usually requires the support of expensive input and readout equipment. In other words, our scheme of using parallel reservoirs provides an effective way to achieve a high recognition speed for a TDRC system with relatively low hardware requirements.

5. Conclusions

Based on an optoelectronic time-delay reservoir computing system, we numerically perform the HWDR task, where the reservoir is composed of an MZM, a PD, two filters, and a delay line. First, we compare four image input methods including the input of a full image, a trimmed image, 2×-downscaled features, and HOG features. It is found that, under the first three input methods, the AR can reach about 90% within a relatively narrow parameter region of β-γ and presents periodic fluctuations with offset phase ϕ. Under the injection of HOG features, the system can achieve ARs of about 97% within a broad parameter region of β-γ and shows strong robustness to the variation in ϕ. Second, taking the injection of HOG features as an example, we investigate the recognition speed improvement based on multi-reservoirs. Using 6 parallel reservoirs with each reservoir possessing 100 virtual nodes, the system can achieve an AR of about 97.8 at a speed of 1 × 10⁶ digits per second. Meanwhile, we find that the system performance is insensitive to the parameter mismatch of β, γ, and ϕ between multi-reservoirs. In the future, there is a lot of work worthy of in-depth studies, such as exploring the upper limit recognition rate of a single reservoir, hardware implementation of the system, system integration, and applications in other fields.