Deep Learning-Based Image Classification through a Multimode Fiber in the Presence of Wavelength Drift

Abstract

Deep neural networks (DNNs) are employed to recover information after its propagation through a multimode fiber (MMF) in the presence of wavelength drift. The intensity distribution of the speckle patterns generated at the output of an MMF when an input wavefront propagates along its length is highly sensitive to wavelength changes. We use a tunable laser to implement a wavelength drift with a controlled bandwidth, aiming to estimate the DNN’s performance in different cases and identify the limitations. We find that when the DNNs are trained with a dataset which includes the noise induced by wavelength changes, successful classification of a speckle pattern can be performed even for a large wavelength bandwidth drift. A single training step is found to be sufficient for high classification accuracy, removing the need for time-consuming recalibration at each wavelength.

1. Introduction

Information transmission through multimode fibers (MMFs) has been a relatively dormant field of research and development for many decades. More recently, MMFs are becoming widely used in telecommunications because of their high information capacity at a lower manufacturing cost [1]. The number of available channels in MMFs is limited by the number of supported spatial modes. The spatial modes are the solutions to the wave equation for a MMF and thus propagate unchanged along the fiber, offering unique mode multiplexing opportunities [2,3,4,5]. MMFs have also attracted significant attention in endoscopy, for which a high information capacity is essential to achieve high quality images [6,7,8,9,10,11]. However, the differences in propagation constants between the spatial fiber modes can hinder the information transmission through a MMF system. When an input image is coupled in a MMF, it distributes among the supported spatial modes, resulting in a speckle pattern at the fiber output.

Several methods have been employed for reversing the scrambling caused by MMFs. Phase conjugation or measurement of the transmission matrix [7,8,11,12] allow to reverse the effects of propagation and unscramble the information. While phase conjugation and transmission matrix can control the light propagation through an MMF probe, they are based on interferometric measurements of the light field using digital holography, and as a result they are susceptible to environmental or experimental perturbations, requiring dynamic recalibration [8,13,14,15]. To overcome the sensitivity of the calibration-based measurements, deep neural networks (DNNs) have been proposed as an alternative for imaging through MMFs [16,17,18,19,20]. The idea of using artificial neural networks (ANNs) to interpret the information after propagation in a MMF was first reported by Aisawa et al. [21,22] in 1991 using a simple neural network architecture. Recently, the advancements in computational means allow the realization of more complex neural network architectures that can handle data of high-dimensionality, bringing DNNs to the forefront of many leading technologies ranging from research and business to military and entertainment [23,24,25]. DNNs have shown remarkable capabilities in recovering information through MMFs for imaging applications, while preserving robustness against perturbations in the system [16,17,18,20,26].

The noise induced at the output speckle patterns due to perturbations caused by thermal, mechanical or source wavelength drift can be catastrophic for a calibration-based technique such as the transmission matrix but they affect, at a lower level, the performance of DNNs [16]. In addition, in cases that multiple wavelengths are needed for imaging, the transmission matrix of the fiber needs to be measured multiple times for different wavelengths [27]. Therefore, it would be time-efficient to investigate a way to remove recalibration of the system when the wavelength is changed. In the work presented in this article, we propose an alternative approach for recovering the information through the MMF system in the presence of a wavelength drift in the light source using DNNs. We investigate the performance of DNNs to classify the input images in a MMF from their corresponding speckle patterns in various wavelength drift cases. Remarkable results were obtained for wavelength drift bandwidth up tp 100 nm, and they are further discussed in the following paragraphs.

2. Materials and Methods

The optical setup built to collect the datasets for training the DNNs is shown in Figure 1. The beam of a wavelength-tunable laser source (M-squared, SolsTiS 2000 PSX XF with Terascan software), emitting in the range 700–1000 nm is used to illuminate a graded-index (GRIN) MMF with 62.5 μm core diameter and numerical aperture (NA) of 0.275 (GIF625, Thorlabs GmbH, Germany). The wavelength of the laser source can be tuned with 10 fm accuracy. GRIN fibers are often used to avoid temporal distortion in a signal due to modal dispersion. A beam expansion is achieved by a set of lenses (L1: f = 200 mm, OBJ1: 10×, Newport, Irvine, CA, USA) placed at the laser output to ensure that the beam size is large enough to illuminate optimally a phase-only spatial light modulator (SLM, 1920 × 1080 pixels, Pluto-Vis, Holoeye, Berlin, Germany). The SLM is used to display the desired patterns and the SLM plane is imaged onto the proximal facet of the MMF by means of a 4f imaging system (lens L2: f = 400 mm, microscope objective OBJ2: 60x, Newport, Irvine, CA, USA). Another 4f system (the same as the proximal one, L3-OBJ3) magnifies the speckle pattern emerging from the distal facet on a CCD camera (CCD, Chameleon 3, 1024 × 1280 pixels, Mono, Point Grey). An additional lens (L4: f = 200 mm) in 2f configuration allows monitoring the images generated by the SLM on a second camera at the proximal side (CCD2). A halfwave plate is placed before the SLM to ensure that the correct polarization of illumination is used (see Figure 1). The fiber length used in the experiments is 10 cm.

The dataset used to assess the performance of the DNN is generated by projecting phase images of handwritten digits at the proximal fiber side [16,17,18,20]. These images are available online in the MNIST database, which is widely used to test the capabilities of different neural network architectures [28]. Wavelength drifting is simulated by a Matlab code written to control the laser wavelength, the speckle acquisition by the camera CCD1 (Figure 1) and the input images on the SLM. We first choose the wavelength range of the drift in the code and then a random array with 100 wavelength values within this range is generated. The wavelength values are sorted in an ascending order because this leads to a more stable operation of the tunable laser instead of tuning the wavelength to far spaced wavelength values. Moreover, the laser stabilization at the specified wavelength is much faster. The inputs are therefore projected on the proximal fiber facet in batches of 100, each one at a specific wavelength. For the next 100 inputs, the random wavelength array is recalculated so that we always get randomly different wavelength values for the images of the full dataset. We collected a total of 10,000 intensity images of speckle patterns. The speckle images captured by CCD1 are cropped to 1024 × 1024 pixels and then downsampled to 32 × 32 pixels before being imported in the DNN code.

For the classification experiments, a VGG(Visual Geometry Group)-type DNN is used with an architecture described in Figure 2 [29]. We use mean squared error (MSE) as a loss function and an Adam optimizer with a learning rate of 1 × 10⁻⁴. The DNN is trained in batches of 100 images for a maximum of 50 epochs. The collected images of the dataset are split so that 80% are used for training, 10% for validation and 10% for testing. For each one of the collected datasets the DNN training is repeated five times to obtain the standard deviation values. The VGG classifier is implemented in Tensorflow 1.5 Python library on a computer equipped with an NVIDIA GeForce GTX 1080Ti graphics processing unit.

3. Results

The number of fiber modes in MMFs determines the available degrees of freedom (channels) for transmitting the information. An input image is mapped to the fiber modes basis. Therefore, a MMF with only few modes would not be able to transmit a complex signal. Figure 3 shows how the image of the digit 5 selected from the MNIST database would be reconstructed through MMFs with the same NA that support different total number of spatial modes. It is apparent in Figure 3 that, below a certain number of modes, the features of the images would not be preserved. Therefore, it is important to evaluate the classification performance of the DNN at different wavelengths in the range of the tunable laser to assess its dependence on the number of supported modes in each case. In particular, in order to test the performance of DNNs in the presence of wavelength drift, it is essential that the classification accuracy is firstly determined for the different wavelengths within the drift bandwidth in no-drift conditions. In this way, it can be assumed that any change in the classification performance is attributed to the induced “noise” in the dataset caused by the wavelength perturbation and not to the inherent incapability of the system to support the input images because of the low number of supported modes at a certain wavelength.

Phase-encoded images of handwritten digits from the MNIST database are projected by the SLM on the GRIN fiber facet, and the corresponding intensity images of the resulted speckle patterns are collected. We collect datasets for different wavelengths covering the range of the tunable laser ranging from 700 to 1000 nm with a 50 nm increment. The classification accuracy of the system is determined by training the VGG DNN for each case [30,31]. The training and test set consist of images captured at a single wavelength, without any drift during recording, for this preliminary assessment. It has been shown in our previous work that with increasing fiber length, slight environmental perturbations can cause severe speckle drifts [16]. For the 10-cm long GRIN fiber used in the experiments, no further instabilities because of mechanical or thermal changes are observed. In Figure 4a, the average classification accuracy obtained at the different wavelengths is presented together with the normalized confusion matrices for the two ends of the wavelength range, namely 700 nm (Figure 4b) and 1000 nm (Figure 4c). No significant change in the classification accuracy is observed for the datasets across the studied range. Hence, we can assume that the total number of spatial modes supported within this wavelength scanning range is sufficient to represent the input information and it does not affect the performance of the VGG classifier.

As a next step, we explore the effect of wavelength drift on the DDN performance for different bandwidths of 6, 12, 24, 48, 72 and 96 nm for a central wavelength at 800 nm (Figure 5). We investigate two different cases of drifting noise [30,31]. The same GRIN fiber of 62.5 μm core and length of 10 cm is used for all the measurements. In the first case, the DNNs are trained with data acquired with the laser wavelength fixed at 800 nm, and then tested on speckle patterns recorded at different wavelengths far from the 800 nm one at which the training set is obtained. In the second case, the wavelengths within the drifting range are included in the training set by randomly acquiring speckle patterns at different wavelengths during the measurement, as described in the Methods section. For comparison, the speckle intensity correlation was calculated for a range of 100 nm around the central wavelength (800 nm) to study the DNN results as a function of the speckle decorrelation. As seen from Figure 5a, the speckle patterns decorrelate at 50% within a range of approximately 30 nm.

Studying the first case, we train the DNN with data recorded at 800 nm and we test its ability to recover the information from data measured at 803, 806, 812 and 850 nm (Figure 5b, red circles). The classification accuracy drops at 69% for the shortest drift of 803 nm, while the speckle correlation has only dropped to 90% at this wavelength distance. The results become completely random at 812 nm, reaching 10% classification. Hence, it is shown that generalization of the VGG network for wavelengths further than the one used in the training is only efficient for a very narrow range of wavelengths with respect to the one used for the training data measurements.

The classification results also show a decrease in performance with an increase in the wavelength drifting bandwidth, but at a much lower rate in the second case for which we include the wavelength drift during the training. For 6 nm drift, meaning ±3 nm around the central wavelength, the VGG DNN shows minimal change of classification performance. Specifically, we achieve classification accuracy of 88% in this case while the one obtained in absence of drift (Δλ = 0) 90% (Figure 4a). This value is remarkably better compared to the 69% observed in the experiments of the first case discussed before for the same wavelength drift of 3 nm. Even for approximately 100 nm drift, the VGG DNN classifies correctly 70% of the inputs (Figure 5b, blue circles). These results indicate high robustness of the DNNs in the presence of severe perturbations in the system as long as the training dataset is collected in the presence of the wavelength disturbance under study. The ability of the DNN to successfully classify the speckle patterns within a wide range of wavelengths can be beneficial for multicolor image transmission through a fiber system [19]. In Figure 6, we show, for comparison, the normalized confusion matrices for the two approaches for a drift of 3 nm and ±3 nm, respectively. It is evident that many labels are mixed up in the first case; for example, the digit 9, which is quite similar to at least two other digits, such as 7 and 5 s, is highly misclassified.

Another aspect evaluated is the effect of including the wavelength drift noise in the training set when we attempt to recover information outside the training bandwidth. We train the VGG classifier with data measured under 12 nm drifting situation and we afterwards test the classification accuracy of the trained model for data outside the 12 nm range, starting from 807 nm to 850 nm. In spite of the high efficiency reported for recovering information about the inputs within the drift range (Figure 5b, blue circles), this is not the case for speckle patterns recorded at wavelengths outside this range (Figure 5c). The classification results obtained for the speckles collected at laser wavelengths outside the 12 nm range follow the trend observed in the first case of experiments discussed before, as shown in Figure 5c. Based on these results, it is found that training DNNs to infer the inputs from speckle patterns in the presence of wavelength drift cannot assist their performance outside this range. In other words, the trained model does not generalize to new wavelength values which lie outside the bandwidth used for training.

Once crucial parameter usually suggested for improving the performance of the DNNs is the number of available samples in the dataset. At the same time, this parameter is considered to be one of the drawbacks of deep learning, because there are cases for which the number of available data is limited. In order to assess whether the dataset used to train the VGG classifier in the case of wavelength drifting is sufficiently large, we investigate the classification accuracy for training datasets of different sizes. The dataset of 10,000 speckle images is shuffled in a random way and each time only a part of the total dataset is used for training and testing the DNN. The classification results are presented in Figure 7. The classification accuracy is observed to vary with the size of the training dataset. It is apparent from the values reported in Figure 7 that the more severe the wavelength drift, the more samples are needed for achieving higher classification accuracies. Nevertheless, for all the tested wavelength bandwidths, a saturation of the classification improvement is observed while increasing the size of the dataset; for the largest bandwidth (96 nm), this happens at about 6000 samples, as suggested by Figure 7. Interestingly, the accuracies measured for each bandwidth saturate at values of total available samples that are below 10,000, which implies that an increase in the dataset size would not significantly improve the results.

4. Discussion

In the analysis presented in the previous paragraphs, we show the effects of wavelength drift on the performance of a VGG classifier of speckle patterns at the output of a 10-cm long GRIN MMF. Although 10 cm is considered short for certain applications, such as telecommunications, it is often sufficient for medical imaging (e.g., brain imaging and diagnosis). It is observed that DNNs can recognize the digits used of the phase image inputs on the proximal facet of a GRIN fiber from their corresponding speckle patterns with high accuracy. These results were presented in our previous work and accurately verified here [16]. However, the trained DNN model with a dataset collected at a certain wavelength can efficiently perform classification of a speckle pattern that is generated using inputs recorded at the same wavelength only. For the fiber length used, drifting of the wavelength of the laser source by 3 nm causes the speckle patterns recorded at this new wavelength value to be incorrectly classified by the trained model. The performance of the DNN degrades and the classification accuracy abruptly drops, as reported in Figure 5b (red circles). Consequently, it is important that the DNN model is trained to in a way which allows classification of an input even if the corresponding speckle pattern is generated at different wavelengths. We found that one solution, which was also proposed for the case of bending by Fan et al. [20], is to include the potential perturbations in the measurements of the training dataset so that the DNN learns to interpolate correctly and properly classify the speckle outputs. However, it is observed that the classification accuracy saturates at values dependent on the size of the wavelength drift bandwidth, and even when increasing the size of the dataset, they never reach the maximum of 90%, which is the measured accuracy in the absence of drift. One possible reason for this result is the randomization of the input–output mapping; the large wavelength drift, the speckle pattern of a specific digit captured at a certain wavelength can be very similar to the speckle pattern of another digit captured at another wavelength. Another interesting observation based on the measurements reported in the presence of wavelength drift, is that the classification accuracy does not notably deteriorate after increasing the drift bandwidth further than 60 nm approximately. MMFs support modes whose shape is independent of the wavelength within a certain bandwidth. These modes are called principal modes [32,33]. However, this effect is usually observed for longer fibers of few meters and further investigation is necessary to evaluate if there is any contribution in the results presented in this work.

Until this point, we have discussed the aspect of training a DNN algorithm to recover information through a MMF system despite the wavelength changes by including the noise in the dataset. Nevertheless, there are cases where sensitivity in the wavelength change is actually crucial; for example, MMFs have been proposed as suitable systems for high-resolution spectrometers [34,35]. In those previous works, the speckle decorrelation with wavelength is used for distinguishing different wavelengths and the resolution is dependent on the speckle decorrelation bandwidth of the specific fiber, which becomes narrower with increases in the fiber length. In the results presented in this work, we observed that the classification efficiency of a trained DNN with data recorded at a single wavelength is sensitive to wavelength changes and deteriorates within a significantly narrower bandwidth (Figure 5b, red circles) than the decorrelation bandwidth of the system (Figure 5a). Therefore, what is considered unwanted for information recovery through the system under a wavelength drift situation, it could potentially be desired for achieving a resolution which is higher than the one given by the decorrelation bandwidth and has been previously proposed in the literature for spectroscopic applications.

5. Conclusions

We have shown that DNNs can recognize distorted images at the output of a MMF from their corresponding intensity speckle patterns. In particular, the deep learning approach presents a remarkable robustness against wavelength drift of the light source within an extended bandwidth for which total decorrelation of the speckle pattern intensity is observed. Despite the randomization of the input–output mapping introduced by the various speckle drifting, image classification of high accuracy of about 70% is reported for almost 100 nm wavelength drift, while more than 80% classification accuracy is shown for a 50 nm bandwidth. The results presented in this article suggest that deep learning can overcome distortion in the signal after propagation through MMFs in the presence of severe wavelength drift. The potential of using a single trained DNN model for retrieving image information in multiple wavelengths is suggested by using the intensity-only images of the respective speckle patterns, which could allow multicolor illumination imaging without extensive system recalibration.