Long-Range Imaging through Scattering Media Using Deep Learning

Abstract

Imaging through scattering media is an important and challenging problem, and the technology has been used in many fields, such as autonomous driving, industrial inspections, remote sensing imaging, and biomedical imaging. However, most of the previous experiments used numbers or letters for close-range imaging, while objects in life are colorful. In this study, a new deep learning network, DesUNet, was constructed to image realistic objects at medium and long distances under sunlight through scattering media, and to realize object recognition. In addition, this study also compares the imaging results of different neural networks, and the results show that the DesUNet network improves the feature information storage ability and enhances the image reconstruction. It not only clearly restores the original appearance of the object, but also extracts the physical information about the object. In order to further verify the power of the DesUNet network, this study also conducted indoor near distance and outdoor medium distance imaging experiments. For indoor reconstructed objects, the appearance of the objects could be clearly identified. For outdoor reconstructed objects, the confidence level could reach above 0.9 through YOLO. The experiments show that the DesUNet network has good robustness and generalization.

1. Introduction

Light is a powerful means of imprinting and recording the characteristics of objects in real time on rewritable molds [1]. The different properties of light, such as intensity and phase distributions, polarization and spectra, allow us to perceive the reflectance and thickness distributions, as well as the birefringence and spectral absorption properties of an object [2]. When light interacts with objects, it is usually characterized by multiple scattering and absorption events [3]. When light passes through some fog, heavy rain, clouds, biological tissues, and other strong scattering media, inter-particle interactions change the wavefront phase of the incident light. As a result, the original orderly light field of the incident light becomes chaotic, which ultimately leads to a loss of information about the object.

The rapid development of photonics has accelerated the development of scattering optics techniques in optical imaging [4]. The simplest and most common form of optical imaging is the use of lenses to reproduce the intensity of light scattered by an object on an imaging plane, where it can be measured by a high-pixel detector [5]. For the inverse problem of scattering medium imaging, if light is scattered during propagation, the wavefront phase of the incident light changes due to inter-particle interactions, resulting in the originally ordered light field of the outgoing light becoming chaotic, and the light from each point on the object will be dispersed to multiple detector pixels. This leads to optical model errors and uncertainties in the imaging process, as well as the need to process a large amount of scattering data, thus requiring the support of high-performance computing devices. In addition, practical applications may be interfered by ambient light, requiring appropriate optical design and processing methods to reduce the interference. These limitations make it difficult for conventional imaging methods to effectively solve the inverse problem of scattering medium imaging.

In order to solve these limitations, researchers have proposed a variety of scattered light imaging methods, such as optical phase conjugate scattering imaging technology [6] and optical transmission matrix scattering imaging technology [7,8]. Based on the optical memory effect scattering imaging technology [9,10,11], researchers have used these techniques to achieve focusing or imaging through the scattering medium. However, each of these methods has its own advantages, disadvantages, and applicable conditions, such as the optical memory effect based scattering imaging technique failing in the face of thick scattering media, and also failing when the size of the observed object is large. In particular, when scattered by thick scatterers, the signal of ballistic light is very weak and the scattering ‘noise’ very strong, which makes the classical optimization algorithms for solving inverse problems unable to help.

Recent studies have shown that when deep learning algorithms are used instead of the traditional inverse problem [12,13,14], they are able to effectively recover object images from recorded scattering, thus overcoming the limitations of traditional methods and are faster, more reliable, and more accurate for data processing, image segmentation, and image reconstruction [15,16,17,18,19,20,21,22,23,24]. They are powerful in solving complex computational imaging problems [25], super-resolution imaging [26], and polarization imaging [27]. With the rapid development of artificial intelligence, deep learning algorithms are making huge improvements to optical imaging [3,28,29,30,31,32,33,34]. In terms of imaging, the earliest study by Lyu et al. completed the use of deep learning through scattering medium imaging [35]. Although the construction of the neural network is very simple, and can only carry out a simple digital reconstruction, this opens up a new path of scattering imaging technology. Subsequently, Li proposed the use of convolutional neural networks to achieve coherent imaging through thin frosted glass [36], and one year later, Lyu also proposed a hybrid neural network for imaging experiments through scattering media [37], which further enriched the through scattering media imaging technology by improving on the original neural network. Ma et al. proposed a plug-and-play algorithm based on a generalized alternating projection optimization framework, and in order to be able to achieve imaging through scattering media in an interfering environment, they combined this algorithm with neural networks and the Fienup phase recovery method [38]. Cheng et al. took the scattered autocorrelation information as the physical constraint, proposed a two-level neural network that realizes background light denoising and object reconstruction through deep learning, and constructed an end-to-end deep neural network, thus overcoming the interference of background light, and the interference of object reconstruction [39]. Zheng et al. built an end-to-end deep neural network that can overcome the interference of background light [40,41], while Zhang et al. designed a space-frequency dual-domain deep neural network using the frequency-domain characteristics of speckle [42]. However, current deep learning-based approaches are still poor in their ability to generalize to unknown scenarios. In 2021, the Zheng research group further realized underwater dynamic incoherent scattering imaging [40], in which the researchers put a fat milk solution in a water tank to simulate the dynamic scattering scenario. In the experiment, in addition to testing the traditional MINST digital set, the image recovery of more complex gray objects was also realized, and a two-camera method was proposed to measure the dynamic characteristics of scattering media (decoherence time). In 2023, Liu et al. proposed a method to obtain a quantitative scattering fog pattern based on a passive luminescence electronic ink screen to solve the problem of data acquisition [43]. The E ink screen simulates passively glowing natural objects to obtain data pairs for training the network and testing fog maps in real scenes. However, the above neural networks all had defects:, the feature storage ability of the image was relatively weak, their experimental objects were simple numbers and letters, for complex real objects, and the neural network they built may not be able to achieve a clear reconstruction. Because real objects are complex and changeable, with more feature information, it is necessary for neural networks to store a large amount of feature information, numbers and letters have simple structures and obvious features, and the performance of neural networks is not high. The traditional neural network has many problems. Even if the training can eventually converge, the existence of scattering medium leads to excessive noise interference in the image, which may lead to an overfitting phenomenon [44]. Due to the weak ability of storing object feature information, the trained network is difficult to generalize and the final imaging effect is poor.

The neural network used in this study will focus on solving these problems, using deep learning to improve the U-net neural network, which is called DesUNet here. In this study, a real scene is taken as the target object for experiments, and the scattering medium imaging is really applied to life. Through the experimental results, it can be proved that the DesUNet neural network has better performance for storing feature information. Firstly, indoor objects are used for imaging through scattering media, and the reconstruction results of other traditional neural networks are constructed and compared with the DesUNet neural networks, which proves that the DesUNet neural network is better than other neural networks in storing feature information. After that, the application scene will be transferred to the outdoors. In contrast to other imaging experiments through scattering media, the target objects used in this study were people, cars and other complex objects in real scenes. This neural network is used to image through the scattering medium in the middle and long distance (1–2 km), complete the experiment, and recognize the reconstructed object. Through indoor and outdoor experiments, it was finally verified that the neural network had better performance for storing feature information.

2. Rationale

2.1. Construction of the DesUNet Neural Network

Scattering imaging is a nonlinear reversible problem that is addressed using deep learning, which compensates for the shortcomings of traditional imaging methods such as complexity and high computational demands. By harnessing the computational power and fitting capability of deep learning, it facilitates the fitting calculation of scattering media and the final imaging results. The mathematical model of its imaging system is as follows:

O_P = F(I_P)

(1)

where O_P is the output of the scattering imaging system, I_P is the input of the scattering imaging system, and F is a positive function of the input light field to the output light field.

Since the optical path is reversible and the absorption of light by the scattering medium is neglected, the mathematical model from the output light field to the input light field is as follows:

I′_P = F⁻¹(O′_P)

(2)

where I′_P is the input to the inverse process, O′_P is the output of the inverse process, and F⁻¹ is the inverse of F.

The result is the ideal mathematical model for the problem of imaging through scattering media. As shown in Equations (1) and (2) above, there is a correspondence between the scattering image and the original image of the target object, and the functions are handed over to deep learning, since the ultimate goal of imaging through scattering media is to make the inverse input approximately equal to the positive input.

Due to the weak storage ability of traditional neural networks for object feature information and to solve this problem, a new neural network named “DesUNet” was builtin this study. The structure of the constructed DesUNet neural network is similar to that of the U-Net neural network [45], as shown in Figure 1. A U-shaped architecture was made to connect low-level features with high-level features, enabling better capture of features at different levels. Due to the network’s architecture, it can be divided into two parts: the down-sampling process and the up-sampling process. The down-sampling process reduces the dimensions of the image and retains valuable information to some extent, helping to prevent overfitting. The up-sampling process then restores the reduced image to its original size. The purpose of building the DesUNet neural network is to increase the storage capacity of the object feature information. Considering this, a dense block was built, as shown in Figure 2. Its role is to stack the features extracted by each layer of the neural network that precedes it. Feature maps can be reused between different layers of the network, enhancing the flow of gradients, thus improving feature utilization efficiency and greatly enhancing the model’s performance and generalization capabilities. Importantly, the feature information from each layer is fully utilized. For example, the feature information from the first layer is fused with that from the second layer, the feature information from the first layer is fused with that from the third layer, and the feature information fused in the previous two steps is fused again. Each dense block consists of Batch Normalization layers, Activation layers (using ReLU as the activation function), Convolutional layers (Conv layers), Dropout layers (used to reduce overfitting), and Concatenate layers. The aim is to obtain more target object feature information. The input is the original object image with dimensions of 256 × 256 pixels, along with the corresponding speckle image. These inputs go through four Conv layers, dense blocks, and Max Pooling layers, resulting in a reduction in the size of the target object and feature extraction. During image processing in the neural network, the smallest image size is 16 × 16 pixels, but it has a high channel count of 1088, which contains rich feature information. This is 10–20 times more feature information than conventional neural networks, making it much better suited to handling complex objects. The inclusion of Batch Normalization at each layer significantly improves the training speed of the network, and the network’s generalization capability is further strengthened. Following this, a path consisting of four Conv layers, dense blocks, and Concatenate layers (including deconvolution operations to restore the image to its original size and improve the neural network’s accuracy and efficiency) is utilized. Subsequently, through Conv layers, dense blocks, and Conv layers, the output goes to the final layer. The final layer’s convolution integrates the results of processing the input data across all previous layers, ultimately generating the reconstructed result image.

Compared with other neural networks, the DesUNet neural network has several advantages. First, due to its structural design, it can combine low-level feature information with high-level feature information, and this ability enables it to better represent and understand the low-level features of complex objects, which makes it better able to capture the features of target objects. Secondly, due to the addition of the dense block, the neural network can improve the efficiency of feature utilization and retain the spatial information of the image. Third, it can provide more powerful feature expression capabilities, resulting in more accurate reconstruction results and image segmentation. Fourth, it can reduce the computational complexity of the network while maintaining high performance, and improve the computational efficiency of the network, which is very valuable for practical applications requiring real-time or resource-efficient processing.

2.2. Experimental Setup

The experimental setup, as shown in Figure 3, differs from most imaging through scattering media experiments, which typically involve illuminating the target object with lasers to enhance its features. In this study, experiments were conducted under natural light (sunlight) conditions to better simulate real-world scenarios and demonstrate the powerful feature processing capabilities of the DesUNet neural network. The detector used is an integrated photoelectric sensor mounted on a platform (HP-DMA2186), as depicted in Figure 4. The scattering medium in the imaging system comprise 2 mm ground glass diffusers (polishing specifications: 220 grits), which are pressed against the lens of the detector. The target object under investigation can be any arbitrary item. The indoor object is placed about 5–10 m away from the detector, and the outdoor object is 1–2 km away from the detector. This experimental configuration allows for imaging through scattering media using natural light, aiming to showcase the DesUNet neural network’s ability to handle feature-rich information in a real-world setting.

2.3. Indoor Data Acquisition and Processing

Firstly, the imaging through scattering medium experiments were conducted indoors in order to fully prepare for the outdoor experiments, and the most targeted object was a schoolbag for testing (the schoolbag was chosen because of its distinctive features), and data collection was conducted for more than 300 h. In order to collect the training data, the object was placed at different distances and different postures, and three groups of experiments were conducted, where the bag was placed 5 m, 7 m, and 10 m away from the detector for data collection. In each group, 8000 original images of the object and 8000 corresponding scattered images were collected and the data used as training data. All images were 256 × 256 pixels and all were in color. The choice of color images aimed to simulate real-world scenarios more accurately, as color images contain richer information. It is worth noting that the “cv2.imread” function in Python reads images in the BGR format (where B stands for blue, G for green, and R for red), so the displayed image colors may appear differently. To address this, the final reconstructed results were adjusted to be displayed in a heatmap-like form and in grayscale.

3. Experimental Results and Analysis

3.1. Scattering Imaging of Indoor Objects

The reconstructed results obtained from training the test dataset with the DesUNet neural network, as shown in Figure 4, appear quite satisfactory. Comparing the original images with the reconstructed ones, it is evident that DesUNet excels in handling details, and is particularly successful in reconstructing the edges of the objects. From the indoor experiments, it can be concluded that DesUNet demonstrates excellent object reconstruction capabilities. However, it is important to note that this alone does not imply that other neural networks cannot perform well in this experiment. It showcases DesUNet’s strong reconstruction ability but doesn’t directly prove its superior feature information storage capability.

To further demonstrate the high performance of the DesUNet neural network, a straightforward approach is to conduct comparative experiments. Specifically, traditional CNN neural networks and U-Net neural networks were set up, and all three neural networks were used for imaging through scattering media experiments. The reconstructed results from these experiments are shown in Figure 5. From the results, it is evident that imaging through scattering media yields different outcomes for the same object at different distances and with different neural networks. Among them, the DesUNet network exhibited the best imaging performance. The reconstructed results based on the traditional CNN neural network, while recognizing the general shape of the object, appeared highly blurred and only showed the rough outline. This indicates the weaker feature processing capability of traditional CNN neural networks. On the other hand, the reconstructed results based on the U-Net neural network, while revealing the basic appearance of the object, lacked clear edge information. This suggests that although the U-Net neural network can handle some feature information, it falls short of the expected results for this experiment. It is important to note that this is a simple backpack, and outdoor scenes are more complex, so achieving such results may not always be possible. In contrast, the reconstructed results based on the DesUNet neural network displayed sharp edge contours and clear object features, essentially restoring the original appearance of the object. Through indoor experiments, it can be concluded that both the U-net neural network and DesUNet neural network have certain capabilities of processing feature information, and to prove that the DesUNet neural network has a stronger capability of processing feature information, it is necessary to conduct imaging experiments through scattering media in outdoor real scenes.

The comparison between the original image of the target object and the reconstruction results of different neural networks was carried out by utilizing SSIM (structural similarity), assuming that the two input images are X and Y, where the formula for SSIM is as follows:

$$\mathrm{SSIM}\left(X,Y\right)={\displaystyle \frac{\left(2{\mu }_X{\mu }_Y+C_1\right)\left(2{\delta }_{XY}+C_2\right)}{\left({\mu }_X^2{\mu }_Y^2+C_1\right)\left({\delta }_X^2{\delta }_Y^2+C_2\right)}}$$

(3)

where μ_X and μ_Y denote the mean of X and Y respectively, δ_X² and δ_Y² denote the variance of X and Y respectively, δ_XY denotes the covariance of X and Y, C₁ and C₂ in order to prevent the denominator from being a constant of 0. The value of SSIM is in the range of [0, 1], and the larger the value means the smaller the gap between the two plots. The final comparison result graph is shown in Figure 6. According to the SSIM score, the reconstruction result based on the DesUNet neural network is the closest to the original image.

3.2. Scattered Imaging Outdoors

By comparing indoor experiments, it can be concluded that the DesUNet neural network exhibits good performance. However, indoor experiments involving simple target objects do not necessarily indicate that DesUNet will perform similarly in real-world scenarios. To address this, outdoor experiments were conducted. Real-world applications often involve complex and diverse scenes, far from just simple digits and letters. To better simulate real scenarios, this experiment used real outdoor backgrounds with vehicles and pedestrians as target objects for imaging through scattering media. In the outdoor environment, we collected a large number of outdoor road pictures and scattered images as a training set. The training data content included three major categories of people, cars and the road environment, where each person’s appearance and morphology were different, and cars were divided into cars, electric cars, bicycles and other car vehicles. In order for the experiment to be closer to the real situation, the data collection time spanned more than 1000 h, because every day’s road conditions may be different. eEach set of photos were color RGB images, through a long period of training and we finally selected one of them as the training results, as shown in Figure 7. It is evident that the real road conditions’ image and the reconstructed image bear a striking resemblance, illustrating the DesUNet neural network’s ability to perform imaging through scattering media in realistic outdoor scenarios.

The training results met expectations, and subsequently, speckle images of different targets were collected as a test dataset for imaging through scattering media experiments. The targets included people, cars, motorcycles, and bicycles. The reconstructed results for different neural networks, focusing on the target regions, are presented in Figure 8. Notably, the target objects are located at distances of 1–2 km from the detector. It is evident that the reconstructed results based on the DesUNet neural network were quite satisfactory. They provided a clear view of the target objects, demonstrating that the DesUNet neural network had the capability to perform imaging through scattering media in complex outdoor environments and at medium to long distances. For the sake of experimental completeness, imaging through scattering media experiments were also conducted with CNN and U-Net neural networks. However, it is clear that these two neural networks, while able to reconstruct the backpack indoors, struggled to perform well when the target objects were moved outdoors. In outdoor scenarios, these two neural networks fell short of the expected results, emphasizing the DesUNet neural network’s superiority in such challenging conditions.

3.3. Recognize Objects after Imaging through a Diffuse Image

To test the accuracy of the neural network in this study, the YOLO (You Only Look Once) algorithm was used for object recognition in the reconstructed results. The YOLO algorithm is based on a single neural network and performs region proposal generation. It does not differentiate between foreground and background but instead classifies and regresses the target objects. One major advantage of YOLO is its ability to significantly improve object detection speed while reducing the number of object parameters and computational load. Using YOLO for object recognition in the reconstructed results based on the DesUNet neural network, the recognition results are shown in Figure 9. Observing the confidence of target objects, the confidence includes two aspects. One aspect is the likelihood that the bounding box contains an object (Pr(object)), where Pr(object) = 1 when the object is present, and Pr(object) = 0 when the object is absent. The other aspect is the accuracy of the bounding box, represented by the Intersection over Union (IoU) between the predicted box and the actual box. The confidence is displayed after the predicted object name for easy observation. From the results, it is clear that the target objects in the reconstructed images were accurately identified.

Through the experiments conducted, this study successfully demonstrated the ability to perform imaging through scattering media in complex outdoor environments and at medium to long distances. It also highlighted the high performance of the DesUNet neural network. In combination with this study, it becomes evident that imaging through scattering media technology can be applicable in real-world environments, extending beyond simple digit and letter reconstruction. This has significant implications for various fields, including transportation, outdoor scenarios, and military applications, where imaging through scattering media can play a crucial role.

4. Conclusions

In the context of imaging through scattering media technology, the experiments conducted in this study have demonstrated that the DesUNet neural network significantly improves the accuracy of reconstructed images through scattering media. The incorporation of dense blocks in the DesUNet network serves multiple purposes, including preventing gradient explosion or vanishing, enhancing network generalization capabilities, and ultimately resulting in clearer and more distinct reconstructed images through scattering media. Comparative experiments were conducted with traditional CNN and U-Net neural networks, and the results clearly show that traditional neural networks produce highly blurred reconstructions, especially in complex conditions. In contrast, the DesUNet neural network not only achieves clear reconstructions at short distances but also excels in reconstructing images through scattering media under complex outdoor conditions and at medium to long distances, coupled with object recognition. The DesUNet neural network has successfully demonstrated imaging through scattering media in real-world environments. This technology has versatile applications across various fields. In the transportation sector, for instance, in vehicular or stationary settings, it can provide clear visibility of the road ahead, even in foggy conditions, allowing for immediate adjustments, and reducing the occurrence of traffic accidents. In outdoor scenarios with complex environments (such as forests or cloudy skies), where much of the field of view is obstructed, this technology can be employed for precise localization and object identification. In future research work, we will continue to optimize the neural network of reconstructed objects after scattering and further improve the network structure. Of course, in addition to optimizing the deep learning algorithm, the design of optical imaging systems is also very important. It is important to obtain more dimensions of optical field data or realize pre-processing methods of optical field regulation to break through the lower limit of imaging depth. On this basis, a more advanced algorithm is designed to post-process the acquired multidimensional data, which can further improve the recovery effect. With the development of optoelectronic devices and the improvement of algorithm computing power, scattering imaging research is expected to further organically combine the front and back ends, stimulate new ideas, deal with more complex application scenarios and objects, and achieve further improvements in imaging depth and recovery quality.