Super-Resolution Image Reconstruction of Wavefront Coding Imaging System Based on Deep Learning Network

Abstract

Wavefront Coding (WFC) is an innovative technique aimed at extending the depth of focus (DOF) of optics imaging systems. In digital imaging systems, super-resolution digital reconstruction close to the diffraction limit of optical systems has always been a hot research topic. With the design of a point spread function (PSF) generated by a suitably phase mask, WFC could also be used in super-resolution image reconstruction. In this paper, we use a deep learning network combined with WFC as a general framework for images reconstruction, and verify its possibility and effectiveness. Considering the blur and additive noise simultaneously, we proposed three super-resolution image reconstruction procedures utilizing convolutional neural networks (CNN) based on mean square error (MSE) loss, conditional Generative Adversarial Networks (CGAN), and Swin Transformer Networks (SwinIR) based on mean absolute error (MAE) loss. We verified their effectiveness by simulation experiments. A comparison of experimental results shows that the SwinIR deep residual network structure based on MAE loss optimization criteria can generate more realistic super-resolution images with more details. In addition, we used a WFC camera to obtain a resolution test target and real scene images for experiments. Using the resolution test target, we demonstrated that the spatial resolution could be improved from 55.6 lp/mm to 124 lp/mm by the proposed super-resolution reconstruction procedure. The reconstruction results show that the proposed deep learning network model is superior to the traditional method in reconstructing high-frequency details and effectively suppressing noise, with the resolution approaching the diffraction limit.

1. Introduction

In digital imaging systems, providing a sharp image with a higher imaging frame rate is often desired [1,2,3]. However, in conventional optical imaging system, the quality of an image rapidly degrades when the object deviates from the focal plane. Conversely, constrained by the transmission bandwidth of the detector, increasing the frame rate is commonly achieved through two methods, but one involves sacrificing field of view, while the other entails sacrificing detector resolution. The approach of sacrificing field of view involves cropping the image and only outputting part of the field of interest (sub-sampling) [4,5,6]. This method reduces the field of view range and has significant limitations in its application. The approach of sacrificing resolution is to extract specific pixel output (skipping) [7], which maintains the field of view while resulting in a significant reduction in image details [8]. In addition, there exists another application scenario in which the ambient illumination is low. To maintain the frame rate without reducing it, it is necessary to expand the size of the photosensitive pixel. The detector operates in the binning mode, where the current responses of adjacent photosensitive units are added together and merged for output, at which point the imaging resolution is reduced. Both operational modes will result in a substantial loss of information in the image, so the reconstruction of high-resolution details based on low-resolution images holds immense significance in practical applications [9,10,11].

The question of how to break through the diffraction limit of optical systems and achieve super-resolution has always been a research hotspot in academic circles. In the microscopy field [12,13,14,15], super-resolution fluorescence microscopy technology [16,17], which utilizes quenching fluorescent molecules and the principle of quenching by stimulated radiation, has broken through the diffraction limit that conventional optical microscopes cannot reach, achieving clear observation of biological structures within 200 nm.

With the development of computer technology and image processing techniques, super-resolution is no longer referred to as exceeding the physical diffraction limit [18,19]. In fact, in numerous imaging fields, the pixel size of the detector is still insufficient to reach the diffraction limit of the optical system. Despite the rapid development of the CMOS process in recent years, the physical size of pixels has become progressively smaller, which will result in a significant increase in noise. Improving the resolution while maintaining the size of the pixels holds significant research value. The significance of super-resolution is not only to expand the size of the image but also more importantly to enhance the details that are not present in low-resolution images. Methods for achieving super-resolution consist of multi-frame reconstruction [20,21] and single-frame reconstruction [22]. The multi-frame super-resolution technique reconstructs the details of a high-resolution image by inputting a series of low-resolution images with sub-pixel shifts and using the complementary information between the low-resolution images. Typical algorithms are the iterative back-projection method [23,24] and convex set projection method. Single-frame image super-resolution reconstruction obtains a high-resolution image by inputting a single-frame low-resolution image after some mapping relationship, which is more challenging due to the absence of complementary information. Currently, single-frame super-resolution reconstruction is mainly realized through two types of methods, namely interpolation-based methods [25] and learning-based methods [26]. Interpolation-based methods include nearest neighbor interpolation, bi-linear interpolation, and bi-cubic interpolation. These methods are applied to expanding image size, but only improve the visual effect of the image without enhancing the image details.

Learning-based methods establish the relationship between low- and high-resolution mapping by training on numerous pairs of low- and high-resolution image samples [27]. The classical method is the super-resolution reconstruction method based on sparse representation [28], which constructs the low- and high-resolution images as an over-complete dictionary during the learning process, and first obtains the sparse representation coefficients of the low-resolution image during the inference process, and then unites the over-complete dictionary of the high-resolution image to obtain the high-resolution image. This method has a strong ability to enhance details, but is limited by sparse expression capabilities and is still a shallow learning method. In recent years, deep learning networks have made substantial advances in many computer tasks, such as image classification [29,30], object detection [31,32], and semantic segmentation [33,34]. Furthermore, Li et al. [35] established links between an interpolated low-resolution image and a super-resolution image using a Hierarchical Swin Transformer (HST) network. Da et al. [36] proposed a channel attention residual enhanced Swin Transformer denoising network (CARSTDn) for image denoising. In infrared polarization imaging, Ma et al. [37] proposed an infrared polarization super-resolution reconstruction detection system based on sparse representation with the Micro-scanning Polarization Projection on Convex Sets (MPPOCS) algorithm to address pseudo-polarized information, image downsampling and mosaicking. For deblurring, Kong et al. [38] showed an improvement compared to other works.

In our work, we propose a method for super-resolution degraded imaging reconstruction by wavefront coding (WFC) technology and deep learning networks. Under the condition of detector downsampling output, super-resolution image reconstruction is achieved through deep learning networks based on the high-resolution point spread function (PSF) in an WFC imaging system. We use three representative deep learning networks to simulate the performance of the super-reconstruction algorithm on the impacts of mean square error (MSE) loss [39], perceptual loss [40], and mean absolute error (MAE) loss [41]. Firstly, we perform a comparative analysis for two convolutional neural network (CNN) structures with different depths. Then, we compare the transformer network structure based on MAE with the neural network structures. Finally, through simulation experiments and real scene reconstruction experiments combined with an actual wavefront coding system, we find that the transformer network structure based on MAE performs the best, also validating the effectiveness of the super-resolution reconstruction method.

2. Wavefront Coding Super-Resolution Reconstruction Principle

Wavefront coding is a depth-of-focus (DOF) technology combining optical coding and digital image processing. The wavefront of the optical system can be coded and modulated by placing a specially designed phase mask at the exit pupil of the optical system, so that the system generates a focus-invariant PSF and optical transfer function (OTF). In the magnitudes of OTFs (MTFs), the cut-off frequency corresponding to the diffraction limit represents the system’s maximum resolving ability. The WFC system ensures that the MTF become almost invariant regardless of the defocus value without any zeros, providing the necessary information closer to the cut-off frequency for super-resolution reconstruction. The process of obtaining coded images can be expressed as:

$$I_{coded}={\left(I_{ground\_true}\ast h+N\right)}_{\downarrow }$$

(1)

where $I_{coded}$ denotes the encoded and captured intermediate image, $I_{ground\_true}$ denotes the original high-resolution image, h denotes the acquired PSF, “*” denotes the convolution operation, N denotes Gaussian noise, and “$\downarrow$” indicates downsampling.

To demonstrate that a wavefront coding system combined with a deep learning network can be effectively applied to super-resolution reconstruction, we utilized a visible light lens–combined WFC imaging system [42] meticulously designed by our team. Diagrams of a traditional imaging system and a WFC system, and the corresponding MTFs with different image distance are presented in Appendix A. As can be seen from Figure A1c, the MTF analysis reveals that the cut-off frequency of traditional imaging systems is significantly lower than the diffraction limit, resulting in a substantial loss of imaging information, which seriously affects imaging quality. In contrast, though the MTFs of the WFC system are approaching zero with increasing frequency and at much lower frequencies than with the traditional lens, the system contains no zero-frequency response. Additionally, as the object distance deviates from the optimal focus position, the amplitude of the curve does not change significantly. This consistency provides better conditions for subsequent super-resolution reconstruction. Figure 1 provides a schematic of super-resolution reconstruction with the WFC system (optical encoding procedure) and computational imaging (digital decoding procedure). A phase mask is placed at the pupil position of the optical system, thereby the system produces focus invariant OTFs and sampled PSFs. The intermediate images are captured by the detector and downsampled, which can be reconstructed via a digital procedure using the sampled PSFs.

Currently, there is limited exploration of super-resolution techniques in the field of wavefront coding [43,44]. Conventional optical systems are optimized so that the aberration is suppressed to the minimum state. When the physical size of the pixel is lower than the theoretical limit of the resolution of the optical system, the system operates in an undersampling state, resulting in signal aliasing. In the wavefront coding system, the MTF decreases, and PSF dispersion size undergoes expansion, thus effectively reducing the aliasing effect. According to the above theoretical analysis, the PSF corresponding to the detector sampling interval is tentatively referred to as the low-resolution PSF, and the PSF corresponding to half the size of the sampling interval is called as the high-resolution PSF. The wavefront coding super-resolution decoding technique can be implemented by the process shown in Figure 2a. If the high-resolution PSF is directly used to filter the low-resolution intermediate image, the image will produce a very severe ringing effect. Therefore, it is necessary to first interpolate the low-resolution image to a resolution twice the original resolution, both horizontally and vertically. Afterward, the image can be restored to a high-resolution image through Wiener filtering [45] and other methods [46].

We propose a method to increase the imaging frame rate based on the above wavefront coding super-resolution theory combined with the designed large-aperture wavefront coding system [42]. By making the detector operate in the downsampling mode, the horizontal and vertical resolution of the image is 1/2 of the original, and the total number of pixels is 1/4 of the original, compared with the full-pixel output mode. The PSF of the central field of view at the optimum viewing distance is measured, and the PSFs of the full-pixel and the downsampling are obtained, respectively, as shown in Figure 2a.

In this paper, we use the PSF of the actual system as the prior of blurring for reconstruction simulation and a deep learning network to replace the bi-linear interpolation process in Figure 2a. In order to improve the operation efficiency of the algorithm, the processes of super-resolution and image decoding are merged to achieve end-to-end super-resolution reconstruction, so that the network inputs the low-resolution coded image and outputs the super-resolution reconstruction image directly. This process is shown in Figure 2b. During this process, the deep learning model is trained to learn the degradation process of the ground truth image as described in Equation (1). This enables the network to reconstruct clearer images from low-resolution encoded images. Compared with the conventional super-resolution tasks, the network is not only responsible for image resolution enhancement, but also for image decoding.

3. Super-Resolution Reconstruction Deep Learning Network

For the task of image reconstruction, deep learning networks have shown excellent performance for image denoising and blind deblurring. In order to learn the mapping F from observed image $I_{\psi }\left(x,y\right)$ to ground truth $O\left(x,y\right)$, the architecture of super-resolution reconstruction mainly consists of three operations: patch feature extraction, non-linear transformation, and reconstruction. The wavefront coding process can be simulated by blurring the PSF and adding noise, and downsampling the blurred image simulates the downsampling process of the detector. The downsampling blurred image is used as the low-resolution intermediate image input to the network, with the original prior-PSF blurred image as the output ground-truth image.

In order to test various super-resolution reconstruction algorithms based on deep learning networks, we tried three network structures. The first is super-resolution CNN (SRCNN), a convolutional neural network based on MSE loss with fewer network layers. The second network structure is super-resolution GAN (SRGAN), a generative adversarial network based on perceptual loss with a relatively complex network. The last network structure is image reconstruction Swin Transformer (SwinIR), a Swin Transformer network based on MAE loss with more complex network architecture and more parameters.

3.1. Convolutional Neural Network Based on MSE Loss

The SRCNN is of great significance in super-resolution algorithms, inspiring many subsequent approaches. The network achieves super-resolution reconstruction through three convolutional layers, where each convolutional layer corresponds to a stage in the reconstruction process, namely the block feature extraction, nonlinear mapping, and reconstruction stages. The network structure is shown in Figure 3. Taking the first layer Conv9-64-s1 as an example, this indicates that the convolution kernel size of the convolution layer is 9, the number of feature layers is 64, and the convolution step size is 1.

In the block feature extraction stage, a series of convolution kernels is used to extract features from low-resolution images and convert them into abstract features, where the $i\mathrm{th}$ feature layer can be represented as:

$$T_1\left(i\right)=f\left(W_1\left(i\right)\ast I+b_1\left(i\right)\right)$$

(2)

where $W_1\left(i\right)$ and $b_1\left(i\right)$ represent the convolution and bias of the i-th filter, the symbol ∗ is the convolution, and f is the nonlinear activation function, here, using a nonlinear rectification unit (ReLU) [47]. For single-channel image input, each convolution kernel has a spatial size of $1\times C_1\times C_1$, and the output contains $n_1$ feature layers equal to the number of convolutional layers.

After feature layer extraction, these features need to be converted to higher dimensions through nonlinear mapping. This process can be expressed as:

$$T_2\left(i\right)=f\left(W_2\left(i\right)\ast T_1+b_2\left(i\right)\right)$$

(3)

Assuming that there are $n_2$ high-dimensional feature layers in total, and the convolution kernel has a single-dimensional space size of $C_2$. Then, according to the size of $C_2$, the nonlinear mapping can be classified into two types: when $C_2$ is equal to 1, this transformation can be interpreted as a single-pixel mapping; and when $C_2$ is greater than 1, the receptive field of the convolution is broadened and the influence of adjacent pixels is introduced into the transformation. Therefore, any number of convolutional layers can be added during this stage, and the number of convolution parameters between the $\left(j-1\right)\mathrm{th}$ layer and the $j\mathrm{th}$ layer is $\left(n_{j-1}\times C_j\times C_j+1\right)\times n_j$.

In the reconstruction stage, the high-dimensional features need to be converted into super-resolution images, and the process can be expressed as:

$$\widehat{O}=W_m\left(i\right)\ast T_{m-1}+b_m\left(i\right)$$

(4)

where $\widehat{O}$ is the final predicted image. Unlike the previous two stages, the convolution in this stage does not require a nonlinear activation function, so the reconstruction process is more like a weighted average of high-dimensional features. Assuming that the final output is a single-channel image, the number of convolution parameters in this process is $n_{m-1}\times C_m\times C_m$.

In addition to the three-layer convolution structure, we add a convolution layer (the second convolution layer in Figure 3 in the nonlinear mapping stage to determine the impact of network depth on the super-resolution results. Including the bias parameters, SRCNN-915 with the three-layer convolutional structure contains 8129 parameters, while SRCNN-9515, with a convolutional layer added, contains 110,593 parameters, which is 13.6 times that of SRCNN-915.

In the training stage, the network needs to learn the mapping relationship from the low-resolution intermediate image to the ground-truth image. Assuming that the parameters to be learned by the network are denoted as $\theta =\left\{W_j,b_j\right\}$, $j=\left\{1,2,\cdots ,1\right\}$, the loss function of the network is defined as the Euclidean distance between the network output and the ground-truth image, which is MSE. The network is trained with the aim of minimizing the loss function. The process can be expressed as:

$$\widehat{\theta }=\underset{\theta }{argmin}\frac{1}{N}\sum _{k=1}^N{∥\widehat{O}\left(k\right)-O\left(k\right)∥}^2$$

(5)

where N is the number of training patches.

3.2. Conditional Generative Adversarial Network Based on Perceptual Loss

The second super-resolution reconstruction algorithm used is a generative adversarial network (GAN), or more precisely a conditional generative adversarial network (CGAN). A traditional GAN contains a generative network and a discriminative network. The generative network needs to learn the mapping relationship from the noisy image to the real image, while the discriminative network needs to discriminate the difference from the output of the generative network and ground-truth. Through alternate training, the generative network and the discriminative network continue to “evolve”, and eventually output a “false” image that is very close to the real image. Based on the principle of GANs, the input of CGAN is not limited to a noisy image, it can be a blurred or low-resolution image, and the network eventually outputs a super-resolution image.

We employ a generative network containing several convolutional layers and residual blocks [48], which is shown in Figure 4. The final stage of the network contains an inverse convolutional layer to achieve the image resolution improvement, and the last activation function is tanh, which maps the output to between (−1, 1). To test the impact of network depth on the super-resolution reconstruction results, we reduced the number of residual blocks in the generative network from 16 (SRGAN-16ResBlocks) to 8 (SRGAN-8ResBlocks). This modification resulted in 780,739 parameters after the reduction, which is almost half of the initial parameter count of 1,374,659.

The loss function of the generation network consists of three parts, consisting of an MSE, auxiliary training network, and discriminant network, respectively, expressed as:

$$L_G=L_{MSE}+\alpha L_{adv}+\lambda L_{vgg}$$

(6)

where MSE represents the pixel difference between the generated image and the ground-truth image. The VGG loss provided by the auxiliary training network is also referred to as the perceptual loss, which is defined as the Euclidean loss between the feature representations of a decoded image $\widehat{O}$ and the reference image O:

$$L_{vgg}=\frac{1}{W_{i,j}{H}_{i,j}}\sum _{x=1}^{W_{i,j}}\sum _{y=1}^{H_{i,j}}{∥{\varphi }_{i,j}{\left(\widehat{O}\right)}_{x,y}-{\varphi }_{i,j}{\left(O\right)}_{x,y}∥}^2$$

(7)

where ${\varphi }_{i,j}$ represents the feature map of the j-th layer convolution before the i-th maximum pooling layer in the VGG19 network structure. Both i and j are used as 4 in this study. The incorporation of perceptual loss enhances the realism of the style and texture of the generated image. The loss provided by the discriminant network is the Wasserstein distance, defined as:

$$L_{adv}=\sum _{n=1}^N-D_{{\delta }_D}\left(G_{{\delta }_G}\left(I\right)\right)$$

(8)

where ${\delta }_D$ and ${\delta }_G$ describe parameters in the discriminator and generator, respectively, and N denotes the number of training samples. The discriminatory network adopts the structure proposed in the literature, comprising five convolutional layers. To avoid the issue of gradient disappearance or explosion in WGAN (Wasserstein GAN) during training, the loss of the discriminator adopts the WGAN-GP form, denoted as:

$$L_D=\underset{{\widehat{O}}^{˜}{P}_g}{\mathbb{E}}\left[D\left(\widehat{O}\right)\right]-\underset{O^{˜}{P}_r}{\mathbb{E}}\left[D\left(O\right)\right]+{\lambda }_B\underset{Y^{˜}{P}_Y}{\mathbb{E}}\left[{\left({∥{\nabla }_{Y}D\left(Y\right)∥}_2-1\right)}^2\right]$$

(9)

where $P_r$ denotes the distribution of the real data, $P_g$ denotes the distribution of the output data of the generative network, and $P_Y$ denotes the randomly interpolated sampling on the line connecting $P_r$ and $P_g$, that is $\widehat{O}=\epsilon O_r+\left(1-\epsilon \right){\widehat{O}}_g$.

3.3. Swin Transformer Based on MAE Loss

The third super-resolution reconstruction algorithm used is the shifted windows version of the hierarchical vision transformer (Swin Transformer), known as SwinIR. The Swin Transformer utilized in the algorithm is based on the standard multihead self-attention of the original Transformer layers, which integrates the advantages of both CNN and Transformer, and demonstrates tremendous potential for various applications. On the one hand, it has the advantage of CNN of being able to process images of large size due to the local attention mechanism, while, on the other hand, it has the advantage of Transformer of being able to model long-range dependence with the shifted window scheme.

The model mainly consists of three components, namely the shallow feature extraction, deep feature extraction, and high-quality image reconstruction modules. The deep feature extraction module is primarily composed of residual Swin Transformer blocks (RSTBs), where each RSTB contains multiple layers of Swin Transformers for local attention and cross-window interaction. Additionally, a convolutional layer is appended at the end of each block for feature enhancement, while residual connections are used to facilitate the aggregation of features. Finally, shallow and deep features are fused in the reconstruction module to achieve high-quality image reconstruction. The structure of the network model used in this paper is shown in Figure 5. The deep feature extraction module of the model includes 6 RSTBs and a $3\times 3$ convolutional layer, in which each RSTB is a residual block composed of several Swin Transformer layers (STL) and a $3\times 3$ convolutional layer.

In the shallow feature extraction stage, shallow feature extraction is performed for three parameters: image height, and width and number of input channels of the input low-resolution blur image, and this process can be expressed as:

$$F_0=H_{SF}\ast L_{LQ}$$

(10)

where $L_{LQ}$ and $H_{SF}$, represent the low-resolution image and a $3\times 3$ convolutional layer used for shallow feature extraction, respectively, and the symbol ∗ is the convolution. Subsequently, the intermediate features are extracted block by block from the shallow features $F_0$ and then outputted, this process can be expressed as:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{F}_i=H_{{RSTB}_i}\left(F_{i-1}\right),i=1,2,\dots ,K\\ F_{DF}=H_{CONV}\ast F_K\end{array}\right.\end{array}$$

(11)

where $H_{{RSTB}_i}$ denotes the $i\mathrm{th}$ RSTB, while $H_{CONV}$ represents the final convolutional layer. The incorporation of a convolutional layer at the end of feature extraction can bring the inductive bias of the convolution operation into the Transformer-based network, and establish a better foundation for the later aggregation of shallow and deep features. Furthermore, within each RSTB, which comprises six STLs and a convolutional layer, and the process of extracting intermediate features from the six STLs for a given $i\mathrm{th}$ RSTB with input feature $F_{i,0}$ can be expressed as:

$$F_{i,j}=H_{{STL}_{i,j}}\left(F_{i.j-1}\right),j=1,2,\dots ,6$$

(12)

where $H_{{STL}_{i,j}}$ represents the $j\mathrm{th}$ STL in the $i\mathrm{th}$ RSTB. Then, a convolutional layer is added before the residual join. This process can be denoted as:

$$F_{i,out}=H_{{CONV}_i}\ast F_{i.L}+F_{i,0}$$

(13)

where $H_{{CONV}_i}$ is the convolutional layer in the $i\mathrm{th}$ RSTB.

In an STL, the attention function is executed for h times in parallel, and the results are concatenated to establish multihead self-attention (MSA). Then, a multi-layer perceptron (MLP) is used for further feature transformations, which contains two fully connected layers with GELU non-linearity between them. LayerNorm (LN) layers are added before both the MSA and MLP, and residual connections are used for both modules. The whole process can be represented as:

$$\left\{\begin{array}{c}{X}_2=MSA\left(LN\left(X\right)\right)+X\\ Y=MLP\left(LN\left(X_2\right)\right)+X_2\end{array}\right.$$

(14)

where X and Y denote the input features and output features of each STL, respectively. Finally, once the deep features have been extracted, the deep features and shallow layers of the image are upsampled and adjusted for aggregation through sub-pixel convolutional layers in the reconstruction module to reconstruct a high-quality image.

In the training stage, the network uses the L1 pixel loss in the selection of the loss function, which measures the average absolute difference between each pixel value in the generated image and the corresponding pixel value in the target image, also known as MAE. The L1 pixel loss is a more widely used loss function in image super-resolution, and the network is trained with the aim that the generated image needs to be as same as possible as the target image. The process can be expressed as:

$$\widehat{\theta }=\underset{\theta }{argmin}\frac{1}{N}\sum _{k=1}^N{∥\widehat{O}\left(k\right)-O\left(k\right)∥}^1$$

(15)

where N is the number of image pairs used in each iteration of training as above.

4. Simulation and Imaging Experiment

The dataset is a crucial factor in assessing the effectiveness of deep learning networks. After comprehensive consideration of the quality and quantity of images in each dataset, we finally selected the DIV2K data set [49], which contains 800 training images and 100 testing images, all of which have a resolution of 2K. For the generation of the training set, the wavefront coding process can be simulated by blurring the known PSF and adding noise. The downsampled blurred image is used as the low-resolution intermediate image input to the network.

4.1. Super-Resolution Reconstruction Simulation Model

For the SRCNN structure, we discretely sampled 800 original high-definition images from the DIV2K dataset at 100 pixels intervals, obtaining a total of 235,904 image pairs. The batch input for a single iteration contains 128 image pairs for a total of 2 million iterations, which is equivalent to 1000 epochs of training on the training set. Stochastic gradient descent (SGD) and momentum methods are used for training, with the learning rate set to 0.0001. Training on the Titan-XP GPUs required a total of approximately 12 h for the SRCNN-915 structure and 34 h for the SRCNN-9515 structure.

For the SRGAN structure, in each training, a ground-truth image of 384 × 384 size is intercepted from the 2K large image, and the input is a degraded image with a size of 192 × 192 at the corresponding location. The input for a single iteration contains four image pairs and is trained on the training set for 1000 epochs using the Adam optimizer with the learning rate set to 0.0001 for the first 500 cycles and 0.00001 for the last 500 cycles. Training on the Titan-XP GPUs required a total of approximately 46 h for the SRGAN-8ResBlocks structure and 58 h for the SRGAN-16ResBlocks structure.

For the SwinIR structure, images of 96 × 96 size were intercepted from the 2K large image for each training, while the input size of the low-resolution degraded image was 48 × 48. The input for a single iteration contained three to four image pairs, which were trained for 1000 epochs on the training set using the Adam optimizer with the learning rate set to 0.002. Unlike the previous training equipment, the SwinIR structure was trained by a 3060ti GPU, requiring 91 h for training. However, as the training performance under 1000 epochs was not satisfactory, subsequent experiments opted for a model trained for 3000 epochs for analysis and comparison.

After completing the network training, we validated the network’s results on three test sets, namely Set 5, Set 14 [50], and DIV2k-Valid. For comparison, we adopted the Wiener filtering method to decode the low-resolution coded image and adopted the method in the literature [51] for super-resolution decoding. The method employed in the literature [51] involves interpolating the low-resolution intermediate coded images to obtain images with both horizontal and vertical resolutions twice that of the original. Subsequently, super-resolution decoding is performed using Wiener filtering in conjunction with the PSF of the wavefront coding system. The final reconstruction results are shown in Figure 6, Figure 7 and Figure 8. For each figure, (a) displays the low-resolution ground-truth image, (b) displays the low-resolution coded image, (c) displays the image that directly decodes a low-resolution coded image, (d) displays the super-resolution decoded image generated Wiener filtering method referred in the literature [51], (e) and (f) denote the super-resolution reconstruction results based on the SRCNN network, (g) and (h) represent the super-resolution reconstruction results based on the SRGAN network, and (i) represents the super-resolution reconstruction results based on the SwinIR network.

Comparing the results, it can be seen that the signal-to-noise ratio of the image generated by the super-resolution decoding algorithm in the literature [51] is marginally higher than that of the low-resolution decoded image, but the improvement effect is not obvious, as numerous details of the image are submerged in the noise. Overall, the super-resolution reconstruction effect using deep learning networks is superior. The reconstruction effect of SRCNN tends to retain a certain degree of blur in the images, and the edges are not sharp enough. In comparison, the reconstructed images based on the SRGAN network has richer details and more realistic subjective feeling. A comparison of the results obtained from different network depths indicates that the network depth has little impact on the results in the SRCNN network structure, while in the SRGAN network, increasing the network depth can bring more details to the images.Furthermore, the reconstructed image based on the SwinIR network exhibits a higher signal-to-noise ratio compared to the SRGAN network structure, while the edge artifacts of the reconstructed images are better eliminated and the details are closer to the original image.

The super-resolution reconstruction images from the three test sets were quantitatively analyzed by peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), respectively, and the results are shown in

Table 1. Peak signal-to-noise ratio (dB) and structural similarity of super-resolution reconstruction images.

	Set 5	Set 14	DIV2K-Valid
double cubic interpolation	23.4834/0.8261	22.0780/0.7287	24.1407/0.7581
literature [51]	24.0998/0.8435	22.2493/0.7560	25.3001/0.7839
SRCNN-915	29.7636/0.9452	26.6595/0.8701	29.6897/0.9124
SRCNN-9515	29.8285/0.9462	26.6706/0.8704	29.7487/0.9128
SRGAN-8ResBlocks	31.9230/0.9609	28.4228/0.9146	30.7186/0.9352
SRGAN-16ResBlocks	31.5226/0.9579	28.1487/0.9057	30.3244/0.9187
SwinIR-6RSTBlocks	33.8065/0.9214	30.1541/0.8595	32.7359/0.9105

. A comparison shows that the signal-to-noise ratio of the super-resolution restored images based on the deep learning network is much higher than that of the interpolation method and the method adopted in the literature [51]. Among them, SRGAN-8ResBlocks performs the best on SSIM with an average enhancement of 0.1424 compared to the method in the literature [51], while SwinIR-6RSTBlocks exhibited the greatest performance on PSNR with an average enhancement of 8.4 dB compared to the method adopted in the literature [51]; however, this network has an average enhancement on SSIM of only 0.103.

Comparing the different network depths, the addition of one network layer in the SRCNN structure does not significantly impact the PSNR results, while in the SRGAN structure it can be detected that the signal-to-noise ratio of the 16-residue block is worse than that of the 8-residue block, which seems to be inconsistent with the subjective perception, and this phenomenon has also been mentioned in the literature [48]. The reason for this phenomenon may be that the network trained based on perceptual loss introduces some “false” components during the reconstruction process and these “false” texture details can be more pleasing to the human visual perception system. However, analysis of the pixel values indicates that the “false” information results in an amplification of the difference between the reconstructed image and the real image. Therefore, the performance evaluation based on PSNR does not entirely correspond to the subjective experience. Compared to the SRGAN network structure with the 8-residue block, the SwinIR structure is capable of further enhancing the PSNR while significantly reducing the presence of “false” texture details, thereby narrowing the gap between the reconstruction image and the real image.

Next, we input a color image with a resolution of 2040 × 1536 and tested the running time of the four convolutional neural networks on the Titan-XP GPU and the results are shown in

Table 2. Comparison of the runtimes of super-resolution reconstruction algorithms using deep learning networks.

	SRCNN-915	SRCNN-9515	SRGAN-8ResBlocks	SRGAN-16ResBlocks	SwinIR-6RSTBlocks
Running time	0.0198 s	0.0612 s	1.1974 s	1.4811 s	3.2560 s

. The reconstruction algorithm based on the SRCNN-915 network structure has a shorter running time of about 0.02 s, and the time is prolonged to three times of the original after adding one layer of network depth, so this network structure is suitable for the case of high real-time requirements; in contrast, the running time based on the SRGAN-8ResBlocks network structure is longer, taking about 1.2 s, and the time is about 1.5 s after increasing the network depth, so this network is more suitable for offline processing. The training equipment for the SwinIR-6RSTBlocks network structure differs from the four aforementioned convolutional neural networks. Due to the limited memory resources available on the training devices, adjustments to the network hyperparameters have a discernible impact on the network operational speed. Thus, testing SwinIR on a color image with a resolution of 1040 × 768 still results in a relatively lengthy running time, approximately 3.3 s. Nevertheless, this network is equally suitable for offline processing.

4.2. Super-Resolution Image Reconstruction Experiment

Based on the previous simulation experiments and the correlation between the reconstruction results and the running time, we finally selected the SRGAN-8ResBlocks network structure and SwinIR-6RSTBlocks network structure as the super-resolution reconstruction algorithms for real imaging.However, since the two networks have their own advantages in terms of PSNR and SSIM performance evaluations, it is necessary to combine them with a real-world scene for comparative analysis in order to select the most suitable super-resolution reconstruction algorithm.

Firstly, in order to test the super-resolution performance of the proposed algorithms with an WFC system, we carried out an experiment using a resolution test target in the laboratory environment, as shown in Figure 9. The wavefront coding lens is designed and fabricated with a focal length of 58 mm, an F number of 0.75, and an optimal imaging distance of about 100 m, and attached on a CMOS (shown in Figure A2) that has 1600 × 1200 pixels with the physical pixel size of 4.5 $\mathsf{\mu }$m. The WFC system acts as a camera and the corresponding sensor-limited spatial resolution is approximately to $\frac{1\mathrm{mm}\times 1000}{2\times 4.5\mathsf{\mu }\mathrm{m}}\approx 111$ lp/mm. To simulate the object plane locating at as far as 100 m, we use a collimator with focal length of 1200 mm and effective aperture of 100 mm. The details of the experimental setup can be found in Appendix C. During the experiment, the test target is positioned close to the focus of the collimator based on the optimal imaging distance. The focal length of the parallel light tube and the optical system have a magnification relationship of the object and image, which can be approximated as:

$$\beta \approx \frac{f_{collimator}}{f_{lens}}=\frac{1200\mathrm{mm}}{58\mathrm{mm}}=20.67$$

(16)

Because of this magnification relationship, the spatial frequency on the resolution test board is $1/\beta$ of the spatial frequency on the image plane. Therefore, the actual spatial frequency corresponding to the lines of group 8 is about 165 lp/mm, of group 6 is about 124 lp/mm, and of group 4 corresponds to 83 lp/mm.

The results of the target test are shown in Figure 10. Figure 10a shows the coded low-resolution intermediate image captured by the WFC system and downsampled, the corresponding sensor-limited spatial resolution of which is approximately 55.6 lp/mm. Finally, the coded intermediate images were reconstructed using the sampled PSF. Figure 10b shows the decoded image using the bicubic interpolation method, in which the lines of group 4 with a spatial resolution of 83/2 = 41.5 lp/mm can be clearly distinguished, while those of group 6 are almost indistinguishable. Figure 10c displays the super-resolution decoded image following the approach detailed in the literature [51], in which the lines of group 6 can be separated but remain blurred. Figure 10d displays the image obtained through super-resolution reconstruction using SRGAN, in which the lines of group 6 can be partially separated, indicating a noticeable super-resolution effect. Figure 10e displays the image obtained using SwinIR for super-resolution reconstruction, in which the lines of group 6 with a spatial resolution of 124 lp/mm are fully separable, demonstrating the most prominent super-resolution effect.

To quantify the performance of our super-resolution reconstruction, we plotted the intensity profiles along the blue dashed line in Figure 10d and the red dashed line in Figure 10e, as shown in Figure 11. Figure 11 shows the intensity distribution of resolution lines of the ideal resolution test target (grey line) and the SwinIR decoding (red line) and SRGAN decoding (blue line) result images. The corresponding full width at half maximum (FWHM) data and the average FWHM value are listed in

Table 3. FWHM for corresponding lines ($\mathsf{\mu }$m).

	1	2	3	4	5	6	7	8	9	Average
SwinIR	0.22	0.23	0.22	0.23	0.24	0.23	0.23	0.22	0.24	0.229
SRGAN	0.28	0.25	0.23	0.25	0.25	0.24	0.26	0.24	0.22	0.247

. Through analysis of the image and corresponding FWHM data, we observe that SwinIR decoding achieves higher resolution, greater uniformity, and more effective reconstruction of image details compared to SRGAN. Overall, the intensity distribution of SwinIR’s lines aligns more closely with the ideal resolution lines. The experimental results show that the proposed method can achieve super-resolution from 55 lp/mm to 124 lp/mm, and the resolving ability is better than that of the original detector (111 lp/mm).

We conducted experiments in outdoor scenes using the assembled camera (WFC) system and the experiment conditions are shown in Figure A4. The results are shown in Figure 12, Figure 13 and Figure 14. In each figure, (a) is the coded low-resolution image captured by the detector and downsampled, (b) is the decoded image using Wiener filtering, (c) is the decoded super-resolution image of the method in the literature [51], (d) is the super-resolution reconstructed image based on the SRGAN-8ResBlocks network structure, and (e) is the super-resolution reconstructed image based on the SwinIR-6RSTBlocks network structure. By judging the image quality by subjective feeling, it can be observed that the super-resolution reconstruction algorithms of the latter two networks are more advantageous in terms of image clarity. Among them, the super-resolution reconstruction algorithm based on the SwinIR-6RSTBlocks network provides richer detail reconstruction, better aligning with subjective visual effect, and exhibits superior performance in detail resolution ability and clarity.

The three images within the actual scene were objectively evaluated using the average gradient [52] evaluation criterion, and the results are shown in

Table 4. Comparison of the average gradients of super-resolution reconstruction images for different real-world scenes.

	Low- Resolution Coded Image	Super-Resolution Decoded Image from the Literature [51]		Super-Resolution Reconstruction Image Based on the SRGAN Network		Super-Resolution Reconstruction Image Based on the SwinIR Network
		Average Gradient Value	Upgrade Ratio	Average Gradient Value	Upgrade Ratio	Average Gradient Value	Upgrade Ratio
Scene I	5.0664	5.6306	11.13%	8.2671	63.18%	8.6006	69.76%
Scene II	4.0588	4.2864	12.99%	6.0935	50.13%	7.0950	74.81%
Scene III	4.7866	5.2424	9.52%	7.0592	47.48%	7.5522	57.88%

. Based on the average gradient values, it is apparent that the reconstruction method based on the SRGAN-8ResBlocks network can significantly enhance the contrast of the images, but the effect is limited and requires further enhancement in terms of sharpness. In Scene II, it has been observed that the reconstruction method based on the SwinIR-6RSTBlocks network can improve the average gradient value by 74.81%. The image gradient can be improved, resulting in higher contrast and clearer images after reconstruction. The improvement in image sharpness is more pronounced, and the reconstruction of image details is richer.

5. Conclusions

In this paper, we investigate a super-resolution reconstruction method in detector downsampling mode with improved frame rate based on wavefront coding technology. By utilizing a combination of deep learning networks to achieve super-resolution image reconstruction through the high-resolution PSF of the coding process, we investigate three representative deep learning network structures, and achieved training of the networks by modifying the depths of two of them. The simulation results show that the super-resolution algorithm based on deep learning network is superior to the traditional super-resolution decoding algorithm, and the network structure SwinIR based on Swin transformer obtained from the MAE loss training can produce more realistic super-resolution images. Finally, the combination of the proposed algorithm and a wavefront coding system has been demonstrated to improve the resolving ability of the system from 55.6 to 124 line pairs per millimeter during a resolution board test, while the average gradient of the image was enhanced by approximately 67% during a real-world scene experiment. To address the issues of low SSIM and extended training and running times of the model, we will implement a series of adjustments and optimizations in future research.