Image Reconstruction Using Autofocus in Single-Lens System

Abstract

To reconstruct the wavefront in a single-lens coherent diffraction imaging (CDI) system, we propose a closed-loop cascaded iterative engine (CIE) algorithm based on the known information of the imaging planes. The precision of diffraction distance is an important prerequisite for a perfect reconstruction of samples. For coherent diffraction imaging with a lens, autofocus is investigated to accurately determine the object distance and image distance. For the case of only the object distance being unknown, a diffuser is used to scatter the coherent beam for speckle illumination to improve the performance of autofocus. The optimal object distance is obtained stably and robustly by combing speckle imaging with clarity evaluation functions. SSIM and MSE, using the average pixel value of the reconstructed data set as a reference, are applied on two-unknown-distance autofocus. Simulation and experiment results are presented to prove the feasibility of the CIE and proposed auto-focusing method.

1. Introduction

In computational imaging, phase retrieval (PR) is a tool to reconstruct a wavefront with diffraction images [1,2,3,4,5,6]. Recently, multi-intensity iterative algorithms [7,8,9,10] have shown stronger noise robustness, however, it is easy to bring an aliasing artifact into the system. Here the diffraction intensity patterns of an object at different distances are employed to improve the convergence of phase retrieval [11,12,13,14,15,16,17,18]. The single-beam, multiple-intensity reconstruction (SBMIR) algorithm [10] belongs to serial iteration. The amplitude-phase retrieval (APR) [16] is a parallel iterative algorithm that renews the complex amplitude of the target with the average of calculated data. The above iterative algorithms have been used to reconstruct the wavefront in CDI [17,18].

For diffraction integral models, such as angular spectrum and Fresnel diffraction, distance is a sensitive parameter for the quality of a reconstructed image. Especially in computational microscopies, higher imaging quality requires a more accurate measurement of the distance between optical elements. If the diffraction distance is inaccurate, the overall sharpness of the image will decline and the phenomenon of defocusing will occur [19,20]. Thus, auto-focusing algorithms were considered to search for an optimal position in imaging systems [21,22,23,24]. Clarity evaluation functions (CEFs) are defined by derivative, statistics, histogram, and some intuitive algorithms [25,26,27,28,29]. A well-established CEF comprises the numerical refocusing (propagation) of the recorded diffraction patterns to a set of distances and evaluating the quality of the propagated field using so-called focus metrics. The distance corresponding to the maximum or minimum of the CEF curve is considered as the indicator of the focal plane [30,31,32]. Different CEFs are adapted to sample images of different pixel types [33]. In a single-lens system, the autofocus of the object distance and image distance finds the mountain peak in two-dimensional space simultaneously.

Firstly, we propose a cascaded iterative engine (CIE) based on the characteristics of single-lens CDI in this paper. The iterative process is only carried out on the diffraction plane without considering the modulation effect of the lens and the unknown object and image distances in the system. The reconstructed image is obtained after the one-time inverse diffraction and the lens modulation of the iterative result. According to the idea of serial iteration and parallel iteration, the CIE can be divided into serial CIE (SCIE) and parallel CIE (PCIE). SCIE shows good convergence characteristics and robustness. Secondly, for a case in which image distance is the only unknown parameter, the diffraction patterns are recorded from the back focal plane of the lens. We insert a diffuser into the system and combine the classic CEFs to accurately obtain the object distance. Speckle illumination improves the sensitivity and accuracy of autofocus in a single-lens CDI as compared to coherent illumination. Thirdly, in the case that both the distances are unknown, we use two error functions, mean squared error (MSE) and structural similarity (SSIM) as CEF to obtain the object distance and image distance simultaneously. This method realizes the accurate acquisition of the auto-focusing curves in three-dimensional space. Simulations and experiments have been performed to test the performance of the auto-focusing scheme.

2. Methodology

The experimental layout of distance scanning and computational imaging is given in Figure 1. The system is mainly composed of four parts: fiber-optic laser with collimating lens, diffuser, single lens, and scientific CCD. The scanning operation of object distance $z_1$ is achieved by changing the position of the lens.

The scattered light through D irradiates the sample and travels a distance $z_0$ in the free space. The beam is further modulated by the lens ${\mathrm{L}}_2$, and then the diffraction images are recorded by CCD at the distance $z_2$. For autofocus, a set of speckle images are recorded and used to scan the distance $z_1$ by CEFs. Here the first recorded position is the back focal plane of the lens ${\mathrm{L}}_2$. For sample reconstruction, coherent illumination is applied for recording diffraction patterns. The first image $I_0$ is recorded at the focal plane of ${\mathrm{L}}_2$, and the others are recorded outside the back focal plane. The images recorded by coherent illumination will be used for image reconstruction after the auto-focusing task is completed.

2.1. Autofocus and Iterative Algorithm

Based on the single-lens CDI system in Figure 1, the insertion and removal of the diffuser realize the switch of speckle illumination and coherent illumination. In this paper, speckle illumination is used to realize autofocus and coherent illumination is used for image construction. The flowchart is given in Figure 2, which includes three parts:

• Reconstruction of the scattered out-of-focus dataset by using the intensity patterns under speckle illumination.
• Clarity evaluation of reconstructed speckle images, and curve drawing between estimated distances and clarity results.
• Reconstruction of coherent patterns by using the quasi-focus distance in step 2.

2.2. Clarity Evaluation Function

Clarity can be used as the standard of image evaluation [34]. The clarity evaluation index produces an extreme value (maximum or minimum value) only for the focused plane in an imaging system [30,33]. In this section, CEFs are utilized to evaluate the object distance $z_1$. We apply different auto-focusing criterias to estimate the position of a retrieved sample as CEF. The expressions are written as follows [23,35,36],

$$\mathrm{GRA}\left(z\right)=\int \int \mid \nabla I\left(z\right)\mid \mathrm{d}x\mathrm{d}y,$$

(1)

$$\mathrm{LAP}\left(z\right)=\int \int {\left[{\nabla }^{2}I\left(z\right)\right]}^2\mathrm{d}x\mathrm{d}y,$$

(2)

$$\mathrm{SG}\left(z\right)=\int \int {[\nabla I\left(z\right)]}^2\mathrm{d}x\mathrm{d}y$$

(3)

$$\mathrm{SPEC}\left(z\right)=\int \int ln\left\{1+\left|\mathcal{F}\left[I\left(z\right)-\overline{I}\left(z\right)\right]\right|\right\}\mathrm{d}x\mathrm{d}y,$$

(4)

$$\begin{array}{c}\hfill \mathrm{TOG}\left(z\right)=\sqrt{std\left(\right|\nabla i\left(z\right)\left|\right)/mean\left(\right|\nabla i\left(z\right)\left|\right)},i\left(z\right)=\sqrt{I\left(z\right)},\end{array}$$

(5)

$$\mathrm{Brenner}\left(z\right)=\sum _x\sum _y{\left[I(x+2,y,z)-I(x,y,z)\right]}^2,$$

(6)

$$Entropy\left(z\right)=-\sum _{i=0}^{L-1}{p}_{i}ln\left(p_i\right),$$

(7)

$$VAR\left(z\right)=\sum _x\sum _y{|I\left(z\right)-\overline{I}\left(z\right)|}^2,$$

(8)

$$\begin{array}{c}\hfill AM\left(z\right)=\frac{1}{MN}\sum \left|I(x,y,z)\right|,\end{array}$$

(9)

where ∇ is the gradient operator. $\left|·\right|$ is the modulus operator. The functions ‘std’ and ‘mean’ are the standard deviation and mean calculation. CEF is regarded as auto-focusing metrics by searching its maximum or minimum. In this paper, instead of creating new CEFs, we combine speckle illumination with CEFs to make existing auto-focusing algorithms more stable and robust. Since the speckle illumination mode makes a re-distribution of gradient data on recording planes.

2.3. Image Reconstruction

In lensless CDI, iterative calculations start from the sample plane after assuming the amplitude and phase of the sample. In single-lens CDI, for our CIE, only the known information is used for iteration. The sample, object distance, and image distance are not considered during iteration, and the modulation effect of the lens is temporarily avoided. The result of the iteration is the diffraction image on the first recoding plane, and the reconstructed sample is obtained after inverse diffraction and lens modulation. The process of PCIE is is given as follows: (1) We use the square root of the intensity pattern on the first recording plane while keeping the phase information unchanged; (2) Forward propagation in turn until all recording planes are traversed; (3) Inverse diffraction propagation to the first recording plane; (4) The above iterative process is repeated to obtain the reconstructed first recording plane diffraction image; (5) Inverse diffraction propagation to the object plane to obtain the reconstructed sample image, as shown by the purple arrows in Figure 3. In SCIE, the diffraction patterns of each recording plane are processed separately, and the estimated average values are used as the output result of each iteration. Finally, it is inverse-diffracted to the object plane as shown by the blue arrows in Figure 3.

The procedure of the CIE is represented as follows:

(1)

$I_n$ represents the pattern of scattered illumination on n-th imaging plane recorded by CCD. $U_n^{k}exp\left(\mathrm{i}{\beta }_n^k\right)$ represents the k-th complex-value guess when the image is on the n-th plane.

(2)

The light field functions of two adjacent imaging surfaces are propagated by the angular spectrum method as

$$U_n^{k}exp\left(\mathrm{i}{\beta }_n^k\right)={\mathbf{A}}_d\left[U_{n-1}^{k}exp\left(\mathrm{i}{\beta }_{n-1}^k\right)\right],$$

(10)

where ${\mathbf{A}}_d$ is forward angular spectrum propagation operator with a distance d. During the iteration, the real part $U_n^{k}exp\left(\mathrm{i}{\beta }_n^k\right)$ of k-th complex-value guess on the plane ${\mathrm{P}}_n$ is replaced by the root of measured intenisty $I_n$. The intervals between the adjacent detecting positions are equal in simulations and experiments.

(3)

When the iteration runs on the last diffraction plane $n=N$, the synthesized complex amplitude propagates backward to the first plane ${\mathrm{P}}_0$. And, ${\mathbf{A}}_{-N\mathrm{d}}$ represents the backforward angular spectrum propagation operator. The complex amplitude at the plane ${\mathrm{P}}_0$ is replaced with

$$U_0^{k+1}exp\left(\mathrm{i}{\beta }_0^{k+1}\right)={\mathbf{A}}_{-N\mathrm{d}}\left[U_N^{k}exp\left(\mathrm{i}{\beta }_N^k\right)\right].$$

(11)

Steps (2) and (3) will be implemented iteratively.

(4)

Convergence evaluation criteria in the current loop is achieved by the function

$${\Delta }_{min}=\sum \left|\left|U_n^n\right|-\sqrt{I_n^n}\right|,$$

(12)

The above steps describe the process of closed loop iteration. Based on the data set retrieved above, the auto-focusing process is listed as follows:

(i)

A specific range for object distance is selected for covering the actual distance.

(ii)

Supposing that $Uexp\left(\mathrm{i}\beta \right)$ is the complex amplitude at the plane ${\mathrm{P}}_0$ after M iterations, the complex amplitude of sample is obtained by back propagation and is expressed as

$$U_L\left(x_L,y_L\right)={\mathbf{A}}_{-z_2}\left\{U\left(x,y\right)exp\left[\mathrm{i}\beta \left(x,y\right)\right]\right\},$$

(13)

$$U_S\left(x_S,y_S\right)={\mathbf{A}}_{-z_1}\left\{t\left(x_L,y_L\right)U_L\left(x_L,y_L\right)\right\},$$

(14)

where $U_L$ and $U_S$ are complex amplitudes at the plane $P_L$ and the sample plane, respectively. The function $t(x,y)$ is the phase of the lens ${\mathrm{L}}_2$.

(iii)

By changing the distance $z_1$, several recovered complex amplitudes of sample are obtained with $U_{Sn}$ at the distance $d_n$, which is given as

$$d_n=d_s+\left(n-1\right)\Delta d,$$

(15)

where $d_s$ and $\Delta d$ represent starting distance and interval.

Finally, the object distance is sought by the operation on CEF as

$$z_1=find\left(d_n\right)=CEF\left[{\left|U_{Sn}\right|}^2\right].$$

(16)

The speckle illumination has superiority compared with coherent light for autofocus. Coherent illumination, however, can receive the reconstructed image with higher quality than the speckle case in our single-lens CDI, because speckle illumination will drown out the details and sharp edges of the sample image [37]. Thereby, coherent illumination is utilized for imaging. A set of diffraction intensity images by coherent illumination are recorded, when the diffuser is removed from the optical system. The complex amplitude of the sample can be retrieved by repeating the above iterative process with the evaluated distance $z_1$ calculated by step (iii).

The above steps are the serial iterative process. For the parallel process, the complex amplitudes are changed as

$$U_n^{k}exp\left(\mathrm{i}{\beta }_n^k\right)={\mathbf{A}}_S\left[\sqrt{A_{n-1}}exp\left(\mathrm{i}{\beta }_{n-1}^{k-1}\right)\right],$$

(17)

$$U_1^{k+1}exp\left(\mathrm{i}{\beta }_1^{k+1}\right)=\frac{1}{N}\sum _{n=1}^N{\mathbf{A}}_{-Nd}\left[\sqrt{A_n}exp\left(\mathrm{i}{\beta }_n^k\right)\right],$$

(18)

where $A_n$ ($n=2,3,\dots ,N$) is intensity images recorded from coherent illumination. The purple and blue arrows represent the iterative process of SCIE and PCIE respectively in Figure 3.

2.4. Speckle Model

The speckle model is given in the numerical simulation. Considering the phase modulation effect of the lens in the actual optical path, we simulate speckle illumination in complex form as

$$\mathrm{Pattern}=A_{z_0}\exp \left(\mathrm{i}\frac{3\pi }{2}Ran{d}_{M\times N}\right),$$

(19)

where ${\mathbf{A}}_{z_0}$ represents forward angular spectrum propagation operator with a distance $z_0$. $Ran{d}_{M\times N}$ is a random binary matrix.

3. Simulation and Experiments

3.1. Comparison of the Two Iteration Methods

PCIE and SCIE are verified in the single-lens CDI system. The numerical simulations of convergence speed and robustness are presented in Figure 4, Figure 5 and Figure 6. The reconstructed images from PCIE and SCIE are shown in Figure 4a1–a4,b1–b4. The reconstructed result of SCIE is distinguishable and normalized correlation coefficient (NCC) reaches 1 after 100 iterations, whereas the result of PCIE is heavily degraded with the same iterations. SCIE performs well in the convergence speed compared to PCIE.

The robustness comparison of PCIE and SCIE is proved in Figure 5 [38]. A zero-mean Gaussian noise is added in all coherent patterns, and the variance of the noise is set as 0.01, 0.05, and 0.1, respectively. Figure 5a1–a3,b1–b3 show the reconstructed results under the noise of different variances by using PCIE and SCIE, respectively, after 1000 iterations. The reconstructed quality of SCIE is visually better than that of the PCIE with the same noise. The corresponding convergence curves are shown in Figure 5c. As the noise level increases, the reconstructed quality decreases for these two methods. NCC values of SCIE are always better than that of PCIE for different noise levels.

For the same condition as the previous set of simulations, Figure 6 shows the reconstructed results of the sample when the number of diffraction pattern is different. Figure 6a shows the ground truth. The speckle pattern incident on the sample is shown in Figure 6b. Figure 6a1–a8 are reconstructed results obtained by SCIE and PCIE, when the number of images is 3, 6, 9, and 11, respectively, by using diffraction patterns. Figure 6b1–b8 show this using speckle patterns. The NCC curves of PCIE and SCIE are plotted in Figure 6c1,c2 for coherent patterns and Figure 6d1,d2 are plotted for speckle patterns. The curves show that, regardless of the illumination mode, the convergence speed of SCIE is always better than PCIE. SCIE can converge after about 20 iterations, while PCIE convergence requires at least 50 iterations. SCIE achieves a good reconstruction from the results based on the simulation results above. SCIE is a better choice for reconstructing samples in the single-lens CDI system.

For a perfect sample reconstruction, object distance and image distance, according to $z_1$ and $z_2$ in Figure 1, are needed to be determined accurately. However, during the acquisition of the images, the distance of $z_2$ can be determined manually. The focus position is identified as the starting position of the recorded image. At this time, $z_2$ is the focal length of the lens ${\mathrm{L}}_2$ in Figure 1. To verify the performance of the auto-focusing method proposed, the following two situations are considered.

3.2. Autofocus for Object Distance

When only the object distance, $z_1$ in Figure 1, is unknown. The positive effect of the diffuser on estimating the distance value will be checked in the simulation. The search boundary of $z_1$ is 10 mm∼60 mm. To compare the results of CEFs in both coherent and speckle illumination, all evaluation results are normalized. The CEF curves are shown in Figure 7. Binary sample and grayscale sample used here correspond to Figure 4 and Figure 6a, respectively. The curves in Figure 7a1,a2,b1,b2 denote the results with the coherent illumination and the curves in the Figure 7c1,c2,d1,d2 are for the speckle illumination. For the binary sample, it is obvious that the speckle illumination is helpful for a stable and robust searching focus for all CEFs except GRA. The original correct result of GRA under coherent illumination is destroyed. The sensitivity of the curves is significantly improved. For the grayscale sample, the autofocus curves of the two illumination modes show good unimodality and high sensitivity, except for TOG. The curve performance of GRA, Entropy, and AM has been improved. However, the originally good TOG curve was destroyed. In summary, a diffuser is more conducive to accurately estimating the diffraction distance by CEFs. Speckle illumination is helpful for a unique and robust auto-focusing search even using the simple metric function in the single-lens CDI system.

The experimental verification is made by using the experimental device shown in Figure 8. A fiber laser (532 nm) is collimated by a lens (f = 200 mm). A diffuser (DG10-120-MD, Thorlabs, 120 grit) serves as a scattering medium. The focal length of lens ${\mathrm{L}}_2$ is 35 mm. A CCD (3.1 $\mathsf{\mu }$m) is moved by a precision linear stage.

In the experiment, the sample is a calibration target engraved with ‘78’. The number of measured patterns is 11 and the interval is 1 mm. Figure 9a–d show the reconstructed results of SCIE and PCIE for the two illumination modes, respectively. Using diffraction patterns, the reconstructed results of PCIE are significantly better than that of SCIE. Even though the reconstructed image in Figure 9c has the twin-image effect and some noise around digits, Figure 9a shows the result of unsuccessful reconstruction after 1000 iterations. For the reconstructed results by using speckle patterns, the reconstructed results of PCIE and SCIE can both distinguish the number ‘78’, in Figure 9b,d, respectively. The reconstructed result of SCIE has higher contrast visually. Figure 9c1–c4,d1–d4 show the CEF curves for coherent and speckle illumination, respectively. Speckle illumination optimizes the large fluctuating CEF curves under coherent illumination. We cropped the information-rich area in the reconstructed image data set corresponding to the yellow box in Figure 9c,d. The clarity of this area is evaluated, and the corresponding CEF curves are shown in Figure 9c3,c4,d3,d4. When evaluating the clarity of the rich-information area of the image, speckle illumination still shows superiority. Therefore, speckle illumination plays a positive role in single-lens CDI object distance scanning, which not only makes the SQF curve unimodal but also improves the accuracy of the auto-focusing results.

The diffuser provides random illumination for the imaging system, which will disorganize the tilt phase factor and counteract the lateral shift effect of intensity pattern. This property is called the memory effect of speckle [39,40]. The speckle pattern is inherently random, so oblique illumination will not affect it. Furthermore, speckle illumination can eliminate the aliasing artifact and enhance the robustness [32].

3.3. Autofocus for Object Distance and Image Distance

The other case in the single-lens CDI we consider is two-unknown-distance autofocus, the object distance, and image distance. In Section 3.2, speckle illumination played a positive role in one-unknown-distance autofocus. We expect it to work for autofocus in two-unknown-distance autofocus. Then we test the experimental data set ‘78’. Based on the distance obtained in Figure 9, we reset the object distance scanning boundary 50 mm∼90 mm, the image distance scanning range from 15 mm∼55 mm, and the step length is 1 mm. The results are shown in Figure 10a1–a9,b1–b9. In a large range and long step length autofocus, neither coherent illumination nor speckle illumination can give a rough focus trend. Therefore, in the single-lens CDI, when the object and image distances are auto-focusing scanned simultaneously, the diffuser makes no sense.

Therefore, for single-lens CDI dual-distance autofocus, a new focus strategy is proposed. Usually, MSE and SSIM are used to detect the effect of image retrieval in simulation [41,42,43,44,45] and the reference value of the function is the ground truth. In the experiment, the ground truth of the sample is unavailable. Therefore, the pixel average is selected as the reference value here. Their expressions are as follows

$$\mathrm{MSE}\left(U_{S_n},I_{ref}\right)=\frac{1}{M}\sum _{n=1}^N{\left(U_{S_n}-I_{ref}\right)}^2,$$

(20)

where $U_{S_n}$ has been defined in Equation (14), M represents the number of pixels and $I_{ref}$ is the pixel mean of the $U_{S_n}$, its data set is written as

$$I_{ref}=\frac{1}{n}\sum _{n=1}^N{U}_{Sn}.$$

(21)

Here the structural similarity (SSIM) function is shown as

$$\mathrm{SSIM}\left(U_{Sn},I_{ref}\right)=\frac{2{\mu }_{U_{Sn}}{\mu }_{I_{ref}}+c_1}{{\mu }_{U_{Sn}}^2+{\mu }_{I_{ref}}^2+c_1}\frac{2{\sigma }_{U_{Sn}{I}_{ref}}+c_2}{{\sigma }_{U_{Sn}}^2+{\sigma }_{I_{ref}}^2+c_2},$$

(22)

where ${\mu }_{U_{S_n}}$ is pixel mean value of image $U_{Sn}$, ${\mu }_{I_{ref}}$ is pixel mean value of image $I_{ref}$. ${\sigma }_{U_{Sn}}^2$ and ${\sigma }_{I_{ref}}^2$ are the variances of images $U_{S_n}$ and ${\mu }_{I_{ref}}$. ${\sigma }_{U_{S_n}{I}_{ref}}$ represents the covariance of the two images. $c_1={\left(k_{1}L\right)}^2$ and $c_2={\left(k_{2}L\right)}^2$ are constants used to maintain stability. L is the dynamic range of pixel values. Generally $k_1=0.01$, $k_2=0.03$.

A negative 1951 USAF target (R3L3S1N, Thorlabs) is used as the sample and the focal distance is 100 mm. The results of MSE and SSIM are normalized. Figure 11a1–a3,b1–b3 show the rough auto-focusing curves of NMSE and NSSIM with the search boundary of 65 mm∼75 mm and 95 mm∼105 mm, the step is 1 mm. The highest peak can be identified in the CEF curves of $z_2$, but not unimodal by using NMSE and NSSIM. We refine the search interval to 0.01 mm with the same search boundary. Figure 11c1–c3,d1–d3 show the fine auto-focusing curves. The results obtained by NMSE are $z_1$ = 69.9 mm, $z_2$ = 100.5 mm while the results of NSSIM are $z_1$ = 70 mm, $z_2$ = 100 mm. As for the CEF curves, $z_2$ of NMSE steal has two peaks, NSSIM shows 2D unimodal.

The retrieved sample with the above two sets of auto-focusing results is shown in Figure 11e1,e2,f1,f2. Figure 11e2,f2 correspond to the content of the yellow area in Figure 11e1,f1, respectively. Figure 11g corresponds to the pixel information of the red and blue lines in Figure 11e2,f2, respectively. The line drawing position is the sixth line pair of the fourth group of the resolution board. The image resolution retrieved based on the auto-focusing results of NMSE and NSSIM are almost the same. MSE and NSSIM can realize automatic focusing at the same time as the object distance and the image distance in single-lens CDI. SSIM shows its superiority in focusing accuracy.

4. Conclusions

In this work, we insert a diffuser into the single-lens CDI system to improve the accuracy and robustness of the autofocus algorithm. Our proposed SCIE algorithm can reconstruct samples with fast iteration speed and robustness in a single-lens CDI. We proposed the use of NMSE and NSSIM in two-unknown-distance autofocusing with taking the pixel mean value of the reconstructed image data set as the reference input. The 2D CEF curves show good unimodality. This paper provides new strategies for reconstruction samples and auto-focusing in the single-lens imaging system, which provides imaging solutions for CDI systems with a lens.