A Hybrid Variable-Resolution GI without Prior Information

Abstract

Ghost imaging (GI) is a novel imaging technique which realizes the fluctuation of the target light field through pattern modulation and obtains the target information in a non-local area. Compared with traditional imaging, GI has the advantages of a simple structure, strong anti-interference ability and imaging resolution beyond the diffraction limit. Illumination patterns are very important for GI, and they are divided into uniform resolution patterns and variable-resolution patterns. The variable-resolution patterns have the characteristics of high resolution in the foveal region and low resolution in the edge region, which can quickly improve the imaging efficiency of the foveal region. But there are still mass mutations and the inability to adapt imaging. Therefore, we propose a hybrid non-resolution-pattern design method and a two-step projection strategy. Simulations and experiments show that the proposed two-step projection strategy can accurately establish the corresponding relationship between the foveal region and the region of interest. The hybrid variable-resolution pattern design method can solve the problem of mass mutation between the foveal region and the edge region of fixed foveal patterns.

1. Introduction

Ghost imaging (GI) is a novel imaging technique which uses the correlation characteristics of light field intensity to recover the information of the scene to be measured. The idea of GI originates from the entanglement behavior of photon pairs under spontaneous parametric conversion. In 1995, Shi et al. [1] used quantum entanglement light sources to achieve GI for the first time in the laboratory. Because GI has the advantages of a strong anti-interference ability [2], wide spectrum [3,4] and high sensitivity [5,6], it has attracted many scholars to study it. GI has been widely used in hyperspectral imaging [7], remote sensing imaging [8] and medical imaging [9,10,11].

With the increasing maturity of GI methods, and the expansion from quantum light sources to classical light sources, the research focus of GI has gradually inclined towards the practical and engineering directions. In the process of its practical application, there have been studies from the perspective of lighting mode, for example, from pseudo-thermal lighting [12,13] to modulated-light lighting [14,15,16]. There are also studies from the perspective of reconstruction algorithms, for example, from a classical association algorithm [17] to compressed sensing algorithm [18,19,20]. Among them, illumination patterns [21,22], as the front end of GI, also play a key role in subsequent reconstruction. In order to solve the problem that a large number of sampling times are needed to obtain clear images in GI, in 2017, Phillips et al. [23] proposed a foveal pattern design method which was inspired by the foveal characteristics of animal vision. The number of pixels in each pattern is reduced by radially increasing the size of pixels away from the high-resolution foveal region. And the sampling times of a full sampling are equal to the total number of pixels in the patterns. The imaging efficiency is improved by reducing the sampling times of full sampling. The bionic patterns in this system are composed of a foveal region with high uniform resolution and a region with variable resolution obtained by log-polar transformation. Cheng et al. [24] proposed a pattern design method based on the region of interest, which is known as interest-guided Compressed Sensing GI (R-CSGI). The prior information is established by extracting the features of the local target region, and the frequency spectrum value of the low-frequency region is obtained by using the Fourier basis patterns. Then, pattern design is carried out on the reconstructed image, and variable-resolution patterns are generated to guide the design of the area of interest. Then, the generated patterns are used to sample the target, and the final reconstructed image is obtained by a compressed sensing reconstruction algorithm. Under the same compression ratio, the image quality of R-CSGI is significantly better than that of traditional CSGI.

Most of the corresponding relationships between the foveal region of variable-resolution patterns and the region of interest are set manually and cannot be obtained automatically. Moreover, only the fixed patterns with variable resolution in the foveal region are projected, which will show the imaging quality difference between the foveal region and the edge region. How to solve the mass mutation phenomenon between the foveal region and the marginal region is also a difficult problem to achieve high-quality imaging. The existing research on foveal-pattern GI focuses on improving the imaging quality of the foveal region, but this method has certain limitations when there is more than one object of interest in the scene. The hybrid non-uniform resolution GI method proposed in this paper can self-image the region of interest, and improve the image quality of the edge region to a certain extent. We proposed the two-step projection strategy. First, the target object is undersampled, and the preliminary reconstructed image obtained determines the location of the foveal region. Then, the fixed foveal region and random foveal region variable-resolution patterns are designed, and the two are mix-projected. Through simulation and experiment, it is proven that the proposed two-step projection strategy can accurately establish the corresponding relationship between the foveal region and the region of interest. And the hybrid variable-resolution pattern design method can solve the mass mutation problem between the foveal region and the edge region of the fixed foveal patterns with variable resolution to some extent.

2. Theoretical Development

In the existing spatial variable-resolution pattern design method, the location of the region of interest is known, but in most actual applications, the type and location of the target are unknown. In order to improve the universality of spatial variable-resolution structure, we propose a two-step projection strategy; its schematic diagram is shown in Figure 1. It is designed to solve the problem that the correspondence between the foveal region and the region of interest can only be set manually.

Step I: Using random patterns with uniform resolution to undersample the target, the fuzzy preliminary reconstructed image can be obtained. The sampling rate can be between 0.01 and 0.1. The parameters of the foveal region, namely the central coordinates and radius of the foveal region (circular region), are determined according to the preliminary reconstructed image.

Step II: According to the foveal region parameters obtained in the first step, the sequence variable-resolution patterns is designed. The specific design method is shown in Figure 1. The designed sequence variable-resolution patterns are projected onto the target object to obtain the final reconstructed image.

The kernel of the two-step method is to determine the parameters of the foveal region of variable-resolution patterns used in the second step based on the preliminary reconstructed image of Step I. In essence, it is an image segmentation, which separates the suspected target from the background and then determines the parameters of the foveal region according to the location and size of the suspected target. The following will describe in detail the realization process of determining the parameters of the foveal region based on the preliminary reconstructed image.

2.1. Image Binarization

According to the preliminary reconstructed image in Figure 1, it can be seen that there is a significant gray value gap between the suspected target and the background, so it can be segmented according to the gray value. Commonly used image segmentation methods include fixed threshold segmentation [25], maximum inter-class variance segmentation [26] and cluster segmentation [27]. Among them, the fixed threshold segmentation method is simple and fast, and the image can be binarized directly according to the pre-set threshold. The maximum inter-class variance segmentation method can automatically find the appropriate segmentation threshold according to the gray distribution in the image, but when using this method, the whole image must be traversed first to obtain the number of pixels of each gray level, so its speed is slower than that of the fixed threshold method. The clustering segmentation method needs to be iterated continuously, and the speed is slow. Considering the requirement of image processing speed, this paper adopts the fixed threshold method to carry out image binarization for preliminary reconstructed images. The method of image binarization can be expressed as

$$M=\left\{\begin{array}{l}255\left(M\ge M_0\right)\\ 0\left(M<M_0\right)\end{array}\right.$$

(1)

where M represents the pixel value of a certain point in the preliminary reconstructed image, and M₀ represents the fixed threshold set. The fixed threshold used in this paper was 80.

2.2. Finding the Maximum Connected Domain

Although there may be some background noise in the initial reconstructed image after binarization, the suspected target is generally the largest connected part of the image. Therefore, the suspected target can be obtained by finding the maximum connected domain in the binary graph, and other non-maximum connected domains can be taken as the background, so as to determine the region of interest and establish the connection between the region of interest and the foveal region, as shown in Figure 2.

Figure 2a is the window that needs to be traversed. This window is traversed in the binary graph from left to right and from top to bottom. In the process of traversing, labels at a, b, c and d in the window were judged to determine the number at e. The specific calculation flow chart is shown in Figure 3. Figure 2b shows the binary image obtained from threshold binarization. Pixels were first labeled during the image traversal. As shown in Figure 2c, a label table was generated during the labeling process, which was used to record the pointing relationship of each label. As shown in Figure 2d, the region corresponding to the largest number of labels was the largest connected domain, as shown in Figure 2e.

2.3. Finding the Maximum External Matrix and External Circle

In the previous step, the maximum connected domain was calculated based on the preliminary reconstruction of binary images, but the maximum connected domain could not directly determine the parameters of the foveal region. Because the parameters of the foveal region were the central position and radius, it was a circular region. But the suspected target corresponding to the maximum connected domain is generally not a regular circular target. Therefore, it was also necessary to find the maximum external matrix of the maximum connected domain. Then, the maximum outer circle was obtained according to the maximum outer matrix, and the parameters of the foveal region were determined. Figure 4a is the binary image obtained by threshold segmentation. Figure 4b is the maximum connected domain solved. Figure 4c is the maximum external matrix solved, and Figure 4d is the maximum external circle solved. The maximum external matrix, the external circle of the maximum connected domain, and their related parameters can be directly obtained by the boundingRect() and minEnclosingCircle() functions in the OpenCV library, from which the central coordinates and radius of the suspected target can be obtained.

The location and size information of the suspected target can be obtained through the above three steps, which provides guidance for the second step of the two-step method.

The traditional spatial variable-resolution pattern design method is a series of fixed variable-resolution patterns in the foveal region. A mixed variable-resolution pattern is a model with random foveal locations added to the former pattern sequence. The purpose of this is to rapidly improve the imaging quality of the foveal region and take into account the imaging quality of the marginal region, and to solve the problem of quality mutation caused by the large difference in imaging quality between the two regions. The comparison between the sequence of hybrid variable-resolution pattern design and the traditional method is shown in Figure 5. In the traditional method, the central coordinates and radius of the foveal region are fixed, with a total of N patterns. In the mixed variable-resolution pattern sequence, one pattern with random center coordinates of the foveal region and the same radius as the fixed method was added every i (i = 1, 2, …, N − 1) amplitude in the fixed foveal variable-resolution pattern sequence. The size of i depended on the actual imaging needs for target and background image quality. If the imaging quality of the edge region was not high, the value of i could be larger. However, if the quality of the foveal region was required to improve rapidly, and the imaging quality of the edge region was required, the smaller the value of i, the better.

We assumed that the hybrid variable-resolution-pattern sequence image was $P_m(x,y)$, where $m=1,2,\cdot \cdot \cdot ,M$ represents the measurement, $(x,y)$ represents the total number of measurements, and C represents the image coordinates. The reflected light of the target object $I(x,y)$ after pattern modulation is received by the barrel detector, and a one-dimensional intensity signal $S_m$ is obtained according to the correlation between $P_m(x,y)$ and $I(x,y)$

$$S_m={\displaystyle \iint P_m(x,y)I(x,y)dxdy}$$

(2)

After M measurements, an estimate $I^{*}(x,y)$ of the target image can be obtained based on the second-order correlation of $P_m(x,y)$ and $S_m$

$$I^{*}(x,y)\approx <S_m{P}_m(x,y)>-<S_m><P_m(x,y)>$$

(3)

where $<·>$ represents the average of M measurements, and $I^{*}(x,y)$ will get closer and closer to the true value as the number of measurements increases.

3. Simulations and Experiments

The simulation and experiment in this paper are mainly divided into two parts. On the one hand, the threshold segmentation part was used to verify that the location and size of the target can be obtained without prior knowledge to guide the design of the variable-resolution speckle. On the other hand, a random foveal variable-resolution speckle was added to reduce the problem of mass mutation in the foveal region and the marginal region.

3.1. Simulations

Before introducing GI simulation experiments using spatial variable-resolution patterns, we clarified some settings. The number of pixels for both patterns and reconstructed images was set to 128 × 128. The number of patterns was set to 1000, 2000 and 3000, and the corresponding sample rates were set to 6%, 12%, and 18%. To be imaged, select “cameraman” were used, where the interested target photographer was not in the center of the field of view. The fixed foveal region was designed by using the method of determining the foveal region parameters in the two-step projection strategy. The comparison of the pattern design methods uses uniform resolution random patterns, spatial variable-resolution patterns and spatial–mixed variable-resolution patterns. GI using these patterns is also called UGI, SVGI and spatial–mixed variable-resolution GI (SMVGI). SMVGI selects different i values, which were set to 2 and 3, respectively, and the segmentation threshold M₀ of the preliminary reconstructed image was set to 100.

The imaging results of the above three different types of patterns are shown in Figure 6. The foveal region of the variable-resolution patterns was designed according to the preliminary reconstructed image obtained from 100 random pattern samples with uniform resolution. Moreover, the uniform resolution patterns of the 100 samples and the subsequent variable-resolution patterns were reconstructed together with the measured values and calculated into the total sampling times. It can be found from the figure that with the increase in sampling times, the images of the four different methods became gradually more clear. Among them, the foveal region of SVGI had the best quality, but it also had the most obvious mass mutation with the background. The reason is that the high-resolution region of SVGI patterns is fixed at the target, the relative resolution of the background is low, and the quality difference between the two is obvious. The two results of SMVGI, although inferior to SVGI in the foveal region, were superior to UGI. The difference in foveal region quality between the two and other regions was not very significant because most of the variable-resolution-pattern foveal regions used by the two methods were set in the region of interest, but not all, so it was better than UGI but not as good as SVGI. Meanwhile, the SMVGI (i = 2) and SMVGI (i = 3) methods inserted a random-foveal-region variable-resolution speck, while improving the quality of the foveal region but also taking into account the quality of the edge region, so there was no obvious quality difference between the two regions.

To further compare these four methods, PSNR was used for quantitative analysis of the imaging results in Figure 6. Firstly, a rectangular region in the foveal region was selected for comparison. The PSNR of the foveal region in the reconstructed image obtained by four methods is shown in Figure 7. The PSNR of the results obtained by the four methods gradually increased with the increase in sampling times, and the qualities in order from high to low were SVGI, SMVGI (i = 3), SMVGI (i = 2) and UGI. The first three were all variable-resolution patterns, which can rapidly improve the imaging quality of the foveal region, so they were better than UGI using uniform resolution patterns. However, the reason why SVGI was higher than SMVGI is that part of SMVGI’s variable-resolution-pattern foveal region was randomly distributed, thus weakening the effect of improving the quality of the set foveal region. The reason why SMVGI (i = 3) was higher than SMVGI (i = 2) is also that the patterns in the random foveal region were higher than those in the former. Then, a matrix region of the non-foveal region was selected to compare the PSNR of the edge region. The PSNR of the edge region of the reconstructed image obtained by the four methods is shown in Figure 8. It can be seen that the results of the PSNR of the edge region and the foveal region are quite different. PSNR was the lowest when the UGI was less than 2000 times, and it was better than SVGI when the UGI was more than 3000 times. The SVGI with higher PSNR in the foveal region was inferior to the two SMVGI methods in the marginal region, because the variable-resolution patterns in the fixed foveal region improved the imaging quality of the foveal region at the expense of the information in the marginal region, so the information in the non-foveal region was less. The PSNR of SMVGI (i = 2) was higher than that of SMVGI (i = 3) because there were more random variable-resolution patterns in the foveal region, so there was more information in the edge region.

Taking the results of the 3000 samples in Figure 7 and Figure 8 as an example, the mass abrupt change between the foveal region and marginal region of the three different variable-resolution methods was analyzed. The difference in PSNR between the foveal region and marginal region using the SVGI method was 1.3 dB, that of SMVGI (i = 2) was 0.3 dB, and that of SMVGI (i = 3) was 0.6 dB. It was found that the hybrid variable-resolution method can alleviate the mutation of the foveal region and the edge region of imaging quality due to the change in pattern resolution to a certain extent. In the hybrid variable-resolution pattern design method, the smaller the value of i, the less patterns in the fixed foveal region, which can improve the imaging quality of the set foveal region while taking into account the imaging quality of the edge region.

In order to further compare SMVGI and SVGI, the target was replaced with a bird, and the variable-resolution speculations used in the previous set of experiments were still used. The comparison methods were UGI, SVGI, and SMVGI. The sampling times were set to 3000, and the comparison results are shown in Figure 9. Comparing the reconstructed images of SVGI and SMVGI, it can be clearly seen that the feathers on the bird’s head are clearer in the reconstructed images of SMVGI. We used PSNR to compare the quality of different regions of the reconstructed image, as shown in

Table 1. PSNR(dB) of mixed variable-resolution patterns and UGI and SVGI imaging results.

	UGI	SVGI	SMVGI
Foveal region	27.3	33.4	31.2
Marginal area	28.9	26.6	27.9
Full image	29.2	28.8	30.9

. For the full image, the SMVGI method was superior to the other two methods. The imaging quality of the foveal region of SVGI was the best, but the imaging quality of the marginal region was worse than that of UGI. The SMVGI method retained the characteristics of SVGI in the foveal region, and the image quality of the marginal region was also improved. To a certain extent, the problem of mass mutation between the foveal region and the marginal region of the fixed variable-resolution patterns was solved.

3.2. Experiments

This section verifies the proposed spatial hybrid variable-resolution pattern design method and the two-step projection strategy for spatial variable-resolution patterns. The schematic diagram of the experimental device is shown in Figure 10. Laser was used to send out beams of light. The computer loaded pre-designed patterns onto the DMD to modulate the beam. The experimental objective was to select a kitten whose position was uncertain in the scene, and the imaging resolution was set to 128 × 128. The reflected light from the target object was collected using a single-pixel detector and recorded in a data acquisition card. The computer then correlated the light intensity information from the data acquisition card with the patterns to create a reconstructed image of the target object. The specific models and parameters of the equipment used in the experiment are shown in

Table 2. GI experimental device.

Name	Model	Parameter
Laser	LED-20W	20 W LED white cold light source
DMD	Discovery 4100	Resolution 1024 × 768 Refresh rate (1 bit) 22 kHz
Projecting Lens	GCL-010143	F120 mm
Convergent Lens	GCL-010172	F50 mm
Data Acquisition Card	Pico6404E	Bandwidth 500 MHz Band 350–1100 nm
Single-pixel detector	Thorlabs PDA36A	Bandwidth 10 MHz Active area 13 mm2
MATLAB	R2019a	PicoScope 6000 Series MATLAB Generic Instrument Driver PicoScope Support Toolbox

. Firstly, the first step in the two-step method was used to determine the area of interest in the scene, that is, the area where the kitten was located.

The preliminary imaging results obtained by projecting 200 random patterns of uniform resolution are shown in Figure 11a. The segmentation threshold M₀ was 120. The maximum connected domain was obtained by threshold segmentation, as shown in Figure 11b. The region of interest was obtained by solving the maximum peripheral circle, and variable-resolution patterns were designed. Then, the variable-resolution patterns of the fixed foveal region and mixed foveal region designed by the second projection in the two-step method were carried out. The sampling times were set to 500, 1000, 1500 and 2000, including 200 sampling times of the first step. The values of i of the mixed variable-resolution patterns were 2 and 3, respectively. The experimental results obtained are shown in Figure 11. With the increase in sampling times, the imaging results corresponding to the three methods also became clearer. It was more obvious that the image quality of SVGI in the foveal region was better than that of the two mixed variable-resolution pattern methods. For example, when the sampling times were 2000, the cat’s eyes were clearly distinguishable in the SVGI results, while the results of the two SMVGI methods were still fuzzy.

In order to further analyze the experimental results in Figure 11, PSNR was used to conduct the quantitative analysis of the foveal region, and the analysis results are shown in Figure 12. The PSNRs in descending order were SVGI, SMVGI (i = 3) and SMVGI (i = 2). Due to the two-step method, the foveal region basically covers the region of interest, so fixing the SVGI of the foveal region can quickly improve the imaging quality of the foveal region. There were random variable-resolution patterns in the foveal region in the mixed variable-resolution patterns, which increased the information of the edge region, but at the same time, there was less information in the foveal region than SVGI. Therefore, the two methods of SMVGI have lower quality in the foveal region than SVGI. The SMVGI of i = 3 was higher than the SMVGI of i = 2 because the SMVGI of i = 2 contained more random foveal variable-resolution patterns, so the quality of the SMVGI in the foveal region was lower than that of i = 3.

In order to further compare SMVGI and SVGI, the target was replaced with a resolution plate, and the variable-resolution speculations used in the previous set of experiments were still used. The comparison methods were UGI, SVGI, SMVGI (i = 2) and SMVGI (i = 3). The sampling times were set to 3000, and the comparison results are shown in Figure 13. The foveal region and marginal region of SVGI showed obvious mass mutation, and the leftmost column of numbers became fuzzy and difficult to distinguish. The SMVGI methods with mixed variable-resolution patterns did not show significant mass mutation, but there was also a quality gap between the marginal region and the foveal region. For quantitative analysis, two sets of line pairs in the foveal region and some numbers in the lower left corner were selected as the foveal region and marginal region for comparison. The PSNR results of quantitative comparison are shown in

Table 3. PSNR(dB) of mixed variable-resolution patterns and UGI and SVGI experimental results.

	UGI	SVGI	SMVGI (i = 2)	SMVGI (i = 3)
Foveal region	18.5	19.6	19.2	19.5
Marginal area	18.8	17.8	18.5	18.4
ΔPSNR	-0.3	1.8	0.7	1.1

. In the foveal region, SVGI had the highest PSNR, followed by SMVGI (i = 3), and the three methods using variable-resolution patterns were superior to UGI in the foveal region. However, in the peripheral region, SVGI had the lowest PSNR and the highest UGI. The PSNRs of SMVGI (i = 3) and SMVGI (i = 2) were similar, and the PSNR of SMVGI (i = 2) was slightly higher than that of SMVGI (i = 3). ΔPSNR represents the PSNR difference between the foveal region and the marginal region. SVGI had the largest difference, followed by SMVGI (i = 3), and the ΔPSNR of SMVGI (i = 2) was lower than that of SVGI and SMVGI (i = 3). It can be clearly seen that the hybrid variable-resolution pattern design method sacrifices part of the foveal region information to improve the imaging quality of the edge region. To a certain extent, the problem of mass mutation between the foveal region and the marginal region of fixed variable-resolution patterns is solved.

4. Summary and Future Works

In view of the problem that the correspondence between the foveal region and the region of interest of variable-resolution patterns cannot be obtained automatically, and the mass mutation phenomenon exists between the foveal region and the marginal region, in this paper, a hybrid variable-resolution speckle design method was proposed and the corresponding two-step projection strategy was designed. The mixed variable-resolution speckle is different from the common fixed foveal region variable-resolution speckle, and the random foveal region variable-resolution speckle is added regularly in its projection sequence. The two-step projection strategy involves obtaining the central coordinates and radius of the foveal region by using the initial reconstructed image obtained from random speckle undersampling with uniform resolution in the first step, so as to guide the design of the variable-resolution speckle in fixed and random foveal region.

The proposed hybrid variable-resolution speckle and two-step projection strategy were simulated and verified by experiments. The simulation and experimental results showed that the two-step projection strategy can automatically establish the relationship between the foveal region and the region of interest, and can accurately obtain the position and size of the interest object in the field of view. Compared with the fixed foveal region, the quality of the mixed variable-resolution speckle in the foveal region is lower than the latter, but the quality of the edge region is better, which solves the problem of mass mutation in the foveal region and the edge region caused by the fixed foveal region of the variable-resolution speckle to a certain extent. However, the method proposed in this paper also has some limitations. When there are multiple targets in the experimental scene, it may be necessary to perform multiple threshold segmentation. In the future, we will try to solve the threshold segmentation problem when there are multiple targets in the scene, and improve the processing speed of multiple targets.

The method proposed in this paper aims to improve the imaging quality of ghost imaging technology, to further narrow the gap between ghost imaging and traditional imaging performance, and to promote the practical application of ghost imaging. This method is expected to be applied in the field of 3D imaging and medical imaging, and is expected to solve the problem that the image quality gap between the target object and the background is too large in complex scenes. In the future, we will try to use deep learning methods to further mine prior information to design a variable-resolution speckle more suitable for the target object and further improve the imaging quality.