Fast Extended Depth-of-Field Reconstruction for Complex Holograms Using Block Partitioned Entropy Minimization

Abstract

Optical scanning holography (OSH) is a powerful and effective method for capturing the complex hologram of a three-dimensional (3-D) scene. Such captured complex hologram is called optical scanned hologram. However, reconstructing a focused image from an optical scanned hologram is a difficult issue, as OSH technique can be applied to acquire holograms of wide-view and complicated object scenes. Solutions developed to date are mostly computationally intensive, and in so far only reconstruction of simple object scenes have been demonstrated. In this paper we report a low complexity method for reconstructing a focused image from an optical scanned hologram that is representing a 3-D object scene. Briefly, a complex hologram is back-propagated onto regular spaced images along the axial direction, and from which a crude, blocky depth map of the object scene is computed according to non-overlapping block partitioned entropy minimization. Subsequently, the depth map is low-pass filtered to decrease the blocky distribution, and employed to reconstruct a single focused image of the object scene for extended depth of field. The method proposed here can be applied to any complex holograms such as those obtained from standard phase-shifting holography.

1. Background

Optical scanning holography (OSH) [1,2] is one of the most effective techniques for capturing a complex hologram of a physical scene. It is different from existing methods such as phase shifting holography (PSH) [3], parallel phase shifting holography (PPSH) [4], geometric phase shifting digital holography (GPSDH) [5], Fresnel incoherent correlation holography (FINCH) [6,7], Fourier incoherent single channel holography (FISCH) [8], and the consumers scanner approach [9] that requires 2-D digital recording devices (e.g., digital camera) to capture the holographic signal, OSH only utilizes a single-pixel sensor. As such, OSH is a unique holographic recording technique and can even be configured to operate in both coherent and incoherent modes. Operating in the incoherent mode is important as the technique can be used to capture fluorescent specimens holographically. As mentioned in [2], OSH has many applications, such is but not limited to 3-D pattern recognition, 3-D microscopy, 3-D cryptography, and 3-D optical remote sensing. An OSH system can also be implemented to operate at high frame-rate for capturing hologram of a dynamic scene. In general, after a complex hologram is captured, it is often necessary to reconstruct a visible image from the hologram for further inspection and analysis. To reconstruct a hologram, we can back-propagate the hologram onto a sequence of regular spaced reconstruction planes that is parallel to the hologram. If a reconstruction plane is containing object points on the scene, those points will appear as a focused image, otherwise they will take the form of a de-focused haze. The operator or analyst will have to inspect each reconstruction plane, extract the focused image of the object points, if any, and discard the de-focused haze. Although this approach is feasible, the process is time-consuming and the quality of the reconstructed image will be affected by the visual judgment of the operator. For an OSH system that is employed to capture moving objects, it is desirable that the image reconstruction process should not be too lengthy. As such, the algorithms that are used for the reconstruction of optical scanning holograms should be automatic and computationally efficient. Automatic reconstruction of simple binary images from a hologram has been attempted utilizing Weiner filtering [10] and the Wigner distribution [11,12]. Lam and Zhang have proposed a hologram reconstruction method known as “blind sectional reconstruction” (BSR) [13,14]. The method can be divided into 2 stages. First, edge detection is applied to the image in each reconstruction plane. A reconstruction plane that exhibits a local minimum of edge points, as compared with the neighboring planes, will be taken as a focused section containing object points of the scene. This process is sometimes referred to as “automatic focus detection” (AFD). Second, a focused image of the object points that are contained in each of the focused section is obtained through an iterative optimization process, while the de-focused haze will be discarded. Subsequently, the focused images in all the sections can be merged to form an overall, focused image of the 3-D object scene with an extended depth of field. Despite the success of the BSR method, the process is computationally intensive, involving a large amount of memory in the optimization process, and so far, only reconstruction of simple binary images has been demonstrated. However, some enhancement of the method has been made to speed up the calculation and lower the memory requirement [15].

A fast variant of the BSR method was adopted in [16]. Similar to the blind sectional reconstruction, a subset of focused sections is obtained from the sequence of reconstructed planes through AFD. Next, simple edge analysis technique is applied to extract the focused object image (and to reject the de-focused haze) in each focused section. Albeit significant enhancement on the computation speed, the process is relying heavily on the edge analysis method and the manual setting of the parameter(s). The method has successfully demonstrated its capability in reconstructing binary objects from a hologram, but it is unlikely that it can be applied in a more complicated scene.

Recently, a more reliable AFD method, known as entropy minimization, was adopted in [17] for detecting the focused image planes. In this approach, a reconstruction plane is taken as a focused section if it exhibits a local minimum on its entropy value amongst the sequence of reconstructed images. This method is autonomous and does not require manual setting of parameters. Afterwards, a hologram reconstruction method was reported, based on the entropy minimization AFD [18]. For each pixel in each reconstruction plane, a small block with the pixel at the center is defined. The entropy of the pixel is then computed based on the pixels within its block. Next, the depth of each pixel is taken to be the depth of the reconstruction plane that has a minimum entropy value for the pixel of interest. As such, a depth map of the scene is deduced, with which the scene image can be reconstructed. Despite the effectiveness of the method, the amount of computation is overwhelming as the entropy of all the pixels have to be determined for all the reconstruction planes. A faster method has been proposed in [19], whereby the entropy is evaluated for each constituting objects in the scene instead of individual pixels. Briefly, a hologram is back-propagated to a virtual diffraction plane (VDP) that is close to the object space. Segmentation is applied on the VDP to separate the hologram into sub-holograms [20] each representing the fringe patterns of an isolated object in the scene. AFD is then applied to each sub-hologram to locate the focused image plane, and reconstruct the image of the object. However, this method is only applicable if the objects on the x-y plane in the scene are spaced far enough to be separated on the VDP, a requirement which is not guaranteed in practice.

Another fast method for hologram reconstruction has been proposed in [21]. The method is based on the assumption that the imaginary component (or phase angle) of a pixel is close to zero on its focused image plane. On this basis, a focused image of the object scene can be built by selecting, from each reconstruction plane, pixels with small imaginary values.

In this paper we present, partly based on the work in [18], a method for fast hologram reconstruction with extended depth of field (EDF). By EDF, it means that a focused image of all the objects (which may differ in depth and geometrical shape) in the scene can be recovered simultaneously from the hologram. Our proposed method is particular suitable for optical scanned holograms that are unrestricted in size, complexity of object scene, and capturing modes (i.e., coherent and incoherent modes). Briefly the method can be divided into 4 stages. First, the hologram is back-propagated into a sequence of parallel, and uniformly spaced reconstruction planes. Second, each reconstruction plane is evenly partitioned into non-overlapping square image blocks. Third, entropy minimization AFD is employed to locate the focused section of each image block, from which the depth is determined, and further refined with low-pass filtering. Fourth, the collection of depth information from all the image blocks is utilized to reconstruct a focused image of the object scene. In our proposed method, we have assumed that in a typical scene, the depth of pixels within close neighborhood should be similar, an assumption that is generally applicable in practice. As such, the depth map of the scene image can be evaluated on a block-by-block, instead of a pixel-by-pixel basis, resulting in significant increase in the computation efficiency as compared with the method in [18].

2. Optical Scanning Holography

The system of optical scanning holography that we have employed for hologram acquisition is shown in Figure 1. A linearly polarized laser of wavelength λ is passed through a half-wave plate (HWP), which rotates the polarization direction of the laser beam along the bisector of the two principal axes of the electro-optic crystal of the electro-optic modulator (EOM). Since there is birefringence induced along one of the principal axes through external electrical saw-tooth signal, the output beam of the EOM contains two orthogonal polarizations with a frequency difference Ω. The beam of light with orthogonal polarization is split into two beams with polarizing beamsplitter PBS1.

One beam, expanded by beam expander BE1, is reflected to beamsplitter BS2 by mirror M1. The other beam passes through another HWP, and is reflected by mirror M2 to BS2 through beam expander BE2 and lens L1. Subsequently the pair of beams goes through lens L2, and impinges on the test sample after passing beamsplitter BS3. Note that one beam projecting on the sample is a plane wave (dotted lines) while the other beam is a spherical wave (solid lines). Hence on the sample, we have a time-dependent Fresenl zone plate (TD-FZP) as the interference of a plane wave and a spherical wave gives an FZP and the frequency difference Ω of the two beams gives running fringes within the overall scanning beam on the sample [1,2]. A x-y table is utilized to move the sample along a zigzag scan path. At each position of the scan path, the optical waves scattered by the sample is directed to photodetector PD1 through beamsplitter BS3 and lens L3. The output of PD1 is band-pass filtered at Ω to give the Signal output. At the same time, the pair of beams is also combined in beamsplitter BS2, impinging on photodetector PD2. The output of PD2 is band-pass filtered at Ω and taken to be the Reference for the lock-in amplifier. The lock-in amplifier mixes the Signal and the Reference to form the hologram which is composing of a sine hologram, $H_{\sin }\left(x,y\right)$, and a cosine hologram, $H_{\cos }\left(x,y\right)$ [1,2]. Mathematically, suppose the 3-D object is divided into N uniformly separated image planes that are parallel to the hologram, and the $j^{th}$ image plane that is located at an axial distance $z_j$ to the hologram is denoted by $I_0\left(x,y;z_j\right)$, the hologram acquired with OSH is given by

$$H\left(x,y\right)=H_{\cos }\left(x,y\right)+i{H}_{\sin }\left(x,y\right)={\displaystyle \sum _{j=0}^{N-1}{H}_k\left(x,y;z_j\right)}$$

(1)

where $H_k\left(x,y;z_j\right)=I_0\left(x,y;z_j\right)\otimes F\left(x,y;z_j\right)$ with ⊗ denoting the operation of convolution, and

$$F\left(x,y;z_j\right)=\frac{1}{i\lambda z_j^{}}\exp \left[\frac{i2\pi }{\lambda }\sqrt{{\left(x\delta \right)}^2+{\left(y\delta \right)}^2+z_j^2}\right]$$

(2)

is the spatial impulse response of propagation of light [1], where $\delta$ is the pixel size of the hologram.

3. Proposed Method

The objective of our proposed method is to obtain an extended field of depth (EFD) image (i.e., an image with all the object points in focused, disregard of their geometry and distance from hologram $H\left(x,y\right)$). The process can be divided into 4 stages as shown in Figure 2, and outlined as follows. In stage 1, the hologram $H\left(x,y\right)$ is back-propagated to a stack of regular spaced reconstruction planes along the axial direction, a process that is referred to as “sectional reconstruction”. The image on each reconstruction plane is evenly partitioned into non-overlapping square blocks. In stage 2, the entropies of the image blocks in all the reconstruction planes are evaluated. The depth of each block is taken to be the distance of the reconstruction plane that exhibits the minimum entropy for that particular block position. It can be envisaged that the integration of the depth of all the constituting blocks (each contributed by one of the reconstructed images) will form a blocky depth map. Next, in stage 3, low-pass filtering is applied to smoothen the depth map. Finally, in stage 4, an extended depth of field image of the object scene is obtained by selecting each pixel from the reconstruction image plane that corresponding to the depth of the pixel in the depth map. Details of each stage are described as follows.

Stage 1: Sectional reconstruction

In this stage, a stack $S$ of evenly spaced images is reconstructed from hologram $H\left(x,y\right)$ through back-propagation. Let $S_j\left(x,y\right)|{}_{0\le j<N}$ and $Z$ denotes the magnitude of the reconstructed image on the j-th reconstruction plane section, and the maximum range of the depth of the object scene, respectively, we have

$$S_j\left(x,y\right)=\left|H\left(x,y\right)\otimes F^{*}\left(x,y;z_j\right)\right|$$

(3)

where $F^{*}\left(x,y;z_j\right)$ is the conjugate of $F\left(x,y;z_j\right)$. The separation between adjacent reconstruction planes is given by

$$\Delta z=z_{j+1}-z_j=\frac{Z}{N}$$

(4)

Subsequently, each reconstructed image $S_j\left(x,y\right)$ is partitioned into a 2-D array of non-overlapping square image blocks of size $M\times M$, as shown in Figure 3. Without loss of generality, we assume that both the hologram and its reconstructed images are square in size, and having the same size $K\times K$, where $K$ is an integer multiple of $M$. As such, there are $K/M$ blocks along the horizontal direction, and $K/M$ blocks along the vertical direction. Each image block in $S_j\left(x,y\right)$ is represented by the block function $b_j\left(p,q\right)|{}_{0\le p;q<\left(K/M\right)}$, which includes the pixels bounded by the square within the region $pM\le x<\left(p+1\right)M$ and $qM\le y<\left(q+1\right)M$. The EDF image $I^{\prime }\left(x,y\right)$ to be determined is also partitioned in a similar way with each block of pixels denoted by $b_{EFD}\left(p,q\right)$.

Stage 2: Non-overlapping block partitioned entropy minimization

In this stage, the entropy of all the image blocks that have been partitioned in stage 1 is computed as

$$E_j\left(p,q\right)=-P_j\left(p,q\right)\log P_j\left(p,q\right)$$

(5)

where

$$P_j\left(p,q\right)={\displaystyle \sum _{x=pM}^{\left(p+1\right)M-1}{\displaystyle \sum _{y=qM}^{\left(q+1\right)M-1}{S}_j\left(x,y\right)}}$$

(6)

An image that is in focus generally exhibits minimum entropy. As such, the depth (or focused plane) of each block $D\left(p,q\right)$ can be taken to be the position of the plane that exhibits minimum entropy of the block, i.e.,

$$E_{D\left(p,q\right)}=MIN{\lfloor E_j\left(p,q\right)\rfloor }_{0\le j<N}$$

(7)

where the expression on the right-hand-side of Equation (7) denotes the minimum value of $E_j\left(p,q\right)$ for $0\le j<N$. Collection of depth of each block results in a crude, blocky depth map of the object scene. Intuitively, we can obtain an EDF image $I\left(x,y\right)$ by selecting each block of pixels from the corresponding block with minimum entropy from the stack of reconstructed images. Mathematically, this can be expressed as

$$b_{EDF}\left(p,q\right)=b_k\left(p,q\right)$$

(8)

and

$$I\left(x,y\right)={\displaystyle \underset{p;q=0}{\overset{p;q=\left(K/M-1\right)}{\cup }}{b}_{EDF}\left(p,q\right)}$$

(9)

where $k=D\left(p,q\right)$, and the block $b_{EDF}\left(p,q\right)$ that is selected is referred to as an extended depth of field block. The symbol $\cup$ denotes the union operator that is used to compose the EDF image by patching it with the extended depth of field blocks (i.e., $b_{EDF}\left(p,q\right)$). However, as we shall show later, an EDF image reconstructed in this manner will exhibit a blocky, visual unpleasant appearance.

Stage 3: Filtering

To reduce the blocky appearance of the EDF image, we propose to smooth the depth map with a simple low-pass filtering process that can be realized as follows. First, we compute the mean intensity for each EDF block as

$$mean\left(p,q\right)=\frac{1}{9}{\displaystyle \sum _{s=-1}^1{\displaystyle \sum _{t=-1}^1\left[b_{EDF}\left(p+s,q+t\right)\right]}}$$

(10)

Next, a corresponding status flag $status\left(p,q\right)$ is determined as

$$status\left(p,q\right)=\{\begin{array}{cc}1& ifmean\left(p,q\right)\ge T\\ 0& otherwise\end{array}$$

(11)

where T is a small fixed threshold value corresponding to the minimum intensity that is visible to the human eye. An EDF block with an average intensity higher than T is referred to as a “visible block”.

Subsequently, a low-pass filtered depth map is obtained by averaging the depth values of visible blocks within a $3\times 3$ window as

$$D_L\left(p,q\right)=\frac{1}{9}{\displaystyle \sum _{s=-1}^1{\displaystyle \sum _{t=-1}^1\left[D\left(p+s,q+t\right)\right]}}\times status\left[m\left(p+s,q+t\right)\right]$$

(12)

Note that in the filtering process, blocks with too low intensity are discarded as they are likely associated to empty space in the scene, and may not carry reliable depth information.

Stage 4: Reconstruction of the EDF image

Finally, we apply Equation (9) to obtain the EDF image, with the variable $k=D_L\left(p,q\right)$, i.e.,

$$b_{EFD}^{\prime }\left(p,q\right)=b_k\left(p,q\right)|{}_{k=D_L\left(p,q\right)}$$

(13)

and

$$I^{\prime }\left(x,y\right)={\displaystyle \underset{p;q=0}{\overset{p;q=\left(K/M-1\right)}{\cup }}{b}_{EFD}^{\prime }\left(p,q\right)}$$

(14)

In comparison with the method in [18], our proposed scheme has a significantly decrease in the computational loading of the entropy values as they are evaluated on a block-by-block, instead of a pixel-by-pixel basis. With our approach, the computation time for deducing the entropy values has been reduced by $M^2$ times. The computation time for the reconstruction of the hologram into different planes are similar for both methods. As will be shown later, our proposed method is around 60 times faster than the method in [18].

4. Experimental Results

To evaluate the performance of our proposed method, 2 sets of holograms have been captured with the OSH system that has been configured to operate in the incoherent mode [22], and with the wavelength of the laser beam being 633 nm. The first hologram “A” is a pair of binary Chinese characters “光” and “電”, and second hologram “B” is a grey level image of a pair of partially overlapping coins. The pixel size of holograms (i.e., the sampling interval with which each hologram pixel is recorded) “A” and “B” are 5 μm and 15 μm, respectively. In both cases, the range of distance between the object space and the hologram plane is [0.015 m, 0.03 m], but there is no prior knowledge on the depth of individual objects in the scene. The sine and the cosine holograms of the 2 samples are shown in Figure 4a–d.

Through a series of trial and error tests conducted with visual judgement, we estimated that there are mainly 2 focused planes that can be identified for sample ‘A’ at 0.021 m and 0.024 m. For ‘B’, again there are mainly 2 focused planes at 0.0197 m and 0.027 m. The reconstructed images of sample ‘A’ at the 2 focused planes are shown in Figure 5a,b. It can be seen that in both figures when certain regions of the image are in focused, the remaining ones are blurred. Similar observation is noted for the reconstructed image of sample ‘B’ at the 2 focused planes in Figure 5c,d. In both cases, it is not possible to have an EDF image where all the contents are in focus.

Next, we apply stages 1 and 2 of our proposed method to obtain the depth map and the EDF image from the 2 holograms. The number of reconstruction planes and the block size are set to $N=40$ and $M=32$, respectively, to provide a sufficiently fine depth resolution of $\Delta z=\frac{Z}{N}=\frac{0.015}{40}=0.375\mathrm{mm}$, and a reliable measurement of the entropy of each image block. The depth maps of the 2 samples are shown in Figure 6a,b, and the corresponding EDF images obtained with the above settings are shown in Figure 6c,d. We observe that all the objects in the scene represented by each hologram are reconstructed as focused images but the appearance, especially for sample “B”, is rather blocky with obvious discontinuities along some of the object boundaries.

Subsequently, we have applied Equation (12) to filter the depth map, and obtained the reconstructed images with Equations (13) and (14). Suppose the dynamic range of the reconstructed image is normalized to the range $\left[0,1\right]$, the threshold T is set to 0.0625, as image intensity lower than this value is hardly visible, and likely to represent empty background or noise signals. The filtered depth maps are shown in Figure 7a,b, and the reconstructed EDF images are shown in Figure 7c,d. In the EDF images of both samples, we observe that the objects in the scenes are reconstructed as sharp focused images, and the blocky appearance has been reduced significantly as compared with those obtained with the unfiltered depth maps. The computation time based on the typical PC is around 5.9 s. The correlation score of the EDF images of holograms ‘A’ and ‘B’ (with reference with the ones obtained with manual judgement) are 0.977 and 0.845, respectively. The high correlation scores reflect that the EDF images obtained with our proposed method are close to those derived from manual judgement.

We would like to point out that out proposed method can function for quite a wide range of M. In general, the blocky appearance could be decreased with a smaller value of M (i.e., a small block size in computing the entropy values). However, this will jeopardize the accuracy of the depth-map, which may lead to blurriness in certain parts on the EDF image. As an example, we apply our proposed method to the 2 sample holograms (with filtering on the depth maps) with M = 20, and the EDF images are shown in Figure 8a,b. It can be seen that while the blocky appearance of EDF image corresponding to sample ‘B’ (Figure 8b) is reduced, certain parts of the 2 EDF images, especially in Figure 8a, are suffering from mild de-focusing. A proper choice of the range of M is dependent on the setup of the OSH system (such as the size of the hologram and the sampling interval). In our present setup, EDF images of favorable quality can be obtain with values of M in the range $\left[20,32\right]$.

Finally, we would like to compare our proposed method with the method in [18]. We have applied the method in [18] to reconstructed the EDF images of the 2 holograms, and the results are shown in Figure 9a,b. We observe that the visual quality of the 2 EDF images are generally favorable, but the EDF image of hologram ‘B’ is more blocky than the one obtained with the proposed method. The computation time based on the same PC is around 356 s, which is about 60 times longer than the proposed method. The correlation score of the EDF images of holograms ‘A’ and ‘B’ (with reference with the ones obtained with manual judgement) are 0.982 and 0.741, respectively, which are more or less similar to the proposed method. As for the visual quality, the EDF image of sample ‘B’ (e.g., around the characters ‘1’ and ‘2’ on the pair of coins) is better in our proposed method.

5. Conclusions

In this paper, we have proposed a method for reconstructing the scene image with extended depth of field (EDF) from a hologram that is captured with an OSH system. Our proposed method is based on a scheme that is referred to as “non-overlapping block-based entropy minimization” method, which is applied to determine the depth (focus) map of the scene image in an autonomous, block-by-block manner. From the depth value, each block is reconstructed as a focused image tile. Subsequently, the complete scene image is reconstructed with all the constituting objects appearing as sharp focused images. Further enhancement on the visual quality of the reconstructed image is attained by smoothing the depth map, so that the blocking effect at the object boundaries is reduced. Experimental results reveal that our proposed method is capable of reconstructing EDF images from OSH holograms representing both continuous tone and binary object scenes with favorable quality.

Further research on the proposed method could be conducted through applying different kinds of focus measurements in the auto-focus detection process. An extensive review on the topic has been reported in [23].