1. Introduction
In the field of three-dimensional (3D) display and processing, a parallax image array (PIA) stands out as an exceptionally efficient format, encapsulating full parallax information of a 3D object. 3D imaging technologies that can acquire 3D scenes include integral imaging [
1,
2,
3,
4,
5,
6], light field imaging [
7,
8,
9,
10,
11,
12], and lensless imaging [
13,
14,
15,
16,
17]. Among these imaging methods, both integral imaging and light field imaging are characterized by acquisition systems comprising arrays of individual imaging optical elements such as lenses and cameras [
18,
19,
20,
21,
22,
23,
24]. In contrast, diffraction grating imaging, one of the lensless 3D imaging methods, presents a unique approach [
13,
14]. Utilizing a single optical element, it has the inherent capability to generate parallax images (PIs), showcasing the efficiency and simplicity of its optics in comparison to systems relying on arrays of lenses and cameras. These differences in acquisition systems lead to distinctions in the reconstruction process of the 3D scene. In imaging systems featuring an array of optical elements, the allocation of space for a single parallax image (PI) within the PIA is dictated by the field of view of each optical element. Consequently, the predominant approach for reconstructing the three-dimensional scene in such configurations is the utilization of the back projection method. Contrastingly, in diffraction grating imaging, the generation of a PIA is via a solitary optical element, posing challenges in precisely defining the area occupied by an individual PI. Consequently, the reconstruction of a 3D scene in diffraction grating imaging adopts a distinctive approach, relying on the convolution operation between the PIA and a delta function array. This method allows for 3D scene reconstruction, taking into account the unique properties of diffraction grating imaging, which generates a PIA by a single optical element.
A PIA has a unique characteristic in which individual PIs within the PIA are formed periodically in the imaging space by dynamically responding to changes in the depth of the object. This periodicity becomes particularly pronounced when subjected to the convolution operation with a delta function array; accentuation occurs when the periods align, while a distinctive reduction is observed when the periods diverge. Therefore, leveraging this characteristic, the reconstruction of an image corresponding to a specific depth is made possible through the judicious application of the sifting property intrinsic to the convolution between periodic functions (SCPF). The spatial filtering method using SCPF can be universally applied to all 3D imaging systems wherein there exists a correspondence between the spatial period of the PIA and the depth of the object.
Various spatial filtering methods have been researched for enhancing the scaling, depth resolution, and image resolution of reconstructed images, using the principles of SCPF [
13,
14,
25]. Within the methods of applying SCPF, research on an advanced spatial filtering technique, namely the controllable extraction of multiple depths (CEMD), has emerged as a notable way for image reconstruction. The viability of this method was substantiated through experimentation, utilizing a PIA generated via computational integral imaging simulation, employing a lens array model rather than an actual optical system [
25]. Nevertheless, this study has limitations because it was conducted only through simulations with optimized optical conditions, excluding various optical factors such as aberration that can affect the resolution and spatial period of a PIA. Additionally, the PIA is not obtained through real optical systems, so verification of the realism and practical applicability of the results is required.
Note that just as a PIA is called an element image array (EIA) in integral imaging, in diffraction grating imaging it is called a diffraction image array (DIA), and an individual PI within the DIA are called diffraction images (DIs).
In this paper, we employ the CEMD to diffraction grating imaging, aiming to demonstrate its practical applicability within real-world optical systems. To verify that it can be implemented in diffraction grating imaging, DIs are generated and acquired through optical methods. To apply the CEMD to a diffraction grating imaging system, we conduct a theoretical analysis of geometric optical relationships. This includes considerations of the spatial coordinates of the DI, taking into account the wavelength of the light source and the grating period of the diffraction grating. Additionally, we examine the spatial period of the PIA, corresponding to the depth of the object. A wave optical analysis is performed to explain the imaging formation of the PIA generated by the diffraction grating. For this purpose, the relevant intensity impulse response, scaled object intensity, and intensity of the DIA are derived. We then derive a mathematical expression for the CEMD in diffraction grating imaging. We perform experiments to simultaneously extract controllable depth images corresponding to multiple depths in a single spatial filtering process and validate its applicability to diffraction grating imaging systems.
3. Controllable Spatial Filtering in Diffraction Grating Imaging
Now, let us delve into the proposed methodology for slice image reconstruction, employing controllable spatial filtering. To regulate the spatial filtering of the recorded diffraction images, we introduce multiple periodic -function arrays. These arrays can be generated with varying spatial periods, corresponding to the depth range of the object space intended for reconstruction. The explanation for the proposed method is as follows.
The CEMD facilitates the extraction of spatially periodic information relevant to the desired depth range of objects from a DIA, and the equation is given by
where
and
are the ranges of the depth for extracting spatially periodic information from the captured DIA and the given distance value, respectively. The later part of Equation (
7) can be given by
Equation (
8) is similar to Equation (
4) and the medium part of Equation (
6), but Equations (
4) and (
6) are varying functions depending on the
z- and
x-coordinate of an object, (
). However, Equation (
8) is just varied depending only on the
z-coordinate of an object. In the spatial filtering process, to extract depth information from a DIA, a single
-function array ideally corresponds to a very thin depth plane in the 3D object space. Equation (
7) and (
8)’s results indicate that depth information can be extracted from a DIA across continuous or partially continuous ranges. This can be represented by
Therefore, Equation (
9) demonstrates the mathematical expression of controllable spatial filtering in diffraction imaging.
Equation (
9) also means that the proposed method can go beyond reconstructing within the range of a single-depth resolution within the depth resolution of the system and can simultaneously extract continuous depth or selective depth images from the entire range of the system’s depth resolution in a single spatial filtering process. Additionally, we establish the depth resolution within the proposed method. The SCPF’s depth resolution refers to the minimum distance between two discernible points along the
z-axis within the 3D object space and is given by
4. Experiment and Results
To verify the feasibility of the proposed method, experiments are performed and explained based on the theoretical analysis described above. In the experiment, one Arabic numeral ‘3’ and three alphabet letters ‘D’, ‘S’, and ‘M’ are used as the test objects, as shown in
Figure 2. As illustrated in
Figure 2a, the experimental setup involves a deliberate arrangement of objects in sequential order, denoted as ‘3’, ‘D’, ‘S’, and ‘M’, in relation to their respective distances from the diffraction grating. Specifically, the distance of object ‘3’ from the diffraction grating is precisely set at 100 mm.
Figure 2b provides a diagonal view of the actual experimental configuration, providing a comprehensive view of the spatial arrangement during the performed experiments. In the actual experiment, a laser light source was used from the side of the object to illuminate the object located behind the diffraction grating. Additionally, in the experimental configuration, the distance between object ‘3’ and the camera is 500 mm. As shown in
Figure 2b, the height of all objects is kept uniform at 6mm. Additionally, for observation and analysis, the distance between each object is separated by 10 mm.
Figure 2c illustrates the DIA acquired through our diffraction grating imaging system, while
Figure 2d provides a detailed enlargement of the 0th-order DI extracted from the DIA presented in
Figure 2c. The captured DIA exhibits a resolution of 2007 × 2007 pixels and consists of 3 × 3 individual DIs. When considering the depth of the objects employed in the experiment, the diffraction image array reveals that the spatial distribution between the diffraction images in the two-dimensional plane undergoes variation corresponding to the depth of the objects. Furthermore, as evident from the DIA presented in
Figure 2c, it is observed that the 0th-order DI exhibits the highest brightness, attributed to the diffraction efficiency. Subsequently, the brightness diminishes with the increasing diffraction order. As shown in both
Figure 2b,d, it is evident that the letters ‘D’, ‘S’, and ‘M’ are partially obscured due to the placement of the letter in front of each respective character. In particular, in the letter ‘M’, almost half of the letter is obscured. The diffraction grating, featuring a spatial resolution of 500 lines/mm, is positioned at a distance of 400 mm from the camera. To generate a two-dimensional DIA, a pair of diffraction gratings are arranged in a configuration where they intersect at right angles. The objects are illuminated by using a diode laser with a wavelength (
) of 532 nm.
In the application of the CEMD to the diffraction grating imaging method, utilizing an optically acquired PIA, the initial step involved the calculation of the depth resolution based on the experimental setup. The depth resolution of the capturing system employed in the experiment is illustrated in
Figure 3. The depicted graphs are computed based on the foundation of Equation (
10).
Figure 3 portrays the horizontal axis denoting the distance from the diffraction grating to an object, while the vertical axis represents the spatial period corresponding to the depth of the object. In
Figure 3a, the graph illustrates the distance from the diffraction grating for the objects utilized in this experiment and the approximate spatial period corresponding to the depth of each object.
Figure 3b,c provide a detailed representation of the spatial periods corresponding to object distances for the specific objects ‘3’ and ‘D’, as well as objects ‘S’ and ‘M’, respectively. The graph depicting the spatial period corresponding to depth illustrates a linear trend, attributed to the linearity of Equations (
1) and (
2) employed in the calculation process.
In the subsequent analysis, we intend to illustrate the reconstructed images obtained through two distinct methods: the conventional approach and our proposed spatial filtering technique applied to the recorded DIA. By comparing the results from both methods, we aim to highlight the effectiveness and advantages of the proposed approach in improving the range of depth perception.
Figure 4 illustrates the outcome of depth extraction for a single depth, representing the spatial filtering result achieved through the conventional method. This method has been commonly employed in previous approaches for 3D image reconstruction.
Figure 4a demonstrates spatial filtering for a limited depth range, as defined by Equation (
8).
Figure 4b presents the outcome of spatial filtering for a singular depth. On the left side of
Figure 4b, the convolution outcome of the DIA and delta function array is depicted, while the right side showcases an enlarged view of the 0th-order region (the center part) of the convolution result.
Next, we want to present the outcomes of our proposed spatial filtering method designed for multiple depths. Our spatial filtering results corresponding to the required depth range from the DIA are illustrated in
Figure 5.
Figure 5a,c depict the depth regions where spatial filtering for multiple depths will be conducted, as per Equation (
9).
Figure 5b is the result of the spatial filtering process, which is the calculation result of a definite integral over the range −99.5 mm
−100.5 mm for the Arabic numeral of ‘3’ and −109.5 mm
−110.5 mm for alphabet letter ‘S’ in Equation (
9).
Figure 5d is also the result of the spatial filtering process, which is the definite integral over the range −119.5 mm
−120.5 mm for the alphabet letter ‘D’ and −129.5 mm
−130.5 mm for the alphabet letter ‘M’ in Equation (
9). To demonstrate the capability of selective depth extraction, we applied spatial filtering to extract depths corresponding to objects ‘3’ and ‘S’, even though the order of the objects is ‘3’, ‘D’, ‘S’, and ‘M’. Also, spatial filtering corresponding to the depth ranges of ‘D’ and ‘M’ was performed.
Figure 5b,d are the results of spatial filtering for multiple depths based on Equation (
9). These illustrate the results of convolving the DIA with an array of delta functions corresponding to multiple-depth ranges. On the left side of
Figure 5b,d, the result of convolving the DIA with the delta function array is depicted, while the right side showcases an enlarged view of the 0th-order region (the center part) of our spatial filtering result. As evident from the spatially filtered DI in
Figure 5, our method facilitates a three-dimensional reconstruction of the object space across multiple-depth regions, achieved by simultaneously and selectively controlling the depth range of spatial filtering.
Upon comparing
Figure 4b with
Figure 5b, it is apparent that the proposed CEMD method may introduce increased blur noise in the resultant images. This blurriness is influenced by both the proximity and number of neighboring objects. Specifically, when neighboring objects are closer to the target object’s depth and there are numerous neighboring objects, the blur noise tends to escalate. Consequently,
Figure 5b exhibits more blur noise due to spatial filtering being conducted at two distinct target object depths. Moreover, the spatial filtering was conducted using a three-by-three array of the DIs in the DIA, resulting in a limited number of parallax images being utilized. Given the convolution operation detailed in Equations (
8) and (
9), as the number of DIs increases, the intensity of the 0th-order region is accentuated in the spatially filtered output. Hence, augmenting the number of DIs can mitigate the blurring effect in the reconstructed image. Objects with adjacent depths can increase noise in the reconstructed image even if they have separate depth resolutions because the difference in the spatial periods they occupy within the DIA is small. Therefore, it is necessary to establish the minimum depth between objects that corresponds to an acceptable noise level and its relationship with the number of DIs. The depth resolution of diffraction grating imaging is proportional to the spatial resolution of the image-acquisition device, such as imaging systems based on repoint imaging. The depth resolution of systems based on lens arrays, such as integral imaging, has a high depth resolution in close proximity to the lens array but decreases rapidly with distance. In contrast, the depth resolution of diffraction grating imaging is constant regardless of the depth of the object because it is linear, as implied by Equation (
10). This characteristic of diffraction grating imaging has applications in depth-sensing cameras. In addition, the characteristic of having a high depth resolution can be applied to microscope imaging systems as far as the diffraction limit of the diffraction grating allows.