Three-Dimensional Integral Imaging with Enhanced Lateral and Longitudinal Resolutions Using Multiple Pickup Positions

Abstract

In this paper, we propose an enhancement of three-dimensional (3D) image visualization techniques by using different pickup plane reconstructions. In conventional 3D visualization techniques, synthetic aperture integral imaging (SAII) and volumetric computational reconstruction (VCR) can be utilized. However, due to the lack of image information and shifting pixels, it may be difficult to obtain better lateral and longitudinal resolutions of 3D images. Thus, we propose a new elemental image acquisition and computational reconstruction to improve both the lateral and longitudinal resolutions of 3D objects. To prove the feasibility of our proposed method, we present the performance metrics, such as mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and peak-to-sidelobe ratio (PSR). Therefore, our method can improve both the lateral and longitudinal resolutions of 3D objects more than the conventional technique.

1. Introduction

Three-dimensional (3D) image visualization is a significant issue for many applications, such as unmanned autonomous vehicles, media content, defense, and so on [1,2,3,4,5,6,7,8]. To visualize 3D images, various methods can be utilized, such as time of flight, stereoscopic imaging, holography, and integral imaging [1,9,10,11]. By utilizing incoherent light with a general camera, integral imaging [1,9,12,13,14] can be used. It is a passive multi-perspective imaging technique that obtains depth information from 2D images, which are called elemental images. Unlike stereoscopic imaging techniques, such as anaglyphs, shutter glasses, and film-patterned retarders, integral imaging can visualize 3D images in full color, full parallax, and with continuous viewing points without special viewing devices. However, since integral imaging utilizes a lenslet array, the resolution of each elemental image is limited by the number of lenses and the resolution of the sensor. Thus, integral imaging by the lenslet array may visualize the 3D image in low resolution with a shallow depth of focus.

To solve lateral and longitudinal resolutions of 3D images in integral imaging, synthetic aperture integral imaging (SAII) [1,9,13,14,15] was reported. The elemental image acquisition obtains different perspectives of elemental images by fixing the distances between scenes and the pickup plane. It can record elemental images with the same resolution as the image sensor. After obtaining high-resolution elemental images by SAII, volumetric computational reconstruction (VCR) [1,14,16,17,18,19] can be used for 3D image generation. It can reconstruct 3D image from elemental images by shifting the pixels. It shifts each elemental image and superposes them to obtain 3D information. However, 3D images may be visualized with limited resolutions because of the fixed distance between scenes and the pickup plane.

To solve this problem, elemental images captured at different pickup positions at in-depth directions are required. Thus, in this paper, we merged SAII with axially distributed sensing (ADS) [2,20,21,22]. ADS is another 3D image visualization method that records different perspectives of elemental images by moving the image sensor along the optical axis. This feature makes ADS have fewer resolution limitations caused by the fixed distance. In the reconstruction sequence, unlike SAII, ADS utilizes the relative magnification ratio for each elemental image from different distances. However, ADS may not generate the 3D images of the center part because of a lack of perspective information. In this paper, our method uses the advantages of both SAII and ADS. In the image acquisition, we obtain elemental images by moving the sensor on the pickup plane with different positions of in-depth direction. The shifting pixel and relative magnification ratio are utilized for the 3D image reconstruction. Therefore, our methods can enhance the lateral and longitudinal resolutions of 3D images simultaneously.

This paper is organized as follows. In Section 2, we present the basic concept of integral imaging, SAII, ADS, and our proposed method. Then, in Section 3, we show the experimental results for supporting the feasibility of our proposed method with the performance metrics, such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), mean squared error (MSE), and peak-to-sidelobe ratio (PSR). Finally, we conclude with our summary in Section 4.

2. Three-Dimensional Integral Imaging with Multiple Pickup Positions

In this section, we present the basic concepts of integral imaging, synthetic aperture integral imaging, axially distributed sensing, and our method.

2.1. Integral Imaging

To visualize natural 3D scenes, integral imaging was reported by G. Lippmann in 1908 [23]. Integral imaging provides full color and full parallax 3D images with multi-perspective elemental images. Figure 1 illustrates an overview of integral imaging. As shown in Figure 1, integral imaging generates 3D images by two sequences: image acquisition and reconstruction. Image acquisition provides multi-perspective elemental images from different locations of the lenslets. In the reconstruction, there is an optical method (Figure 1b) and a computational method (Figure 2). In the optical method, the 3D image is displayed by backpropagating the elemental image through the same lenslet array. In the computational method, as shown in Figure 2, a nonuniform volumetric computational reconstruction (VCR) is utilized. It uses different shifting pixels for elemental images through various reconstruction depths and superposes them. It can be described as follows [16]

$$\Delta x_s=\frac{N_x\times p_x\times f}{s_x\times z_d},\Delta y_s=\frac{N_y\times p_y\times f}{s_y\times z_d}$$

(1)

$$\Delta x_k=\lfloor \left(k\times \Delta x_s\right)\rceil ,\mathrm{for}k=0,1,2,\cdots ,K-1$$

(2)

$$\Delta y_l=\lfloor \left(l\times \Delta y_s\right)\rceil ,\mathrm{for}l=0,1,2,\cdots ,L-1$$

(3)

$$I{(x,y,z_d)}_{VCR}=\frac{1}{O{(x,y,z_d)}_{VCR}}\sum _{k=0}^{K-1}\sum _{l=0}^{L-1}{E}_{kl}(x+\Delta x_k,y+\Delta y_l)$$

(4)

where $\Delta x_s,\Delta y_s$ are the actual shifting pixels in a real number, $N_x,N_y$ are the number of pixels for each elemental image in x and y directions, $p_x,p_y$ are the pitches for the elemental images, f is the focal length of the camera, $s_x,s_y$ are the sensor sizes of the camera, $z_d$ is the reconstruction depth, $\Delta x_k,\Delta y_l$ are the shifting pixels assigned to the kth row and lth column elemental images, $K,L$ are the number of row and column elemental images, $\lfloor ·\rceil$ is the round operator, $O{(x,y,z_d)}_{VCR}$ is the overlapping matrix for nonuniform VCR, $E_{kl}$ is the kth row, lth is the column elemental image, and $I{(x,y,z_d)}_{VCR}$ is the 3D image by the nonuniform VCR. However, since the resolutions of elemental images are limited by the number of lenses and the resolution of the sensor, the resolutions of 3D images may be degraded. To solve this problem, high-resolution elemental images are required in integral imaging.

2.2. Synthetic Aperture Integral Imaging

Synthetic aperture integral imaging (SAII) is an elemental image acquisition technique. Unlike lenslet array-based integral imaging, SAII utilizes a full sensor for capturing each elemental image. Therefore, 3D images by SAII can have high resolutions. Figure 3 illustrates the SAII. As shown in Figure 3, it fixes the distance between the 3D object and the pickup plane and generates the elemental images by moving the image sensor on the pickup plane with the same pitch. However, 3D objects by using elemental images with single distances may not be visualized well due to the limited resolutions. Figure 4 illustrates the resolution problems of 3D image visualizations. In Figure 4, $P_s$ is the pixel size, f is the focal length, $Z_n$ is the distance between the sensor and 3D scenes, L is the distance between the center of the sensor and pixel, ${\theta }_r,{\theta }_g$ are the angles of the red and green rays, and $d_n$ is the distance where we recognize a pixel as single information. It can be described as follows:

$${\theta }_r={tan}^{-1}(\frac{L-(\frac{P_s}{2})}{f}),{\theta }_g={tan}^{-1}(\frac{L+(\frac{P_s}{2})}{f})$$

(5)

$$d_n=(\tan \left({\theta }_g\right)-tan\left({\theta }_r\right))\sum _{n=1}^N{Z}_n$$

(6)

As shown in Figure 4 and Equations (5) and (6), the distance between the sensor and the scene (${\sum }_{n=1}^N\left(Z_n\right)$) is the main factor for calculating the distances ($d_n$). In short, as the distance between the 3D object and the pickup plane decrease, the acquisition area per pixel becomes smaller. Therefore, it can obtain more precise information.

2.3. Axially Distributed Sensing

Axially distributed sensing obtains different perspectives of the images by changing the distance between the pickup and reconstruction plane. Then, it can reconstruct the 3D image by using a relative magnification ratio for each elemental image. Figure 5 illustrates image acquisition in ADS. As shown in Figure 5, the image sensor is aligned on the optical axis with different depths. These depth differences lead to ADS having fewer resolution limitations, which are caused by the distance between the 3D object and the pickup plane. Elemental images by ADS contain different pickup areas per pixel. Therefore, in ADS reconstruction, a relative magnification ratio can be utilized. It can be described as follows [2,9,20,21]

$$M_k=\frac{z_k}{z_o}$$

(7)

$$O{(x,y,z_d)}_{ADS}=\sum _{k=0}^{K-1}\mathtt{1}\left(\frac{x}{M_k},\frac{y}{M_k}\right)$$

(8)

$$I{(x,y,z_d)}_{ADS}={\frac{1}{O(x,y,z_d)}}_{ADS}\sum _{k=0}^{K-1}{E}_k\left(\frac{x}{M_k},\frac{y}{M_k}\right)$$

(9)

where $z_o$ is the nearest distance between the pickup position and the reconstruction plane, $z_k$ is the distance between the kth pickup position and the reconstruction plane, $M_k$ is the relative magnification ratio between $z_k$ and $z_o$, $\mathtt{1}$ is the matrix, $O{(x,y,z_d)}_{ADS}$ is the overlapping matrix for ADS, $E_k$ is the elemental image at the kth pickup position, $I{(x,y,z_d)}_{ADS}$ is the 3D image by ADS. However, since the center parts of elemental images have no perspectives, ADS may not reconstruct the center part of a 3D scene.

2.4. Our Method

To visualize 3D images with enhanced lateral and longitudinal resolutions, we merged the SAII, ADS, and VCR techniques in this paper. In elemental image acquisition, we set the camera arrays at different pickup planes with different lateral perspectives. Figure 6 illustrates the alignment of the camera array in our method. As shown in Figure 6, elemental images with different perspectives are recorded per pickup plane. On each pickup plane, different pitches among cameras are considered. In Equation (1), the shifting pixels of each elemental image can be decided by the number of pixels ($N_x$), pitch ($p_x$), focal length (f), sensor size ($s_x$), and depth between the pickup and reconstruction plane ($z_d$). However, the number of pixels, focal lengths, and sensor sizes are constant values when the camera and lens are chosen. Thus, in this case, the shifting pixel can be described as follows

$$\Delta x_s\propto k\frac{p_x}{z_d}$$

(10)

k is the merged constant value of $N_x$, $s_x$, and f. In Equation (10), shifting pixels ($\Delta x_s$) are considered by pitch ($p_x$) and depth ($z_d$). Figure 7 illustrates the relationship between the shifting pixel, pitch, and depth. As shown in Figure 7, the number of shifting pixels gradually decreases as the depth increases. However, in Figure 7, the shifting pixel can be compensated by using a higher pitch. Therefore, a higher pitch is used for the longest pickup plane to compensate for the lack of shifting pixels. Then we reconstruct the elemental images by considering various shifting pixels and the relative magnification ratio. It is described as follows:

$$N_x^m=N_x^0\times \frac{z_m}{z_0},\mathrm{for}m=0,1,2,\cdots ,M-1$$

(11)

$$N_y^m=N_y^0\times \frac{z_m}{z_0},\mathrm{for}m=0,1,2,\cdots ,M-1$$

(12)

$$\Delta x_m^s=\frac{N_x^m\times f\times p^m}{s_x\times z_d^m},\mathrm{for}m=0,1,2,\cdots ,M-1$$

(13)

$$\Delta y_m^s=\frac{N_y^m\times f\times p^m}{s_y\times z_d^m},\mathrm{for}m=0,1,2,\cdots ,M-1$$

(14)

$$\Delta x_{km}^s=\lfloor (k\times x_m^s)\rceil ,\mathrm{for}k,m=0,1,2,\cdots ,K-1,M-1$$

(15)

$$\Delta y_{lm}^s=\lfloor (l\times y_m^s)\rceil ,\mathrm{for}l,m=0,1,2,\cdots ,L-1,M-1$$

(16)

$$E_{klm}=\beta \left(N_x^m,N_y^m\right),\mathrm{for}k,l,m=0,1,2,\cdots ,K-1,L-1,M-1$$

(17)

$$E_{klm}^p=\Theta \left(E_{klm},N_x^{M-1},N_y^{M-1})\right),\mathrm{for}k,l,m=0,1,2,\cdots ,K-1,L-1,M-1$$

(18)

$$O_{klm}^p=\Theta \left(\mathtt{1}\left(E_{klm}\right),N_x^{M-1},N_y^{M-1})\right),\mathrm{for}k,l,m=0,1,2,\cdots ,K-1,L-1,M-1$$

(19)

$$O(x,y,z)=\sum _{m=0}^{M-1}\sum _{k=0}^{K-1}\sum _{l=0}^{L-1}{O}_{klm}^p(x+\Delta x_{km}^s,y+\Delta y_{lm}^s,z_m)$$

(20)

$$I(x,y,z)=\frac{1}{O(x,y,z)}\sum _{m=0}^{M-1}\sum _{k=0}^{K-1}\sum _{l=0}^{L-1}{E}_{klm}^p(x+\Delta x_{km}^s,y+\Delta y_{lm}^s,z_m)$$

(21)

where $z_0$ is the nearest distance between the pickup and reconstruction plane, M is the number of pickup planes, and $z_m$ is the distance between the mth pickup and reconstruction plane. $N_x^0,N_y^0$ are the number of pixels for the elemental image located at the nearest distance between the pickup and the reconstruction plane, $N_x^m,N_y^m$ are the number of pixels for the elemental image located at the distance between the mth pickup and reconstruction plane, $\Delta x_m^s,\Delta y_m^s$ are the actual shifting pixels for the mth pickup plane, f is the focal length of the camera lens, $p^m$ is the pitch among cameras at the mth pickup plane, $z_d^m$ is the depth between the mth pickup and the reconstruction plane, $\Delta x_{km}^s,\Delta y_{lm}^s$ are the shifting pixels for the kth row and lth column elemental image on the mth pickup plane, $E_{klm}$ is the kth row, lth column, and mth pickup plane elemental image with $N_x^m\times N_y^m$ pixels, $\beta$ is the magnification function that magnifies the elemental image by using the bicubic interpolation method with $N_x^m\times N_y^m$ pixels, $\Theta$ is the zero padding function, $E_{klm}^p$ is the expanded elemental image for reconstruction by our method, $O_{klm}^p$ is the one matrix with zero padding, $O(x,y,z)$ is the overlapping matrix for the reconstruction by our method, and $I(x,y,z)$ is the reconstructed 3D image by our method.

Figure 8 illustrates the procedure of our computational reconstruction. In the procedure, we needed to calculate the relative magnification ratio for every elemental image and resize all elemental images considering this magnification ratio ($\frac{z_m}{z_0}$). Then, we calculated the shifting pixels ($\Delta x_{km}^s,\Delta y_{lm}^s$) and assigned them to every elemental image. For convenience, in the reconstruction sequence, we applied zero padding with ($N_x^{M-1},N_y^{M-1}$) sizes to every elemental image. Finally, 3D images with enhanced lateral and longitudinal resolutions can be reconstructed.

3. Results of the Simulation

3.1. The First Simulation Setup

To prove the feasibility of our method, we implemented a simulation using ‘Blender’. Figure 9 illustrates the simulation setup and 3D scenes, where 5(H) × 5(V) camera arrays for each (different) depth are used, the focal length is f = 50 mm, the pitches of the camera arrays are 6, 4, and 2 mm, to set the same pitch for each pickup plane (Figure 7), the sensors size is 36(H) × 36(V) mm, and 1000(H) × 1000(V) is the number of pixels used for recording each elemental image. The conventional method only used the elemental images at the furthest pickup plane to reconstruct the 3D images. We used a resolution chart [24] as the object to prove the feasibility of our method.

3.2. The First Simulation Result

Figure 10 shows the simulation results reconstructed at 150 mm. As shown in Figure 10, both conventional and proposed methods visualize 3D images correctly. However, our method has better visual quality than the conventional method. To show the feasibility of visual quality enhancement, we calculated the performance metrics, such as peak signal-to-noise ratio (PSNR), mean squared error (MSE), and structural similarity (SSIM) by using the cropped image in Figure 10.

Table 1. Performance metrics of the first simulation results.

Image	Method	MSE	PSNR	SSIM
Line	Proposed method	141.175	26.6167	0.912548
	Conventional	412.612	21.9755	0.735038
Diagonal line	Proposed method	70.2242	29.6659	0.934659
	Conventional	191.215	25.3156	0.822554
Circle	Proposed method	120.501	27.3209	0.928587
	Conventional	362.636	22.5361	0.775436

shows the performance metric results. As shown in Table 1, sample images reconstructed by our methods have lower MSE and higher SSIM and PSNR values than the conventional method. Moreover, when we reconstruct the 3D sample images via various depths, the proposed method has better depth resolution than the conventional method. Figure 11 shows the 3D sample images reconstructed via various depths. As shown in Figure 11, it is difficult to find the correct depth (150 mm) in the conventional method. In contrast, the correct depth can be found easily in our proposed method.

3.3. The Second Simulation Setup

However, the first simulation was unfair, because the proposed method used three times the elemental images than the conventional method. Thus, in the second simulation, we used fewer elemental images for the proposed method than the conventional method. Figure 12 shows the elemental images used in the second simulation. As shown in Figure 12, we extracted eight elemental images on every pickup plane.

3.4. Simulation Result

Figure 13 and

Table 2. Performance metrics of the second simulation results.

Image	Method	MSE	PSNR	SSIM
Line	Proposed method	150.162	26.3652	0.907574
	Conventional	412.612	21.9755	0.735038
Diagonal line	Proposed method	71.6068	29.5813	0.934551
	Conventional	191.215	25.3156	0.822554
Circle	Proposed method	128.099	27.0554	0.925271
	Conventional	362.636	22.5361	0.775436

show the result of the second simulation. As shown in Figure 13 and Table 2, our proposed method has better visual quality than the conventional method. Finally, we calculated the peak-to-sidelobe ratio (PSR) value to show the enhancement of the depth resolution as shown in Figure 14. Each plot used a different sample (in the exact same locations as Figure 13) from the original image. As shown in Figure 14, our method provides a higher peak on the correct depth compared to the conventional method.

4. Conclusions

In this paper, we propose a 3D visualization method with an enhanced 3D resolution by merging SAII, VCR, and ADS. It may improve the lateral and longitudinal resolutions of 3D images by acquiring elemental images from different depths. Considering PSNR, MSE, SSIM, and PSR values as performance metrics, we prove that our method has better results than the conventional method. We believe that our method can be applied to many industries, such as unmanned autonomous vehicles, media content, and defense. However, it has some drawbacks. First, it requires great effort to implement real optical experiments due to lens aberration and alignment problems. Thus, we need more research and accurate optical components to apply our method in real optical experiments. Second, its processing speed is slower than the conventional method because it requires more processing steps to use the relative magnification ratio. Therefore, we will investigate the solutions to these drawbacks in future work.