Enhancement of Three-Dimensional Computational Integral Imaging via Post-Processing with Visibility Coefficient Estimation

Abstract

This paper introduces an enhancement technique for three-dimensional computational integral imaging by utilizing a post-processing method. Despite the advantages of computational integral imaging systems, the image quality of the systems can suffer from scattering artifacts due to occluding objects during image reconstruction. The occluding objects in out-of-focus locations, especially, can offer scattering artifacts to other objects at in-focus locations. In this study, we propose a novel approach to remove scattering artifacts in reconstructed images from computational integral imaging reconstruction (CIIR). Unlike existing methods such as synthetic aperture integral imaging systems with pre-processing methods, our technique focuses on a post-processing method to remove scattering artifacts. Here, the scattering artifacts are analyzed using a dehazing model with spectral analysis. To enhance the quality of reconstructed images, we introduce a visibility model and an estimation method for a visibility coefficient, a crucial parameter of the dehazing model. Our experimental results from computer simulations indicate that the proposed method is superior to existing computational integral imaging reconstruction (CIIR) methods.

1. Introduction

Three-dimensional integral imaging is one of the auto-stereoscopic three-dimensional imaging techniques proposed by G. Lippmann in 1908 [1]. Integral imaging offers several advantages in that it captures three-dimensional objects using white light and provides full parallax and continuous viewing points. These merits enable integral imaging to be employed in various applications using real 3D images, such as medical imaging, 3D object detection, and automation systems. Thus, many studies on integral imaging have been actively conducted [2,3,4,5,6,7,8,9,10,11,12].

An integral imaging system consists of two processes: pickup and reconstruction. During the pickup process, different perspectives of a 3D object are captured as an elemental image array (EIA) using a camera array, a moving camera, or a lens array. This captured EIA is reconstructed into a 3D image using optical or computational methods. Optical reconstruction is complex and subject to various optical restrictions. To overcome these drawbacks, computational integral imaging reconstruction (CIIR) methods have been proposed [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38].

A CIIR technique reconstructs a 3D image of an object without the limitations of 3D optical display. Since each elemental image contains angular information about a 3D object, a CIIR method can provide potential solutions to address occlusion problems. However, scattering or hazing artifacts from partial occlusion significantly impact the quality of the reconstructed images; thus, many studies have been performed to enhance the image quality [30,31,32,33,34,35,36,37,38]. These studies typically employ synthetic aperture integral imaging (SAII) to acquire an elemental image array (EIA), subsequently aiming to eliminate scattering artifacts from each elemental image before the computational reconstruction. However, an SAII method often requires a complicated pickup process involving a camera array or a moving camera, making its application system expensive in terms of both cost and space.

In this paper, we propose a new method to reduce scattering artifacts in reconstructed images after computational reconstruction. Unlike a SAII method, where elemental images are acquired using an array of high-resolution cameras, our approach utilizes elemental images obtained through a lens array. These elemental images often contain only partial information about a 3D object and have low resolution, making them unsuitable for preprocessing. Consequently, in contrast to existing approaches, we remove scattering artifacts from the reconstructed images. The proposed method for removing scattering artifacts is based on the dehazing model via spectral analysis [38]. We also propose a new, efficient method for automatically estimating the visibility coefficient according to the signal analyses, since the visibility coefficient is the most important parameter affecting the image quality. To evaluate the proposed method, we carry out computer simulation experiments. The experimental results indicate that the proposed method improves the image quality compared with that of the CIIR images of existing methods.

2. Related Works

2.1. Computational Integral Imaging Reconstruction

The computational reconstruction of a 3D object is based on the inverse of the pickup process, which is a simulation version of geometric ray optics in the digital space. This method converts an elemental image array into a reconstructed image at any output plane. As shown in Figure 1, elemental images are back-projected through an array of virtual pinholes and inversely mapped onto the output plane. Here, the elemental images are magnified with a factor of M = z/g, where z is the distance between the virtual pinhole array and the reconstruction plane and g is the distance between the elemental image array and the virtual pinhole array. The distance g is also considered to be the focal length of the virtual pinhole. Then, each enlarged elemental image is overlapped on the reconstruction plane to create a reconstructed image. By adjusting the z value, the process repeatedly applies to the elemental image array, resulting in a sequence of images along the z axis.

2.2. Existing Model for Scattering Medium with Spectral Analysis and Integral Imaging

A scattering model is derived from the relationship between a captured image, a scattering medium, and the original image without the scattering medium. It can be employed to reduce the scattering artifacts in the captured image. The standard model used in dehazing methods is derived from the Bouguer–Lambert–Beer law [39,40], as follows:

$$g\left(x\right)=t\left(x\right)f\left(x\right)+\left(1-t\left(x\right)\right)A.$$

(1)

Here, g(x) represents the image captured by a camera, f(x) is the original image to be reconstructed, and the intensity of the ambient light is denoted by A. The medium transmission, denoted as t(x), is the weight function affecting the original image. It can be seen that determining the values of A and t(x) is important in order to reconstruct f(x) from g(x). However, this model makes it somewhat complicated to separate t(x)f(x), requiring a complicated dehazing algorithm [41]. Thus, a simpler model is introduced [38], which is written as:

$$g\left(x\right)=\alpha f\left(x\right)+\left(1-\alpha \right)t\left(x\right).$$

(2)

In Equation (2), g(x) is the image signal affected by the scattering medium, f(x) is the original signal unaffected by the scattering medium, and t(x) is the scattering medium. Here, α is the visibility factor. This model can be easily analyzed in the spectral domain since the model consists of a linear combination of f(x) and t(x). Thus, applying the linearity of Fourier transformation to Equation (2), we can easily obtain Equation (3) as follows.

$$G\left(\omega \right)=\alpha F\left(\omega \right)+\left(1-\alpha \right)T\left(\omega \right),$$

(3)

where G(ω), F(ω), and T(ω) represent the Fourier-transformed signals of g(x), f(x), and t(x), respectively. By estimating T(ω) and α, we can obtain the reconstructed F(ω) from G(ω). We utilize the frequency characteristics of the scattering medium to predict T(ω).

The results presented in Figure 2e demonstrate that the influence of the scattering medium dominates the very-low-frequency components of the image. Therefore, by utilizing the frequency characteristics of the scattering medium, we can determine the low-frequency components of the observed signal G(ω) as the components of T(ω) and the high-frequency components as the components of F(ω). These are then represented by

$$T\left(\omega \right)\simeq \frac{1}{\left(1-\alpha \right)}G\left(\omega \right), \omega \le f_c,$$

(4)

$$F\left(\omega \right)\simeq \frac{1}{\alpha }G\left(\omega \right), \omega >f_c.$$

(5)

The spectrum-based method [38] is illustrated in Figure 3. To estimate the effect of the scattering medium, f_c is determined to separate lower-band and higher-band signals. The cut-off frequency f_c is used for a low-pass filter to remove the effect of the scattering medium. The remaining high-frequency signal is then utilized to restore the high-frequency components of the original signal using the visibility coefficient α. The cut-off frequency f_c is determined through Equation (6).

$$f_c={\int }_0^1|G_N\left(\omega \right)d\omega |,$$

(6)

where G_N(ω) is the normalized value of G(ω).

The visibility coefficient α is determined by applying the Beer–Lambert–Bouguer law. According to this law, the amount of transmitted light I(z) = I₀·exp(−kz) is defined, where I₀ is the amount of light from the original source and z is the distance from the sensor. Here, k denotes the total attenuation coefficient depending on the wavelength of the illumination and the medium. This coefficient primarily determines the amount of attenuation, absorption, and scattering of light within the scattered environment. Since visibility decreases due to the attenuation of the original signal, the total attenuation coefficient and the visibility coefficient α are related. Therefore, α can be rewritten as

$$\alpha =e^{-kz}.$$

(7)

The parameter k is related to the variance v of G_N(ω) as

$$k=\frac{k_0}{v},$$

(8)

where $v={\int }_0^1{(G_N\left(\omega \right)-f_c)}^{2}d\omega$. Here, k₀ is a constant number which can be obtained experimentally.

3. Proposed Method

In this chapter, we discuss image enhancement of computational integral imaging via reducing the scattering artifact. A computational integral imaging reconstruction (CIIR) method can convert an elemental image array into a series of reconstructed images by adjusting the distance. These reconstructed images can suffer from blurring artifacts at out-of-focus locations. In particular, blurring artifacts from occluding objects can be scattered to the target object in reconstructed images. Thus, the reconstructed images for the target object can suffer from the scattering artifact from occluding objects, especially blurring or color changing. To enhance the reconstructed images, it is necessary to remove scattering artifacts caused by occlusion.

In the previous chapter, it can be observed that the visibility coefficient α is the most crucial factor in removing the scattering components of the image. Based on this, this paper presents a novel approach for estimating the visibility coefficient and applies it to the proposed computational integral imaging reconstruction method to improve image quality. The proposed method is depicted in Figure 4.

First, a reconstructed image is obtained from an EIA using the standard CIIR method. It is then separated into a low-band image and a high-band image using a low-pass filter. The visibility coefficient α is determined by using the probability density function (PDF) of the high-band image. The histogram of the band image is stretched using the parameter α, and the low and high bands are merged to obtain the enhanced image.

The process that demands the most computational time in our proposed algorithm is the low-pass filter, because the filter size is very large. Thus, we employ a moving average filter as the low-pass filter. As the size of the filter increases, the computational complexity also increases, so we introduce integral images for fast computation. The integral image [42,43] is an image that contains the sum of pixels up to the current position in the image. By using an integral image, we can quickly calculate the sum of a rectangular region in an image regardless of its size.

3.1. Formulations of Proposed Model

We derive a new formula using Equation (3) from the spectrum-based method [38]. In contrast to the existing method, we eliminate the complicated process of determining the cut-off frequency f_c because a moving average filter is less sensitive to the cut-off frequency. We also introduce a new method for estimating α in our post-processing method, since the previous method of estimating α cannot be directly applied to a single image.

Figure 5 shows the spectra of G(ω), F(ω), and T(ω) images, assuming that only the low frequencies are present in T(ω), based on the results of Figure 2e.

In Figure 5, we can observe that the low-frequency components of G(ω) are greatly influenced by T(ω), while the high-frequency components of G(ω) are predominantly influenced by F(ω). These characteristics indicate that the low-frequency band F_l(ω) and the high-frequency band $F_h\left(\omega \right)$ of F(ω) can be represented as

$$F_h\left(\omega \right)=\frac{1}{\alpha }{G}_h\left(\omega \right).$$

(9)

$$F_l\left(\omega \right)\simeq \alpha G_l\left(\omega \right).$$

(10)

where G_l(ω) is the low-frequency band of G(ω) and G_h(ω) is the high-frequency band of G(ω). Since T_l(ω) and F_l(ω) cannot be directly determined in this model, F_l(ω) is estimated from G_l(ω) and the visibility coefficient α. Based on the analysis, our model suggests that the F(ω) of the original image, unaffected by the scattering medium, can be written as

$$F\left(\omega \right)=F_l\left(\omega \right)+F_h\left(\omega \right)={\alpha G}_l\left(\omega \right)+\frac{1}{\alpha }{G}_h\left(\omega \right)+128.$$

(11)

3.2. Estimation of Visibility Coefficient

To estimate the visibility coefficient α in our model, we employ the probability density function (PDF) of the high-frequency band. Figure 6 shows the typical PDF of a high-frequency band of an image. In a general image unaffected by the scattering medium, the PDF of its high-frequency band exhibits a broad Gaussian shape, as shown in Figure 6b. However, in an image affected by the scattering medium, the presence of strong low-frequency components results in a narrower shape for the PDF of the high-frequency band. This is illustrated in Figure 6d. These characteristics can be utilized to approximate the PDF of the image affected by the scattering medium to that of the normal image. Thus, the coefficient α can be determined by a range ratio of two PDFs, which is written as

$$\alpha =\frac{R_1}{R_2}=\frac{Range\left[pdf\left(G_h\left(\omega \right)\right)\right]}{Range\left[pdf\left(F_h\left(\omega \right)\right)\right]}, (R_2=128)$$

(12)

$${\int }_{-R_1/2}^{R_1/2}p\left(x\right)dx=c, (c=0.95)$$

(13)

In Equation (12), G_h(ω) is the high-frequency band image affected by the scattering medium and F_h(ω) is high-frequency band image of the original image. The coefficient α is determined by the ratio of the range of the probability density function of G_h(ω) to the range of the probability density function of F_h(ω). The range of the probability density function of F_h(ω) is experimentally determined with a value of 128. The range of the probability density function of G_h(ω) is defined as Equation (13), where p(x) represents the probability density function of G_h(ω). The range of p(x) is calculated by the point where a rate of 95% of the total number of pixels is populated. The reference rate is also determined experimentally. Once the coefficient α is determined, the proposed histogram stretching is performed using Equation (11). For example, illustrated in Figure 7 is the comparison of the two ranges of PDFs for the hazed image and the original image. And the proposed histogram stretching method can be effective for matching the two ranges.

4. Experimental Results and Discussion

To evaluate the proposed algorithm, we conducted two experiments. First, we carried out an experiment with respect to visibility coefficient estimation. The objective of the first experiment was to evaluate the accuracy of the estimated visibility parameter. To achieve this, experimental images were generated from the original images and scattering medium images, as shown in Figure 8. Here, the experimental images ere generated based on the α-blending method using Equation (3), as illustrated in Figure 9. A total of 24 images were generated using Equation (3), with the visibility coefficients adjusted to 0.3, 0.5, and 0.7. The experiment compared the estimation of α using the previous method with that using the proposed method. And the size of the moving average filter was set to 131 × 131 for the proposed method.

Figure 10 shows some of the resulting images from the experiment, indicating that the quality of the reconstructed images improved subjectively. It can be observed that the resulting images obtained from the previous method with inaccurate visibility coefficients suffered from over-amplification in color. To quantitatively compare these results, we computed the peak signal-to-noise ratio (PSNR) and the structural similarity index measure (SSIM).

Table 1. Estimation results of visibility coefficient.

Visibility Coefficient	0.3		0.5		0.7
	Inaccurate Alpha	Proposed Method	Inaccurate Alpha	Proposed Method	Inaccurate Alpha	Proposed Method
Baboon_hazed image1	0.18	0.32	0.29	0.53	0.40	0.74
Baboon_hazed image2	0.19	0.33	0.30	0.53	0.40	0.74
Lenna_hazed image1	0.15	0.34	0.25	0.54	0.35	0.76
Lenna_hazed image2	0.16	0.34	0.24	0.55	0.34	0.76
Kodim22_hazed image3	0.20	0.27	0.33	0.42	0.46	0.59
Kodim22_hazed image4	0.20	0.27	0.33	0.44	0.46	0.59
Kodim23_hazed image3	0.19	0.34	0.30	0.56	0.42	0.79
Kodim23_hazed image4	0.19	0.34	0.31	0.56	0.42	0.79
AVE	0.18	0.32	0.29	0.52	0.41	0.72

presents the visibility coefficients estimated by the proposed method and the previous method. The proposed method demonstrated more accurate estimated values for α, with averages of 0.32 for α = 0.3, 0.52 for α = 0.5, and 0.72 for α = 0.7, in comparison to the previous method. The PSNR results shown in

Table 2. PSNR results with visibility coefficient estimation.

Visibility Coefficient	0.3		0.5		0.7
	Inaccurate Alpha	Proposed Method	Inaccurate Alpha	Proposed Method	Inaccurate Alpha	Proposed Method
Baboon_hazed image1	17.82	21.06	17.96	23.10	17.86	26.10
Baboon_hazed image2	18.33	21.72	18.14	23.66	17.95	26.50
Lenna_hazed image1	16.96	21.66	16.64	24.44	16.59	27.64
Lenna_hazed image2	16.75	22.43	15.96	24.98	16.21	28.20
Kodim22_hazed image3	16.73	17.15	17.56	18.20	18.17	19.26
Kodim22_hazed image4	16.60	17.20	17.24	18.04	17.89	18.96
Kodim23_hazed image3	15.69	16.01	16.08	16.89	16.45	17.92
Kodim23_hazed image4	15.79	16.30	16.02	17.09	16.35	18.04
AVE	16.83	19.19	16.95	20.80	17.18	22.83

indicate that the proposed method yielded a PSNR improvement of 3.99 dB on average.

Table 3. SSIM results with visibility coefficient estimation.

Visibility Coefficient	0.3		0.5		0.7
	Inaccurate Alpha	Proposed Method	Inaccurate Alpha	Proposed Method	Inaccurate Alpha	Proposed Method
Baboon_hazed image1	0.84	0.93	0.86	0.97	0.86	0.99
Baboon_hazed image2	0.88	0.96	0.87	0.98	0.86	0.99
Lenna_hazed image1	0.85	0.93	0.84	0.98	0.83	0.99
Lenna_hazed image2	0.84	0.97	0.81	0.99	0.82	0.99
Kodim22_hazed image3	0.69	0.71	0.85	0.89	0.88	0.94
Kodim22_hazed image4	0.76	0.83	0.85	0.91	0.87	0.94
Kodim23_hazed image3	0.68	0.69	0.81	0.86	0.83	0.92
Kodim23_hazed image4	0.77	0.86	0.81	0.91	0.82	0.93
AVE	0.79	0.86	0.84	0.94	0.85	0.96

presents the results of the SSIM values between the original images and the resulting images. The SSIM results also demonstrate that the proposed method was superior to the previous method in terms of the improvement in image quality. This is because the visibility coefficients were accurately estimated by the proposed method with the spectral model. These results indicate that accurate estimation of the visibility coefficient plays an important role in improving image quality.

In addition, we carried out another experiment by applying the proposed method to CIIR. The objective of the second experiment was to evaluate the performance of the proposed CIIR method. Here, the proposed method was compared with two existing methods. As shown in Figure 11, EIAs were generated from original test images. Then, CIIR methods were used to convert the generated EIA into the reconstructed images. The sizes of the input images were either 512 × 512 or 768 × 512. The number of elemental images in EIA was 17 × 17, and each elemental image consisted of either 150 × 150 pixels or 300 × 300 pixels. The distances between the virtual pinhole array and the objects were z mm and 2z mm, respectively. Here, a moving average filter was employed as the low-pass filter, with a filter size of 131 × 131.

Table 4. PSNR results with CIIR methods.

	CIIR	Inaccurate Alpha	Proposed Method
Baboon	18.89	17.64	23.66
Lenna	19.07	16.10	24.58
peppers	19.74	16.03	19.18
fruits	17.10	15.75	17.82
Kodim07	21.99	16.66	18.80
AVE	19.36	16.44	20.81

and

Table 5. SSIM results with CIIR methods.

	CIIR	Inaccurate Alpha	Proposed Method
Baboon	0.90	0.85	0.95
Lenna	0.93	0.81	0.97
peppers	0.92	0.85	0.89
fruits	0.87	0.79	0.89
Kodim07	0.92	0.77	0.90
AVE	0.91	0.81	0.92

show the PSNR results and the SSIM results, respectively. The PSNR results reveal that the proposed method provided an average improvement of 1.45 dB over the standard CIIR method and 4.37 dB over the previous CIIR method. Furthermore, the SSIM results demonstrate the superiority of the proposed method over existing approaches. As depicted in Figure 12, the resulting images from the proposed method provided better performance in terms of subjective image quality compared with the standard CIIR method. Therefore, the proposed method achieved both objective and subjective improvements in the image quality of computational integral imaging systems.

5. Conclusions

We have presented an image enhancement method for computational integral imaging by introducing an efficient estimation technique for the visibility coefficient. Our method is based on a new spectral model for the scattering artifact. Unlike existing approaches that utilize a pre-processing method for elemental images before CIIR, our method employs a post-processing method for the reconstructed images after CIIR. The experimental results indicate that our approach enhances the image quality of computational integral imaging systems by eliminating scattering artifacts. Additionally, we introduced a novel method for estimating the visibility coefficient, a crucial parameter to eliminate scattering artifacts. The computer experiments indicated that our CIIR method was superior to the existing methods. Thus, we expect that the proposed method will be useful in improving the image quality of 3D computational integral imaging systems and image-dehazing systems [44,45,46,47,48,49,50,51,52,53,54,55].