A Fast Computer-Generated Holographic Method for VR and AR Near-Eye 3D Display

Featured Application

The proposed technique has potential for head mounted near-eye VR or AR applications.

Abstract

A fast computer-generated holographic method with multiple projection images for a near-eye VR (Virtual Reality) and AR (Augmented Reality) 3D display is proposed. A 3D object located near the holographic plane is projected onto a projection plane to obtain a plurality of projected images with different angles. The hologram is calculated by superposition of projected images convolution with corresponding point spread functions (PSF). Holographic 3D display systems with LED as illumination, 4f optical filtering system and lens as eyepiece for near-eye VR display and holographic optical element (HOE) as combiner for near-eye AR display are designed and developed. The results show that the proposed calculation method is about 38 times faster than the conventional point cloud method and the display system is compact and flexible enough to produce speckle noise-free high-quality VR and AR 3D images with efficient focus and defocus capabilities.

1. Introduction

A holographic display is a promising candidate for 3D display because the phase and amplitude of a 3D scene can be completely reconstructed. It is possible to implement computer-generated holograms (CGH) for a holographic 3D display [1,2]. High resolution holograms such as rainbow holograms and Fresnel holograms have been proposed [3,4]. Many dynamic holographic display solutions have also been implemented [5,6,7,8,9,10]. Among them, notable frequency domain multiplexing methods are used for color video display through Fresnel hologram with red, green and blue lasers as illumination [7,8]. However, such an optical system is often bulky and the image quality is poor because of the coherent illumination, which affects the practical application of the technology. A lot of efforts have been made, but the progress on the holographic 3D display still has proven difficult due to the following constraints, such as the computational difficulty for the large size of 3D point cloud data, the limited bandwidth of spatial light modulator (SLM), and the speckle noise of the reconstructed 3D images due to the use of coherent illumination sources. The narrow diffraction angle of the current SLM often dictates that the holographic 3D display is restricted to quite small 3D images and narrow viewing angles, with no sign of a significant improvement in the short run.

While these difficulties exist, Virtual Reality (VR) and Augmented Reality (AR) displays have become emerging technologies. In the VR display, the human eye views the virtual image through the display device. However, in the AR display, users can view both real scenes and some virtual images simultaneously [11]. In this technique, the virtual image is displayed in front of the human eye through an optical system and AR eyepiece and ambient light from a real object can pass incident on the human eye through the AR eyepiece without distortion. The combination of holography and VR or AR display to achieve a true 3D display is quite intuitive and thus desirable, which can avoid the accommodation convergence conflict problem existing in binocular parallax-based 3D displays [12,13].

The Fresnel hologram is often chosen to produce 3D displays with large depths of field using a coherent source such as laser as illumination. However, it is not quite suitable for holographic VR or AR display because of the large imaging distance between the reconstructed image and the Fresnel hologram. Additionally, for use of a coherent illumination source, speckle noise of holographic 3D display can also bring a negative impact on image quality.

Several hologram computation methods such as point cloud-based, layer-based, and triangular mesh-based algorithms [14,15,16,17,18,19] have been proposed for speeding up the calculation of the Fresnel hologram. To improve the calculation efficiency, holographic stereogram methods using numerous projection images have been proposed [20,21]. The essence of those methods is to represent the information of one point in the frequency domain by the superimposition of one orthogonal projection image multiplied by the phase of the corresponding direction. Subsequently, the frequency is converted into a spatial domain for an image plane hologram calculation or using Fresnel diffraction for Fresnel hologram calculation. While those methods have merits, a huge number of projected images are required for high-resolution hologram calculation.

In order to achieve an efficient and more compact holographic display system, a hologram calculation method similar to holographic stereogram and an optical display system for holographic near eye VR and AR display are proposed. Within this calculation method, a plurality of images with different angles are projected on a projection plane near the holographic plane. Subsequently, the optical field on the holographic plane is computed by superposition of the convolution of each 2D projected image with specific point spread functions (PSF) corresponding to propagation from the projection plane to the holographic plane in the appropriate directions. Additionally, in order to reduce the speckle noise, LED is chosen as the light source for the optical display system. 4f optical filtering system is utilized for filtering out the unwanted noise. Another lens is used as the eyepiece for VR display and reflective volume holographic optical element (HOE) is used as a combiner for AR display. Overall, the proposed display system is compact and displayed 3D images are free of speckle noise.

2. Methods

The concept of hologram calculation is shown in Figure 1 with two projection planes for ease of explanation since the projection of 3D object is multidirectional. In Figure 1a, A and B are two points in the space, and layer1 and layer2 are two projection planes. On each projection plane, the projected rays from the spatial 3D object points are in the same direction. p_1A and p_1B are the projected points of the spatial points A and B on layer₁, respectively. p_2A and p_2B are the projection points of space points A and B on layer2, respectively. The projection of 3D object points on one projection plane forms a projected image corresponding to this projecting angle. In the hologram calculation process, the illumination direction of each point on different projection planes is set as the propagation direction, as shown in Figure 1b. The complex amplitude distribution on the holographic plane H is a superposition of light field from all projection planes. When the hologram is reconstructed, the beamlets in these directions are diffracted from the hologram. Similar to integral imaging 3D displays, the overlapping regions of the different beamlets form a 3D object for 3D display [22,23], where the propagation direction of the beamlets determines the outcome of the 3D display.

To simplify calculation, the projection layers are all assumed to coincide in the same plane. On this projection plane, projected images from the 3D object are obtained with the designed projection angles. The light in different directions of the 3D object is calculated onto holographic plane H from the projected images. The direction of propagation of light on a projected image is ${\theta }_{ix}$ and angular interval is $\Delta {\theta }_{ix}$ as shown in Figure 2. The distance from the projection plane to the holographic plane H is z. The boundary of the propagation of the light from point p on the image on the holographic plane falls between $x_{ih1}$ and $x_{ih2}$, respectively.

Assume the point p is an on-axis point and the PSF of light propagating from the projection plane to the holographic plane can be expressed as:

$$ps{f}_{i,j}=\exp [i\frac{2\pi }{\lambda }(\sqrt{x_h{}^2+y_h{}^2+z^2})]$$

(1)

where $x_{ih1}\le x_h\le x_{ih2},y_{jh1}\le y_h\le y_{jh2}$.$x_{ih1}$ and $x_{ih2}$ are the boundary of the light field on the holographic plane limited by the projection angle and angle interval in the x direction, respectively. i and j are the index of projected images in x and y directions, respectively. $y_{jh1}$ and $y_{jh2}$ are the boundary of the light field on the holographic plane limited by the projection angle and angle interval in the y direction, respectively.

The boundary of the light field distribution area on the holographic plane limited by the projection angle $({\theta }_{ix},{\theta }_{jy})$ and angle interval $(\Delta {\theta }_{ix},\Delta {\theta }_{iy})$ can be expressed as:

$$\begin{array}{l}{x}_{ih1}=z\tan ({\theta }_{ix}+\frac{\Delta {\theta }_{ix}}{2});x_{ih2}=z\tan ({\theta }_{ix}-\frac{\Delta {\theta }_{ix}}{2})\\ y_{jh1}=z\tan ({\theta }_{jy}+\frac{\Delta {\theta }_{jy}}{2});y_{jh2}=z\tan ({\theta }_{jy}-\frac{\Delta {\theta }_{jy}}{2})\end{array}$$

(2)

where ${\theta }_{iy}$ and $\Delta {\theta }_{jy}$ are the projection angle and angular interval in the y direction, respectively.

The complex amplitude on the holographic plane H is the superposition of the convolution of each projected image with its corresponding PSF, which can be expressed as:

$$U(x,y)={\displaystyle \sum _{i=1:I}{\displaystyle \sum _{j=1:J}i{m}_{i,j}}\otimes ps{f}_{i,j}}$$

(3)

where I and J are the number of projected images in x and y directions, respectively. The complex amplitude U(x, y) on the hologram plane can be further encoded as a double phase hologram or as an off-axis amplitude hologram by introducing reference light [24,25]. Thus, the proposed approach has good degree of freedom for potential downstream of visualization tasks.

From above, it is evident that the simplified model consists of one projection plane on which 3D objects yield multiple projection images, as shown in Figure 3. The convolution of each projected image with a PSF yields complex amplitude distribution of the 3D object. Assume that the projection angles are ${\theta }_{ix}$ and ${\theta }_{jy}$ in the x and y directions, respectively. The coordinates of the spatial point $p(x_p,y_p,z_p)$ on the projection plane can be expressed as:

$$\begin{array}{l}{X}_P=x_p+z_p\tan ({\theta }_{ix})\\ Y_P=y_p+z_p\tan ({\theta }_{jy})\end{array}$$

(4)

where X_p and Y_p are the coordinates of projected point of p on the projection plane. The projected image im_i,j can be expressed as:

$$i{m}_{i,j}(X_p,Y_p)=A_P$$

(5)

where Ap is the amplitude of object p.

3. Experiments and Result Analysis

Our experiment utilizes the principle of off-axis amplitude hologram for achieving holographic VR and AR 3D display. Within the experiment, a liquid-crystal-on-silicon (LCoS) SLM is utilized for modulating the holograms. The parameters of the SLM are shown in

Table 1. Optical parameters in the experiment.

Parameters	Values
SLM horizontal resolution	1920
SLM vertical resolution	1080
SLM pixel size	8 μm
LED size	1 mm × 1 mm
Center wavelength of LED	528 nm
Bandwidth of LED	28 nm

. Green light LED from Osram with 1 W power, center wavelength of 528 nm, light-emitting area of 1 mm × 1 mm, bandwidth of 28 nm is used as the illumination source. The center wavelength is used for hologram calculation.

For the center wavelength, the diffraction angle of the SLM is

$${\theta }_d={\sin }^{-1}(\frac{\lambda }{2pix})=1.89°$$

(6)

where $pix$ is the pixel size of the LCoS. Given that the distance between the projection plane and the holographic plane is 10 mm, it is possible to calculate the PSF. It is assumed that the plane wave perpendicular to the holographic plane is used as a reference wave, which means that the phase of the reference wave on the holographic plane is a constant value. Under this situation, the real part of the PSF in Equation (1) on the holographic plane is an amplitude type hologram. This amplitude type hologram can be regarded as new PSF, as shown in Figure 4. The resolution of the PSF is about 84 pixels × 84 pixels, which corresponds to the viewing angle of 3.78° both in x and y directions from the hologram. For calculation of off-axis amplitude type hologram, only the upper half of the PSF (outlined with red rectangles) is used for the hologram calculation, as shown in Figure 4. The eight PSF sections correspond to eight PSFs of a 3D object projected onto one projection plane with eight different projection angles. Each PSF section has a resolution of 20 pixels × 20 pixels. Each PSF section is named as psf_i,j according their positions. Hence in this design, the angular separation between the two adjacent projected images in the x and y directions is set as 0.9°.

Figure 5 shows the calculation of the hologram. A 3D object with size of 4.23 mm × 7.05 mm × 3.52 mm and a distance of 5 mm from the center of 3D object to the projection plane is used for hologram calculation. This 3D color model contains 41,022 object points. Two rows and four columns of the projected images of the 3D object are calculated according to Equation (5), and each projected image is firstly converted to grayscale image with a resolution of 600 × 600 and then interpolated and zero padded into a projected image with resolution of 1920 × 1080. The process of convolving the projected images with the corresponding PSF is shown in Figure 5. The projected image $i{m}_{i,j}$ convolves with the corresponding $ps{f}_{i,j}$. Then, the results are superimposed to obtain the hologram.

The optical setup of the experiment for holographic 3D VR display is shown in Figure 6. In this setup, the divergent illumination light emitted by the green LED is collimated by the lens₁, and the LCoS is illuminated by the reflected light from beam splitter BS. The computer-generated hologram is loaded on the LCoS, and the illumination light is modulated. The modulated light field is filtered by the 4f optical filtering system to form a real 3D image free of zero order and high order image noise finally incident on human eye through the eyepiece lens₄. The lenses used for collimating, 4f optical filtering system and eyepiece are all cemented doublet achromatic lenses with focal length of 50 mm and diameter of 30 mm. In this case, the distance from the LCoS to the eye is less than 25 cm, which means this setup is a relatively compact display system.

The experimental results of the VR setup are shown in Figure 7. Figure 7a shows the reconstructed image of the hologram calculated by the proposed algorithm. For comparison, Figure 7b shows the result of the hologram calculated using the direct point cloud algorithm [14] of the red channel data from the color 3D model. The reproductions are similar, but the computational speed of our proposed method takes only about 2 s on average whereas the 3D cloud point data approach consumes about 76 s on average (38 times slower than our method). The comparison is based on the MATLAB 2015 with a laptop with i7-7700HQ CPU and 16G RAM.

Similarly, with the same experimental conditions, we also show another sample to demonstrate the capability of our method that can focus and defocus objects at a scene. For the tested scene, we utilize a stationary coral and a swimming fish 3D animation model (constructed using Unity3D), as shown in Figure 8. For projection of images, eight images at 0.9° angle interval both in the x and y directions are projected by Unity3D to generate projected images for each hologram calculation. In Figure 8a, the camera is aimed at two objects, and both are in focus yielding clear images because distance between the coral and the fish is close. As shown in Figure 8b,c, the distance between fish and coral becomes far in comparison to Figure 8a. Figure 8b,c show different focuses, where b is focused at coral and c is focused at fish.

The results of Figure 7 and Figure 8 show that the proposed holographic 3D VR display is high in image quality, free of speckle noise, and also able to realize 3D display.

For the AR experiment, volume HOE is used for producing the AR 3D image due to its angle and wavelength selectivity [24,25,26]. The recording of the HOE is illustrated in Figure 9. The light wave emitted from the laser is filtered and collimated by lens₁, and then split into two parts through the beam splitter (BS). The transmitted light passes through lens₂ to form a convergent spherical wave, which undergoes interference with the plane wave reflected by mirror₁ and mirror₂ on the holographic material to form the HOE. The laser used for HOE recording is a single mode semiconductor laser with a wavelength of 532 nm and power of 400 mW. The lens₂ is a commercially available aspherical mirror with focal length of 40 mm and a diameter of 50 mm (Thorlabs, AL5040M-A, NA = 0.55). The recording material we used in the experiment is a commercial holographic film (Ultimate Holography, U04), which requires an exposure density of ~600 μJ/cm² [27].

Using the optical recording system, the angular selectivity of the reflective volume HOE recorded without using lens₂ was tested, and the angular selectivity was within $\pm 5°$. Using the SLM described above, the diffraction angle is within $\pm 1.89°$, which means that most light diffracted by the SLM can be diffracted by HOE in a high efficiency.

Figure 10a,b are a schematic and a photograph of AR 3D display system, respectively. The angle between the illumination light and the HOE is approximately 30°. The holographic 3D real image reproduced by the optical system is imaged by the HOE and incident at human eye. Ambient light from the real object does not meet the angular selectivity of the fabricated reflective HOE, and thus can directly reach the human eye, forming a holographic AR 3D display.

The diameter of the fabricated HOE for AR 3D display is about 40 mm, which is shown in Figure 11a. The 3D model used for the AR experiment is the same as the second experiment of the VR display. Figure 11b,c shows two reconstructed images taken at different depths. Fish and coral are about 1.2 m and 3 m deep, respectively, which are calibrated with a CCD camera. The calibration process is as follows: Given the camera is focused on a virtual image (e.g., either fish or coral), we move a real object away from the camera. When the captured image of the real object becomes clear, the distance between the real object and the camera is the depth at which the camera can clearly image the real object. The distance is also the distance between the virtual image and camera as well. In our display experiment, when the image of the fish is clear, the image of the coral and external environment are blurred due to the defocus. The results show that the proposed method is able to produce correct depth cues, and hence the accommodation convergence conflict problem can be eliminated.

4. Discussion

The VR results of our method are faster (38 times faster) than the traditional point cloud method. The VR results also show that the proposed method is able to achieve focus and defocus of the 3D images. Similarly, AR results also show that the proposed method is able to achieve a good image quality with 3D display capability as well. More importantly, use of LED as illumination source results in speckle noise-free VR/AR 3D images so that the image quality is better than the use of coherent source as illumination for holographic display. Another advantage of this method is that the depth of the 3D image can be easily controlled through the SLM.

It is worthwhile to point out that the calculation time is determined by the number of views of the projected image and the distance from the projection plane to the holographic plane. The computation time is a trade-off between the number of the projected images and the distance between the projection plane and the holographic plane. Further increase of computational speed is still possible through the use of GPU acceleration for the convolution.

It is also worth noting that the difference between the display modes of AR and VR can be easily changed with different eyepieces, and all the underlying computation should be identical. This offers convenience for easily switching between VR and AR devices for practical applications. The 3D object can be set between the holographic plane and projected plane or the projected plane can be set between the 3D object and holographic plane. In the experiment we used the former method. For an extreme situation, the holographic plane is the projection plane. Under this condition, the convolution in spatial domain is not possible because the PSF cannot be calculated. However, the calculation of the hologram can be performed in the frequency domain [4].

It should be pointed out that use of LED as the illumination source is also associated with potential drawbacks. For example, due to the large light-emitting area of the LED and poor spatial coherence, it is more difficult to produce a large depth of field image. Another disadvantage is that when the increase of the distance between the holographic plane and the 3D object is over a certain threshold (e.g., 20 mm), the blur of the VR/AR image becomes more evident. While this may not be significant for near-eye VR/AR 3D display, a careful control of coherence of the light source as well as more controllable propagation of the light can be potentially useful to solve these problems.

5. Conclusions

We demonstrate a holographic 3D VR and AR technique with a multiple projection image-based method to successfully create high quality near-eye VR and AR 3D display free of speckle noise and without accommodation convergence conflict problems. The proposed calculation method has high computational efficiency and the proposed display system is compact and flexible, which has potentials for head mounted near-eye VR and AR applications.