Recording of Full-Color Snapshot Digital Holographic Portraits Using Neural Network Image Interpolation

Featured Application

This work will make it possible to easily set up snapshot holographic portrait studios and allow holographic portraits to be developed and disseminated to the public on a larger scale.

Abstract

For 60 years, high-energy pulsed lasers have enabled the recording of extremely realistic portraits of living people within a few nanoseconds. In the 21st century, this confidential technique, which requires great skill and equipment, has almost disappeared because of the final monochromatic color of the holograms, which does not appeal to the public. In this study, we show that ultra-realistic, full-color, and full-parallax snapshot holographic portraits can be recorded using a limited number of synchronized HD cameras. Experiments were successfully conducted using the digital CHIMERA holographic stereogram printing technique combined with image interpolation using a neural network. This new recording method allows holographic portraits to be developed and disseminated to the public on a larger scale.

1. Introduction

Currently, the most precise way of representing a person is through pulsed holographic portraiture [1,2,3,4]. A pulsed portrait is a hologram recorded using high-energy (several Joules) ruby (red) or neodymium doubled in frequency (green) pulsed lasers. The extremely short exposure time of approximately 20 ns allows successful recording of living subjects in a dark room. The originally recorded transmission master hologram (H1) is then transferred to obtain a second reflection hologram (H2) on a scale of 1:1 with a limited full parallax, usually at 40° and 30° with respect to the horizontal and vertical axes, respectively, which is observable with white light. In the 21st century, this confidential technique, which requires heavy and expensive equipment and holographic skills, has almost disappeared. Indeed, if the three-dimensional (3D) rendition is spectacular, the main obstacle to success is the final monochromatic color of the holograms. Green or red colors often make holographic portraits cadaverous and ghostly for the public, which is used to seeing images of human beings in black-and-white or in color.

In 1995, Yamaguchi et al. [5] proposed the recording of holographic stereograms divided into a matrix of small elements or hogels [6]. In 2019, the third generation of holoprinters called CHIMERA [7]—based on three low-power, continuous, red, green, blue (RGB) lasers combined with the silver-halide material Ultimate U04 [8]—printed full-color and black-and-white digital holograms with a hogel resolution of 250 μm, a speed of 60 Hz, and a full parallax of 120°.

Holographic stereogram recordings require the prior acquisition of a series of perspective images. In the case of a portrait, the recordings are usually conducted with a video camera rotating around the subject (Figure 1a) with an arc-curved rail guide, or by placing it on a turntable (Figure 1b) using a fixed video camera. The vertical rotation axis is the vertical axis of the final hologram, and everything in front of this axis is derived from the image. For aesthetic reasons, this axis is usually placed at eye level. For successful recordings, the model must keep their head still during shooting and ignore the relative movement of the camera.

Unlike the pulsed laser technique, the recording of these portraits is not instantaneous and lasts for approximately 30 s, during which the subject has difficulty remaining motionless, mainly at eye level, yielding an unnatural result. Moreover, the portraits obtained had only a horizontal parallax. To obtain more realistic images, this study proposes a method for recording instant full-color and full-parallax holographic stereogram portraits using multiple synchronized cameras. Images were interpolated using a neural network [9] to limit the number of cameras used as much as possible.

2. Materials and Methods

2.1. Holographic Plates for CHIMERA Recordings

CHIMERA holograms were recorded on a silver halide holographic Ultimate U04 glass plate (Ultimate Holography, Bordeaux, France). This material was specially designed for recording full-color holograms without any diffusion and was set to be isopanchromatic for all visible wavelengths. The plates were developed in two safe and easy-to-use chemical baths.

2.2. CHIMERA Recordings

A half-parallax CHIMERA was created by recording a series of 768 horizontal perspective images on a 120° circular arc. A full-parallax CHIMERA maintains the same requirements; however, the procedure is repeated at multiple elevation levels (at most 192). In-house software generated all hogels from the perspective images. Each hogel was recorded sequentially using an RGB display system made of three spatial light modulators (SLM) and a 120° full-color optical printing head. After interference with the reference beam, the information corresponding to each RGB hogel was recorded in the U04 plate. The hogel size was 250 µm and the printing rate was 60 Hz. The CHIMERA holoprinter used three RGB DPSS 20 mW lasers. The wavelengths were 640, 532, and 457 nm. The CHIMERA can be used as a master and can be contact-copied with the same quality on both U04 silver halide material (film or plate) and industrial photopolymer films.

2.3. Sealing and Illumination of CHIMERA Holograms

To reconstruct the CHIMERA holograms to the same wavelengths used when recording, and to prevent any emulsion thickness variations due to changes in humidity or temperature, they were sealed with a second glass using optical ultraviolet (UV) glue. Moreover, the reconstructed light must be a source point containing the wavelengths of the original recording lasers. RGB light-emitting diodes (LED) currently offer the best solution because their wavelengths are centered on laser wavelengths [10].

2.4. Image Recordings

The images were recorded using a set of identical high-definition (HD) cameras connected to a computer via a universal serial bus (USB). For the half-parallax portrait, cameras were placed facing the model horizontally and at equal distances on a circular arc. To obtain a full-parallax portrait, the recording setup was completed using additional camera levels. The model was uniformly illuminated by soft boxes that mimic daylight; these constitute the standard for fashion and portrait photography lighting. It offers a flattened light source for models and a diverse set of lighting options for photographers.

2.5. Image Interpolation

To limit the number of cameras used, frame-interpolation synthesized intermediate frames between two input images. These are frequently used to produce slow-motion videos. Several studies have been conducted [11,12] and recent techniques based on a new neural network approach by Reda et al. in 2022 [13] have provided impressive results for large-motion interpolation. Near-duplicate photos were transformed into slow-motion sequences that resembled shots captured by a video camera. Codes are available on GitHub [14] with a pre-trained model from triplet images. An in-house program installed on a fast graphics computer used modified codes to automatically resize, rotate, and interpolate pictures recorded by the cameras.

2.6. Simulations with a 3D Computer Graphics Program

The larger the angle between two images is, the more difficult it is to achieve smooth interpolation. To determine the parameters and build an appropriate shooting studio, preliminary tests were conducted using a 3D computer-generated imagery (CGI) head model and the 3D computer graphics program Autodesk 3ds Max. A series of interpolations were performed by changing the rotation angle between the two frames. The obtained images were compared with actual images to determine the maximum acceptable angle between the two cameras for faithful reconstruction.

2.7. Evaluation of the Accuracy of the Reconstruction

To evaluate the accuracy of the images created by interpolation, they were compared with actual images. The color of each pixel obtained was checked and transformed to red if it differed from the real color by more than 2% in any of the three channels (R, G, or B).

3. Results

3.1. Tests with the 3D CGI Head

A series of interpolations between frames were obtained using the 3D CGI head model and 3ds Max with rotation angles in the range of 5–60°. The interpolated images were compared with the actual images by evaluating the colors of the obtained pixels. Figure 2 shows the percentage accuracies of these images as a function of the rotation angle and indicates the poorly reconstructed areas in red. The results show that the interpolated images remain consistent to the original images for rotation angles up to 20°. Beyond this angle, unnatural deformations appear on the face and neck and are no longer correctly reconstructed. We determined that the maximum acceptable angle between the two cameras with no or few deformations was 20°.

A test was then performed with seven horizontal images separated by 20° on a 120° circular arc, and three elevation levels separated by 20° (Figure 3a). First, a total of 128 images were interpolated between each pair of images horizontally; 64 images were then interpolated vertically. A final total of 99,072 images were obtained, among which only 21 were actual images (0.02%). The interpolation time of all images with a fast-graphics computer was 6 h. Hogels were then generated from these perspective images and recorded with a 250 µm resolution one after another into a 15 × 20 cm U04 holographic plate. The hologram was then developed using two chemical baths and sealed with optical glue to prevent variations in the emulsion thickness. When CHIMERA was illuminated 50 cm from the center of the hologram and at an angle of 45° with an RGB LED lamp, a fine full-color and full-parallax reconstruction of the 3D CGI face was generated, as shown in Figure 3b and Video S1. Observations show that the unnatural deformations present in some of the interpolated images are not visible in the final hologram.

3.2. Recordings of Live Portraits

A half-parallax shooting studio was constructed with seven cameras placed horizontally every 20° on a 120° arc of a circle and at 1 m from the center, as shown in Figure 4. To obtain a full-parallax portrait, the device was completed with two additional camera levels separated vertically by 20° on either side of the initial level, as shown in Figure 4b. Owing to the limited number of cameras available for the first experiment, only four cameras were installed on each of the three levels on a 60° arc of the circle; in total, 12 cameras were used.

Seven photographs of the half-parallax portrait (Figure 5a) were simultaneously recorded within 100 ms. The coiffed model of a labeled cap, uniformly illuminated by three soft-boxes, was recorded in a three-quarters pose, which is standard for classical portraiture. The head was positioned halfway between the front and profile views, and the gaze was directed toward the central camera. The rotation axis was placed at eye level. In total, 128 images were then interpolated between each pair of frames in less than 5 min; this generated a total of 768 images, among which only seven were actual (0.92%). The rotating movement of the camera around the model seemed smooth and natural for the face and clothes, especially for the label, as shown in Video S2. When the final 15 × 20 cm sealed full-color 120° half-parallax CHIMERA recorded from these perspective images was illuminated at an appropriate distance with an RGB LED lamp, a fine reconstruction of the model head appeared, as shown in Figure 5b and Video S3. This portrait was realistic and natural. The eyes were perfectly still and all parts of the face, clothes, and writing on the cap were perfectly reconstructed. Moreover, at a resolution of 250 µm, the hogel grid was nearly invisible to the naked eye. However, as the hologram was half-parallax, the point of view remained unchanged when moving vertically.

To obtain a full parallax portrait, 12 photographs of the new model distributed on three different levels were recorded (Figure 6a). The recording time of all pictures was slowed down to 300 ms owing to the higher number of cameras to be managed in the USB. In total, 128 images were interpolated horizontally between each pair of images, then 64 vertically, to obtain a final total of 49,152 images, among which only 12 were actual images (0.12%). The interpolation time for all the images was 3 h. When the final 15 × 20 cm CHIMERA was illuminated, an even more realistic reconstruction of the model appeared owing to the addition of vertical parallax, as shown in Figure 6b and Video S4; this provided the observers with the sense that the actual person was in front of them. The final hologram had a horizontal parallax of 60° and a vertical parallax of 40°.

4. Discussion

This study’s results confirmed that ultra-realistic, snapshot, digital holographic portraits can be recorded using a limited number of cameras. Because of the image interpolation implemented by the neural network, the maximum angle between the two cameras was fixed at 20°; beyond this angle, unnatural deformations appeared. Only seven cameras were required to record a half-parallax portrait and twelve for a full portrait. The images were recorded almost instantaneously, and the models remained perfectly intact without any unnatural eye movements. Complex elements, such as writing, can also be perfectly reproduced.

The two recorded CHIMERA portraits presented a fine 3D reconstruction of the models’ heads, with the full-parallax portrait being the most realistic. However, the half-parallax method has several advantages. It requires fewer cameras and shorter shots (100 ms versus 300 ms), thus resulting in fewer errors. The time required to interpolate all images was also faster (5 min versus several hours), which shortened the time between shooting and delivery of the final hologram to a client. Moreover, when the portrait is hung on a wall, most observers do not perceive that the hologram only contains a horizontal parallax.

This new camera-based technique has several advantages and is much easier to implement than pulsed lasers for recording portraits of living subjects. These methods do not require expensive lasers or advanced technical holographic skills. As the recording of the images is performed in the light and not in the dark, the operations are easier for both the photographer and model. The recording is so simple that we can consider creating an automated shooting studio or “holo-booth.” It is not necessary to record and develop an H1 hologram to assess the final result because the photographs recorded by the cameras can be viewed instantly on a computer screen. The recording time with the pulsed laser was 20 ns, and the model did not move during the shooting. A longer recording time with synchronized USB cameras (100 ms) can sometimes lead to slight model movements. This is not a problem if it is possible to repeat the operation as many times as necessary until the most satisfactory portrait is obtained.

A pulsed H2 portrait has a spectacular 3D rendition and very high-resolution features; however, it is monochromatic with limited parallax. The CHIMERA portrait had the same 3D rendition characteristics. If the resolution is lower (with 250 µm hogels) the grid is nearly invisible to the naked eye. The CHIMERA parallax is also more important (up to 120°) depending on the number of cameras used.

The largest difference is the addition of colors, which make the portrait much more natural and pleasant for the observer. The use of UV glue to seal the CHIMERA and the use of LED lamps to illuminate them allow faithful reproduction of the original colors, which is very important in the case of realistic human portraits. Moreover, a pulsed H2 portrait is always rendered on a 1:1 scale, whereas a CHIMERA hologram can be printed at all reduced or enlarged sizes. Finally, because the CHIMERA portraits can be used as masters, they can be reproduced in large quantities on industrial photopolymer films and disseminated on a large scale. Furthermore, the portraits were recorded from a series of photos that can be modified using a graphic editor or artificial intelligence program to embellish or rejuvenate the model or change the background. The CHIMERA also allows the printing of black-and-white holograms.

5. Conclusions

The combination of the CHIMERA technology with neural network image interpolation offers a new, simpler, and faster method for recording holographic portraits. It provides better results in terms of color and parallax and meets the expectations of the public. This method is envisaged to allow the widespread dissemination of modern holographic portraits.