Watermarking and Encryption for Holographic Communication

Abstract

Holographic communication is a three-dimensional (3D) video communication technology based on computer-generated holograms (CGHs) which has the potential to give users a more realistic visual perception. As this is an emerging field, the encrypted encoding and decoding methods in holographic communication have not been widely studied. In this work, a watermarking and encryption method for holographic communication is proposed. A watermark is inserted into the original image using the discrete cosine transform before the calculation of the CGH, while a secret key is employed to produce the encrypted CGH during the holographic calculation. Through the proposed watermarking and encryption method, the signal of holographic communication is difficult to decrypt. Even if the signal is decrypted, the source of the leak is easy to trace due to the existence of the watermark. The watermarking and encryption method can provide a practical solution for the privacy protection and copyright protection of 3D video communication.

1. Introduction

The increasing demand for emerging applications including online conferences, remote collaboration, and the metaverse has significantly promoted the development of video communication technology. Compared to conventional two-dimensional (2D) video communication, three-dimensional (3D) video communication provides depth information, which gives users a more immersive experience. Therefore, it has been widely studied in recent years. For example, Holoportation, which is a binocular 3D communication system, demonstrates a high-quality real-time 3D reconstruction of the entire space with an AR helmet [1]; TeleHuman2, which is a cylindrical 3D communication system, conveys a full-body 3D video of interlocutors with a large-size cylindrical screen [2]; Starline, which is a flat 3D communication system, lets two people who are separated by distance have a face-to-face conversation with a light-field display system [3].

Considering the realizability, the monitors of current 3D video communication systems are often designed using binocular 3D displays and light-field 3D displays. In addition to these mentioned display technologies, the holographic 3D display is an option that can be applied in future 3D video communication. It is an approach that realizes 3D reconstruction using the diffractive pattern that records both the amplitude and phase of the 3D target. It presents all the 3D depth cues that human eyes can perceive, providing perfect authenticity and immersion for the users, which is widely considered the ultimate solution for 3D video communication [4].

For the holographic display, the generation of the diffractive pattern can be realized through either the optical configuration or the holographic algorithm. Compared to the optically recorded pattern, the algorithm-generated pattern, which is often called a computer-generated hologram (CGH), has the advantages of better consistency and easier replication. Thus, the CGH-based holographic display shows a better application prospect in the field of 3D video communication.

In a CGH-based holographic communication system, the main concerns can be classified into four categories: the rendering of 3D contents, the calculation of CGHs, the transmission of CGHs, and the construction of the holographic display system. In our previous work, the rendering of 3D contents [5], the calculation of CGHs [6,7], and the construction of the holographic display system [8] have been properly addressed. In this work, the transmission of CGHs is discussed in detail.

To protect privacy and copyright, the signal of holographic communication should keep its encryption during transmission. Holographic encryption has been widely studied since double random phase encoding was proposed [9]. For a CGH-based system, holographic encryption is often realized based on optical diffraction via spatial light modulators (SLMs) [10,11,12,13,14,15,16]. However, for the application of holographic communication, these encryption approaches are limited by the computational load and the system complexity. A general and efficient encryption method independent of the holographic display system is significantly necessary for holographic communication. Additionally, if the encrypted signal is decrypted by non-target users, the source of the leak should be easy to trace. In traditional video communication, by inserting a watermark containing some important information including the IP address and account number of a target user into the original image, the source of the leak can be readily traced [17,18,19,20,21,22]. However, for holographic communication, the encoding and reconstruction of the CGHs bring a lot of noise into the displayed images [23]. Influenced by the introduced noise, the watermark inserted into the original image might be difficult to extract. The watermark insertion and noise elimination should be conducted simultaneously.

In this work, a watermarking and encryption method for holographic communication is proposed. A watermark is inserted into the original image using discrete cosine transform (DCT) before the calculation of the CGH. An iterative angular spectrum method (IASM) is used to eliminate the noise introduced through the encoding and reconstruction of the CGHs. A secret key is employed to produce the encrypted CGH during the holographic calculation. Through this method, the signal is encrypted and watermarked, protecting privacy and copyright in holographic communication.

2. Methods

The signal side and the display side of the holographic communication system are shown in Figure 1a,b, respectively. On the signal side, the original image to be displayed is first watermarked using DCT. Then, the watermark that is used as the tracking tag is hidden in the processed image, which cannot be observed by human eyes. To encode the watermarked image into a CGH, the accuracy of the holographic algorithm should be considered. If the accuracy of the obtained CGH is inadequate, the error introduced during the calculation would appear as speckle noise in the holographic reconstruction. The speckle noise is likely to disturb the inserted watermark, making it difficult to be extracted. Therefore, an accurate holographic calculation method called IASM is employed to obtain the unencrypted CGH from the watermarked image. To enhance the confidentiality of the transmission, the obtained CGH is then encrypted via a secret key, which is essentially a pseudo-random phase. The encrypted CGH can be transferred from the signal side to the display side by the routers.

On the display side, the received CGH is first decrypted using the same secret key. The decrypted CGH is then uploaded onto a phase-only SLM and illuminated with a coherent plane wave. After the optical reconstruction, the displayed image can be observed by the human eye. If a user records and disseminates the image contents without permission, the inserted watermark can be extracted from the recorded image using DCT. According to the information in the watermark, the source of the leak can easily be traced. The detailed processes of watermark insertion and holographic encryption and decryption are described in the following parts.

2.1. Insertion of the Watermark

The DCT-based insertion of the watermark is shown in Figure 2. For simplicity, the resolution of the original image is set as M × M pixels in this work. A binary image, with a resolution of N × N pixels, is employed as the watermark. Typically, the value of M/N is an integer. In the application of holographic communication, two critical issues should be considered during the watermark insertion: the invisibility of the watermark and the time efficiency of the insertion. The invisibility ensures that the inserted watermark does not occlude the displayed result, while the time efficiency ensures that the insertion does not bring time delay to the communication. DCT is a widely studied method in the field of watermark insertion. Through DCT, the image can be processed in the frequency domain, rather than the spatial domain. More importantly, the processing of high-frequency components in the frequency domain does not significantly affect the intensity distribution in the spatial domain. The invisibility of the watermark can easily be realized through the DCT-based insertion method [24,25]. Meanwhile, through parallel acceleration technology, the computational speed of the DCT-based insertion method can be guaranteed. Therefore, a DCT-based method together with parallel acceleration is employed in this work to insert the watermark.

In this method, the original image is first divided into N × N blocks. DCT is then conducted on each block in parallel to obtain the distribution in the frequency domain, which can be expressed as:

$$E_{j,k}\left(m_1,n_1\right)=\mathrm{dct}2\left[I_{j,k}\left(m_1,n_1\right)\right]$$

(1)

where I_j,_k is the intensity distribution of the (jth, kth) block, E_j,k is the frequency distribution of the (jth, kth) block, dct2 is the symbol of 2D-DCT, j and k are positive integers that do not exceed N, and m₁ and n₁ are positive integers that do not exceed M/N.

To realize the watermark insertion, the first pixel of E_j,k is replaced by the (jth, kth) pixel of the watermark, which can be expressed as:

$$E{\prime }_{j,k}\left(m_1,n_1\right)=\left\{\begin{array}{c}{E}_{j,k}\left(m_1,n_1\right)\times \left[\beta +\alpha I_{wm}\left(j,k\right)\right],\left(m_1,n_1\right)=\left(1,1\right)\\ E_{j,k}\left(m_1,n_1\right),\left(m_1,n_1\right)\ne \left(1,1\right)\end{array}\right.$$

(2)

where I_wm is the intensity distribution of the watermark, E’_j,k is the frequency distribution of the (jth, kth) block after the replacement of the pixels, and α and β are two scale factors. Finally, the watermarked image in the spatial domain can be recovered by the IDCT, which can be expressed as:

$$I{\prime }_{j,k}\left(m_1,n_1\right)=\mathrm{idct}2\left[E{\prime }_{j,k}\left(m_1,n_1\right)\right]$$

(3)

where I’_j,_k is the intensity distribution of the (jth, kth) block after the insertion of the watermark, and idct2 is the symbol of 2D inverse DCT (IDCT).

2.2. Calculation of the CGH

In conventional one-step computer-generated holography, the holographic reconstruction is always significantly affected by the noise. The irregularly distributed noise in the reconstructed image might disturb the frequency distribution obtained by DCT which makes the watermark difficult to extract. Therefore, a precise holographic algorithm is necessary to encode a watermarked image into a CGH. The iterative method that uses multiple forward and backward wavefront propagations with some constraints is often employed to encode a CGH with less noise [26,27,28]. Therefore, it is more suitable than the conventional one-step method to encode the CGH for a watermarked image.

In this work, an iterative method called IASM is employed, as shown in Figure 3. The IASM is formed via a forward propagation model and a backward propagation model. In the forward propagation model, the input amplitude is of uniform intensity, while the input phase is obtained from the backward propagation model. Considering that there is no calculation result from the backward propagation model during the first forward propagation, a random phase is employed as the initial input phase. The complex distribution on the holographic plane synthesized by the input amplitude and input phase is then processed through the ASM, which can be expressed as:

$${\tilde{D}}_{\mathrm{obj}, p}\left(m,n\right)=\left\{\begin{array}{c}\mathrm{ifft}2\left\{\mathrm{fft}2\left\{\exp \left[\mathrm{i}\times 2\mathsf{\pi }\times ran\left(m,n\right)\right]\right\}\times \exp \left[\mathrm{i}\frac{2\pi }{\lambda }{z}_0\sqrt{1-{\left(\lambda u_m\right)}^2-{\left(\lambda v_n\right)}^2}\right]\right\}, p=1\\ \mathrm{ifft}2\left\{\mathrm{fft}2\left\{\exp \left[\mathrm{i}\times \arg \left({\tilde{C}}_{\mathrm{holo},p-1}\left(m,n\right)\right)\right]\right\}\times \exp \left[\mathrm{i}\frac{2\pi }{\lambda }{z}_0\sqrt{1-{\left(\lambda u_m\right)}^2-{\left(\lambda v_n\right)}^2}\right]\right\}, p>1\end{array}\right.$$

(4)

where $\tilde{D}$_obj is the complex distribution on the object plane, $\tilde{C}$_holo is the complex distribution on the object plane, i is the symbol of the imaginary number, arg is the symbol of extracting the phase from the complex amplitude, p is the number of iterations, λ is the wavelength of the illumination wave, z₀ is the propagation distance, ran is a 2D real random matrix, fft2 and ifft2 are symbols of 2D fast Fourier transform (FFT) and 2D inverse FFT, and u_m and v_n are the discrete spatial frequencies that can be expressed as:

$$u_m=-\frac{1}{2\Delta }+\frac{m}{M\Delta },v_n=-\frac{1}{2\Delta }+\frac{n}{M\Delta }$$

(5)

where Δ is the pixel pitch of the 2D matrix.

In the backward propagation model, the input amplitude is the watermarked image, while the input phase is obtained from the forward propagation model. The complex distribution on the object plane synthesized via the input amplitude and input phase is then processed by the inverse ASM, which can be expressed as:

$$\begin{array}{c}{\tilde{C}}_{\mathrm{holo},p}\left(m,n\right)=\mathrm{ifft}2\left\{\mathrm{fft}2\left\{I\prime \left(m,n\right)\times \exp \left[\mathrm{i}\times \arg \left({\tilde{D}}_{\mathrm{obj},p}\left(m,n\right)\right)\right]\right\}\right.\\ \left.\times \exp \left[-\mathrm{i}\times \frac{2\mathsf{\pi }}{\lambda }{z}_0\sqrt{1-{\left(\lambda u_m\right)}^2-{\left(\lambda v_n\right)}^2}\right]\right\}\end{array}$$

(6)

where I’ is the entire watermarked image, which can be spliced by each block obtained from Equation (3). The optimized complex distribution can be obtained through repeating the forward and backward propagations described by Equations (5) and (6) until the number of iterations reaches a preset value p. The phase of the optimized complex distribution is finally extracted as the CGH for the watermarked image.

2.3. Encryption of the CGH

To enhance the confidentiality of the transmission, the obtained phase-only CGH should be encrypted. The process of encryption is shown in Figure 4. Firstly, a database for secret keys is established. Each secret key in the database is essentially a pseudo-random phase. To encrypt the phase-only CGH, a secret key located in row a and column b is selected. Then, the selected secret key is multiplied with the obtained CGH, generating an encrypted phase-only CGH, which can be expressed as:

$$H_{\mathrm{encrypted}}\left(m,n\right)=\arg \left[{\tilde{C}}_{\mathrm{holo},P}\left(m,n\right)\right]\ast S_{a,b}\left(m,n\right)$$

(7)

where $\tilde{C}$_holo,p is the complex distribution after p iterations, S_a,b is the secret key located in row a and column b of the database, and * is the symbol of the element-wise product. To avoid the value of the matrix element in H_encrypted being modulated to 0 by the secret key, the value of the matrix element in S_a,b is never equal to 0. Additionally, to mark the location of the selected secret key, the values of a and b are also encoded into the encrypted CGH, which can be expressed as:

$$H{\prime }_{\mathrm{encrypted}}\left(m,n\right)=\left\{\begin{array}{c}\exp \left[\mathrm{i}\times \left(2\mathsf{\pi }\times a/2^{D_{\mathrm{bit}}}-\mathsf{\pi }\right)\right],\left(m,n\right)=\left(1,1\right)\\ \exp \left[\mathrm{i}\times \left(2\mathsf{\pi }\times b/2^{D_{\mathrm{bit}}}-\mathsf{\pi }\right)\right],\left(m,n\right)=\left(M,M\right)\\ H_{\mathrm{encrypted}}\left(m,n\right),\mathrm{others}\end{array}\right.$$

(8)

where D_bit is the bit depth of the encrypted CGH. In this work, the employed image always has a bit depth of 8 bits. Therefore, the maximal values of a and b can reach 256. This means that a database containing up to 256 × 256 secret keys can be established.

2.4. Reconstruction of the CGH

After being encrypted and encoded, the obtained phase distribution is transferred from the signal side to the display side using a transmitter and a receiver. A database that is totally the same as that in Figure 4 is also stored on the display side. To recover the phase-only CGH of the watermarked image, the location of the secret key that is employed to encrypt the CGH should be first searched, which can be expressed as:

$$I{\prime }_{\mathrm{encrypted}}\left(m,n\right)=\frac{2^{D_{\mathrm{bit}}}\times \left\{\arg \left[H{\prime }_{\mathrm{encrypted}}\left(m,n\right)\right]+\mathsf{\pi }\right\}}{2\mathsf{\pi }}$$

(9)

$$a=I{\prime }_{\mathrm{encrypted}}\left(1,1\right), b=I{\prime }_{\mathrm{encrypted}}\left(M,M\right)$$

(10)

where I’_encrypted is the normalized distribution extracted from H’_encrypted. By the coordinate value (a, b), the selected secret key S_a,b can be easily found, being employed to decrypt the CGH, which can be expressed as:

$$H_{\mathrm{unencypted}}\left(m,n\right)=H_{\mathrm{encrypted}}\left(m,n\right)./S_{a,b}\left(m,n\right)$$

(11)

where / is the symbol of the element-wise division process. The decrypted CGH is then uploaded onto a phase-only CGH, optically reconstructing the watermarked image under the illumination of a coherent plane wave. Finally, the watermark inserted in the image can be extracted from the reconstructed image.

The extraction of the watermark is shown in Figure 5. The reconstructed watermarked image is divided into N × N blocks again. DCT is conducted on each block of the reconstructed image in parallel to obtain the distribution in the frequency domain, which can be expressed as:

$$G_{j,k}\left(m_1,n_1\right)=\mathrm{dct}2\left[R_{j,k}\left(m_1,n_1\right)\right]$$

(12)

where R_j,_k is the (jth, kth) block of the reconstructed image and G_j,k is the frequency distribution of the (jth, kth) block. To realize the watermark extraction, the first pixel of G_j,k is picked and processed, which can be expressed as:

$$I{\prime }_{wm}\left(j,k\right)=\left[G_{j,k}\left(1,1\right)/E_{j,k}\left(1,1\right)-\beta \right]/\alpha$$

(13)

where I’_wm is the extracted watermark. According to the information in the watermark, the source of the leak can easily be traced.

3. Results

3.1. Numerical Experiments

To verify the effectiveness of the proposed watermarking and encryption for holographic communication, a series of simulations were conducted in this work. The images with a resolution of 2000 × 2000 pixels are employed as the original images, as shown in Figure 6a. A binary image shown in Figure 1a with a resolution of 250 × 250 pixels is employed as the watermark. The values of α, β, λ, z₀, Δ, and p are set as 0.02, 1, 532 nm, 150 mm, 3.74 μm, and 50, respectively. The database of the secret keys is formed by four pseudo-random phases.

The watermarked images obtained via the method described in Section 2.1 are shown in Figure 6b. It can be observed from the results that the invisibility of the watermark is well-satisfied by employing the mentioned parameters. In addition, the time consumption of inserting the watermark into an original image can be reduced to 0.19 s by using parallel computation, presenting prominent applicability in holographic communication. When the encrypted CGHs are employed in the holographic reconstruction, the reconstructed images are shown in Figure 6c. Obviously, when the secret keys are not used to decrypt CGHs, the reconstructed images cannot be recognized. If the scale of the secret key database is large enough, it would be more difficult to recover the distributions of secret keys, making it significantly hard to decrypt the CGHs without permission. The reconstructed images by using decrypted CGHs are shown in Figure 6d. By applying IASM in the calculation of CGHs, displayed images with shape details and suppressed noises are readily obtained. To extract watermarks from the displayed images, the method mentioned in Section 2.4 is applied. The extracted watermarks are shown in Figure 6e. It can be seen from the results that if the encrypted signal is decrypted by non-target users, the source of the leak can also be easy to trace.

3.2. Optical Experiments

To further verify the effectiveness of the proposed watermarking and encryption method, a practical holographic communication system was established. The transmitter and receiver used in the system are a Xiaomi CR6609 router and a TP-Link AC1200 router, respectively. After receiving the encrypted CGHs, a notebook (Xiaomi Notebook Pro 15) is employed to process these CGHs and upload them onto an SLM. The SLM is placed in a holographic display system, which is shown in Figure 7a. A laser with a wavelength of 532 nm is employed as the illumination of the system. The illumination beam passes through an attenuator, a pinhole, and a polarizer and then is collimated with a convex lens. The collimated beam illuminates the SLM, being modulated by the CGH uploaded on it. Finally, the reconstructed wave is reflected via a beam splitter and is captured with a camera (Canon 60D).

The reconstructions of encrypted CGHs and decrypted CGHs are shown in Figure 7b,c, respectively. To promote the image quality of holographic reconstructions, the look-up table (LUT) of the SLM was corrected in advance [29]. Like the results in numerical experiments, the reconstructions can only be recognized when CGHs have been decrypted. As shown in Figure 7d, by applying the watermark extraction method described in Section 2.4, the inserted watermark can be readily recovered, making it easier to trace the source of the leak when the displayed images are captured without permission. Therefore, the watermarking and encryption method is functional in optical experiments. Through this method, the signal can be encrypted and watermarked in holographic communication, which can be used to protect the privacy and copyright in future applications.

4. Discussion

4.1. Effect of Scale Factors

As mentioned in Equation (2), α and β are two scale factors during the watermark insertion. When the value of α remains unchanged (α = 0.02), the watermarked images under different values of β are shown in Figure 8a. It can be seen from the results that the brightness of the watermarked image increases with the value of β. When the value of β is closer to 1, the brightness of the watermarked image is closer to the original image. When the decrypted CGHs are used, the reconstructions of watermarked images under different values of β are shown in Figure 8b. Correspondingly, the watermarks extracted from these reconstructions are shown in Figure 8c. Obviously, the image quality of the extracted watermark gradually decreases with the value of β.

To quantitatively evaluate the image quality of holographic reconstructions and the extractability of watermarks, the peak signal-to-noise ratio (PSNR) was employed. As shown in Figure 8b,c, the PSNR of the holographic reconstruction reaches its maximal value when the value of β is 1, while the PSNR of the extracted watermark decreases with the value of β. By adjusting the value of β, the image quality of holographic reconstructions and the extractability of the watermarks can be satisfied simultaneously.

In addition to the effect of β, the influence caused by the value of α is also explored. When the value of β remains unchanged (β = 1), the watermarked images under different values of α are shown in Figure 9a. It can be seen from the results that the visibility of the watermark increases with the value of α. When the decrypted CGHs are used, the reconstructions of watermarked images under different values of α are shown in Figure 9b. Correspondingly, the watermarks extracted from these reconstructions are shown in Figure 9c. The PSNR of the holographic reconstruction decreases with the value of α by a small degree, which means that the image quality of the holographic reconstruction deteriorates slowly. However, the visibility of the watermark increases dramatically. Considering the results of Figure 8 and Figure 9 together, it can be concluded that the image quality of the extracted watermark improves with the value of α/β. Although there is no limit on α and β, setting the values of α and β as 0.02 and 1 might be a reasonable choice when both the visibility and extractability of the watermark are considered.

4.2. Effect of Iteration-Based Optimization

As mentioned in Section 2.2, the holographic reconstruction is often affected by the noise in conventional one-step computer-generated holography. Being influenced by the noise, the image quality of the holographic display is significantly limited, while the extraction of the watermark from the reconstruction is difficult. To address this issue, the IASM is employed. The reconstructions and extracted watermarks under different iteration numbers p are shown in Figure 10. It can be seen from the results that speckle noise in the reconstruction by a one-step CGH is plainly observed. The watermark extracted from the reconstruction is extremely fuzzy. With the increase in the iteration number p, the speckle noise can be suppressed. The clarity of the extracted watermark is improved. However, when the iteration number is larger than 20, the density of speckles in the reconstruction and the clarity of the extracted watermark see little improvement with the increase in the iteration number. This trend can also be seen in the change in the PSNR value. Considering the consumption of the computing power, 20 iterations might be an optional choice in most cases.

4.3. Robustness

To measure the robustness of the proposed watermarking and encryption method, some typical noises, including Gaussian noise, speckle noise, salt and pepper noise, and occlusion, are introduced into the encrypted CGHs. The CGHs under different introduced noises are shown in Figure 11a. When the encrypted CGHs are decrypted and reconstructed, holographic reconstructions and their locally enlarged results are shown in Figure 11b,c, respectively. Correspondingly, the watermarks extracted from these reconstructions are shown in Figure 11d.

It can be seen from the results that the image quality of holographic reconstructions is obviously influenced by the introduced noises. The decline in the PSNR caused by the Gaussian noise or speckle noise is more severe than that caused by salt and pepper noise or occlusion. However, even if the image quality of holographic reconstructions is obviously influenced by these introduced noises, the inserted watermarks can still be easily extracted from the reconstructed images. Therefore, the proposed watermarking and encryption method shows its reliability in holographic communication.

5. Conclusions

In this work, a watermarking and encryption method for holographic communication is proposed. In this method, the original image to be presented is first watermarked via the DCT-based insertion strategy. The precise CGH of the watermarked image is then calculated by the IASM. To encrypt the obtained CGH, a secret key selected from the database is employed. The location of the secret key is also encoded into the encrypted CGH. The time consumption of inserting the watermark into the original image can be reduced to 0.19 s by using parallel computation, presenting a prominent applicability in holographic communication. The reconstruction of the encrypted CGH can only be recognized when the correct secret key is employed. If the scale of the secret key database is large enough, it would be difficult to recover the distributions of secret keys, making it significantly hard to decrypt the CGHs without permission. Even if the encrypted signal is decrypted by non-target users, the source of the leak can also be easy to trace by extracting the inserted watermark from the holographic reconstruction.

The proposed watermarking and encryption method provides an optional solution for the privacy protection and copyright protection of holographic communication. In our future work, the concealment and robustness of the watermark will be improved; the effectiveness of the watermark insertion for full-color 3D images will be studied; the acceleration strategy for precise phase-only CGHs will be explored; and a secret key that is more difficult to decrypt will be designed.