Visual-Feedback-Based Frame-by-Frame Synchronization for 3000 fps Projector–Camera Visual Light Communication ^†

Abstract

This paper proposes a novel method for synchronizing a high frame-rate (HFR) camera with an HFR projector, using a visual feedback-based synchronization algorithm for streaming video sequences in real time on a visible-light communication (VLC)-based system. The frame rates of the camera and projector are equal, and their phases are synchronized. A visual feedback-based synchronization algorithm is used to mitigate the complexities and stabilization issues of wire-based triggering for long-distance systems. The HFR projector projects a binary pattern modulated at 3000 fps. The HFR camera system operates at 3000 fps, which can capture and generate a delay signal to be given to the next camera clock cycle so that it matches the phase of the HFR projector. To test the synchronization performance, we used an HFR projector–camera-based VLC system in which the proposed synchronization algorithm provides maximum bandwidth utilization for the high-throughput transmission ability of the system and reduces data redundancy efficiently. The transmitter of the VLC system encodes the input video sequence into gray code, which is projected via high-definition multimedia interface streaming in the form of binary images 590 × 1060. At the receiver, a monochrome HFR camera can simultaneously capture and decode 12-bit 512 × 512 images in real time and reconstruct a color video sequence at 60 fps. The efficiency of the visual feedback-based synchronization algorithm is evaluated by streaming offline and live video sequences, using a VLC system with single and dual projectors, providing a multiple-projector-based system. The results show that the 3000 fps camera was successfully synchronized with a 3000 fps single-projector and a 1500 fps dual-projector system. It was confirmed that the synchronization algorithm can also be applied to VLC systems, autonomous vehicles, and surveillance applications.

1. Introduction

Due to the increasing demand for radio-frequency-based wireless communications, an alternative communication system using light as a source has emerged, known as visible-light communication (VLC) [1,2,3]. A light source with frequencies ranging between 400 and 800 THz (750 and 375 nm, respectively) is used by VLC systems for encoded transmissions. A photodiode detector is used to detect the modulated visible light and decode it to extract information. With the use of semiconductor light-emitting diodes (LEDs) as a light source that can modulate light at a high speed that is imperceptible to human vision, a VLC system provides the functionality of both communication and room luminescence [4,5]. VLC systems in indoor scenarios, such as homes, offices, or hospital facilities, can provide high-speed internet in a short distance with little interference from other light sources. If a camera is used as a receiver, the most prominent application of an indoor VLC system is visible light positioning (VLP), which combines image processing and data transmission. Image processing is used to recognize the geometry of the environment and monitor mobile nodes [6,7,8]. With the development of image sensors for the mobile camera, they can be used as a potential receiver in a VLC system [9,10,11]. Various LED-based VLC applications include intelligent automotive systems [12,13], indoor navigation systems [14], survey measurement systems [15,16], robot control systems [17,18,19] and LED camera-based VLC [20,21]. The projector-based VLC (PVLC) was studied for imperceptible data projection along with the image [22,23]. Various image-sensor-based VLC systems were developed to decode information transmitted from light sources (e.g., light-emitting diodes (LEDs), display screens, and projectors). The screen intensities of display monitors and LCD panels were modulated using various encoding techniques, and a receiving camera was used to decode the information [24,25]. In a screen-camera-based VLC system, the data shared on the screen can be hidden from view and do not necessarily depend on what is displayed. However, display screens are limited in their encoding functionality. Conversely, the bandwidth limitations of the display-camera-based VLC system can be overcome by using a high frame-rate (HFR) projector and an HFR camera system, which increases the overall bandwidth of the system. The major issue with this system is synchronization, which limits its application.

With the recent development of high-speed complementary metal-oxide-semiconductor (CMOS) image sensors, high frame-rate image processing is highly effective in recognizing a wide variety of high-speed phenomena in the real world. Various HFR vision systems that can process images at hundreds or thousands of frames per second (fps) were developed [26,27,28,29,30]. Many tracking algorithms (e.g., feature-point tracking [31], cam-shift tracking [32], and optical flow estimation [33,34]) were accelerated by using field-programmable gate arrays (FPGAs) and graphics processing units. Therefore, image sensors are widely used in VLC systems as efficient receivers. Additionally, with the increasing demand for projector-based applications, such as structured, light-based, three-dimensional (3D) sensing [35,36,37,38], projection mapping [39,40,41,42], and interactive applications for augmented reality, there have been extensive developments in projection technology. There are now HFR projector-based systems that can project binary image patterns at thousands of fps or faster, using digital micro-mirror devices (DMDs) [43,44], proving to be the best candidate for a state-of-the-art VLC source.

Because camera synchronization is crucial to some applications, there are some in which the interpolation and prediction of the frames of unsynchronized cameras are used to synchronize the camera [45]. Alleviation of synchronization between the projector and camera can be achieved by using wired external triggers; however, in some applications, there is a need for synchronization without an external trigger, such as in a VLC system. Many industrial cameras are equipped with external trigger-signal ports in which the long wires may cause unstable synchronization, owing to the distance between the projector and camera, whereas short wires may obtrude the constraints of spatial camera configurations. To mitigate these problems, some wireless synchronization applications have been used to display pattern sequences. Correspondingly, the camera achieves synchronization by adjusting its exposure time [46]. There have been studies in which geometric points were used for synchronization, but these require a sufficient number of corresponding points across images to carry out simultaneous geometric calibration and synchronization [47,48,49,50]. In some studies [51,52], existing wired standard buses (e.g., IEEE1394 and Ethernet) were used to send trigger signals to the camera for synchronization. Time synchronization is another method that uses Wi-Fi with which the master subsystem provides connectivity and records video, while the slave subsystem provides low-power event detection via ZigBee connectivity [53]. In this system, a Wi-Fi mesh network is used to transmit video data, and a ZigBee network is used to define the network topology and synchronize multiple surveillance-camera systems. In [54], synchronization was achieved in two stages. During the first stage, the phones were synchronized to the clock of a leader device, using the network time protocol (NTP); during the second stage, all client phone cameras captured a continuous stream that was phase-shifted to achieve better accuracy. The prominent drawback of wireless time synchronization is the nondeterminism of media access time [55]. Another study involved illumination-based intensity-modulated synchronization, using a 1000-fps camera with a phase-locked loop algorithm [56]. This algorithm is robust to background light and locks-in a high camera frame rate to the LED by adjusting the gain parameter to fit the brightness. However, all systems involving camera clock controls using visual feedback require an LED as the light source, or they must be controlled using a wire triggering device or NTP. The wired synchronization system has higher accuracy, but it is unsuitable for an optical wireless communication system.

This paper is an extended version of our previous journal paper [57] in which we proposed a real-time VLC video-streaming system, using an HFR projector and camera. In this paper, we propose a method for synchronizing the HFR camera and projector to resolve inconsistent estimates. This method efficiently reduces data redundancy, mitigates long-term inconsistency issues, and increases the overall bandwidth of the system. A visual feedback-based control algorithm is used to synchronize the HFR camera with the HFR projector by providing a delay to the camera clock to match the phase of the projector’s frame rate. The algorithm generates a delay using the total brightness of an image captured by the HFR camera when a predefined pattern of white and black image sequences is projected. A detailed explanation of the operation of the visual feedback-based synchronization algorithm is described in Section 2. The real-time video reconstruction using a VLC system is explained in Section 3. Section 4 provides an evaluation of the performance of the synchronized system via experiments in real time at 60 fps, with the HFR projector and camera both working at 3000 fps. Finally, in Section 5, the conclusions are drawn.

2. Visual Feedback-Based Projector-Camera Synchronization

2.1. System Configuration

An HFR projector that can control its projection rate is required for synchronization with an HFR camera for high-speed VLC communication. The DLP LightCrafter 4500 (Texas Instruments, Dallas, TX, USA) projector provides a 4000 fps frame rate with bit-plane projection, using 1- to 8-bit projection, supporting a resolution of 912 × 1140. The DLP LightCrafter 4500 is a DMD projection system, where a pixel is represented with a two-dimensional array of electrically addressable and mechanically tiltable micro-mirrors, which are widely used in consumer electronics [58,59,60]. The DLP projector reproduces the signal by modulating the exposure time of the mirrors over a specific operating refresh time based on the projected frame-bit planes. This feature helps with projecting data at the pixel level, and it transforms the image to a pixel-wise bit-plane projection for the VLC system.

The projected images are captured by an extended version of the monochrome FASTCAM SA-X2 (Photron, Tokyo, Japan) HFR camera, which enables 10,000 fps (real-time) image processing of a megapixel image, using a global electronic shutter with excellent light sensitivity [29]. Figure 1 presents an overview of the FASTCAM SA-X2 with the embedded external board, which was designed to control the delay signal for the camera clock so that it outputs images having a resolution of 512 × 512 with a 12-bit dynamic range at 3125 fps in real time. For an external trigger source, a function generator, AFG1022 (Tektronix, Beaverton, OR, USA), was used to provide a 3000 Hz square-wave trigger at 3.3 V.

2.2. Visual-Feedback-Based Projector–Camera Synchronization

The synchronization method used in the previous paper consists of software-based synchronization. In software synchronization, the HFR camera frame rate is three times the HFR projector frame rate. For example, if the HFR projector frame rate is 1000 fps, then the HFR camera frame rate is 3000 fps. The HFR camera captures three images of a single 1-bit projected image, out of which only the second image with better image brightness, compared to the other two images, is selected. The second image is sometimes not in phase with the HFR projector, which results in the variation of brightness of an image. As a result, we observe inconsistent estimates, data redundancy, and that the overall bandwidth of the system decreases. To overcome the above limitations of software-based synchronization, phase synchronization of HFR camera with HFR projector is required, which can be achieved with the proposed visual feedback-based projector–camera synchronization system as shown in Figure 2. The HFR projector is triggered using an external trigger 1 source (function generator 1), whereas the HFR camera is triggered using another external trigger 2 source (function generator 2). A predefined binary pattern is projected and captured by the HFR camera to calculate the maximum brightness and generate a delay, using a proportional control-based algorithm. This delay is added to the trigger signal of the HFR camera clock to match the HFR camera and projector phases. As a result, we attain maximum and minimum brightness by adding a delay to the camera clock of the HFR camera system.

To achieve this, the control logic of the timing controller was implemented on an FPGA on the external board attached to the HFR camera, as shown in Figure 3. The sync timing controller accepts the input delay value $\tau$, calculated using the visual-feedback-based algorithm in the software. The value $\tau$ is then fed to the sync signal generator as ${\tau }_1$ with limits between ${\tau }_{min}$ and ${\tau }_{max}$. The sync signal generator accepts the input clock frequency f from the external trigger source, and then adds a delay ${\tau }_1$ to the input clock to generate the output clock signal as $SyncOut$. This delayed $SyncOut$ clock signal is given as a $SyncIn$ signal to the HFR camera for triggering image capture, thereby matching the phase of the HFR camera with the HFR projector.

To calculate the delay value, $\tau$, for synchronization, the projector projects a predefined binary pattern of ’10101010’ continuously, as shown in Figure 4, at a frame rate of $F_p$ with phase $P_p$, while the camera captures individual frames at a frame rate of $F_c$ with phase $P_c$. The white and black images are represented by 1 for maximum brightness and 0 for minimum brightness, respectively. The ideal case where the HFR camera is synchronized with the HFR projector is shown in Figure 4, where the phase $P_c$ is synchronized with phase $P_p$.

There are three cases of phase difference as shown in Figure 5. In the first two cases, the phase of the HFR camera is out of phase with the HFR projector (i.e., case-1 and case-2). In case-1, there is a delay in the camera trigger, compared with the projector, whereas in case-2, the camera is triggered before the projector. Thus, projector phase $P_p$ is not equal to camera phase $P_c$. In case-3, the camera and projector are both triggered simultaneously, which is the desired result after synchronization.

Therefore, case-1 and case-2 can be synchronized, and the HFR camera can be triggered with the same phase as that of the HFR projector by adding a delay of $\tau$ to $F_c$, as shown in Figure 6. As with case-1 in Figure 6a, there is a delay in triggering $F_c$; therefore, to match $P_c$ with $P_p$, we must generate a large delay, $\tau$. In case-2, we require a small delay, $\tau$, to match $P_c$ with $P_p$ as $F_c$ is triggered prior to the HFR projector, as shown in Figure 6b.

To calculate the value of $\tau$, we must initially calculate the brightness-based index, $R(k)$, which is the ratio of the total brightness of two images at consecutive frames, where $S(k)$ is the total brightness sum of the input image at the current frame, k, and $S(k-1)$ is the brightness at the previous frame, $k-1$.

$$R(k)={\displaystyle \frac{S(k)}{S(k-1)}},$$

(1)

$$R(k-1)={\displaystyle \frac{S(k-2)}{S(k-1)}}.$$

(2)

Next, we evaluate the error, C(k), where $R(k-1)$ is the previous brightness-based index calculated, using Equation (2).

$$C(k)=R(k)-R(k-1).$$

(3)

The delay, $\tau$, is calculated for proportional control by considering the delay, $\tau (k-1)$, at the previous frame and error, $C(k)$, multiplied by a constant proportional gain, $K_{d_p}$. The value of $\tau$ lies between ${\tau }_{max}$ and ${\tau }_{min}$; ${\tau }_{max}$ is set to maximum exposure duration of the camera, and ${\tau }_{min}$ is zero.

$$\tau (k)=\tau (k-1)+K_{d_p}·C(k){\tau }_{max}>\tau \ge {\tau }_{min}.$$

(4)

2.3. Verification

Considering the above-mentioned synchronization algorithm, experiments were conducted to verify its performance at a high frame rate. The HFR projector alternately projected bit-plane images of black and white patterns through a 24-bit color image. The value of each bit plane image (black or white) is decided by the 8-bit pixel value of a single channel, which was set to decimal 170 (10101010 in binary). The HFR projector frame rate was set to 3000 fps, and the exposure of each frame was set to a maximum exposure time of 331 μs. The HFR camera was set to 3000 fps; the exposure time of each frame was set to 1/3015 s (331 μs) to capture the black and white patterns alternately. Because the visual feedback control algorithm depends on the total brightness of an image, the black and white patterns were determined by the maximum and minimum total brightness of the captured image. The output graph of synchronization is shown in Figure 7, where the zoomed portion shows a pattern that the total image brightness drops after every 24 bits, due to the projection of a blank image by the HFR projector between two images. Figure 7 shows that the HFR camera phase, $P_c$, was initially not in sync with the HFR projector phase, $P_p$, and due to this, the blending of projected black and white patterns resulted in inaccurate total brightness of the captured images. Therefore, during the initial duration of approximately 6 s, the total brightness information was inaccurate. After this, the synchronization algorithm was executed, and the value of delay, $\tau (k)$, was calculated using $S(k)$, $R(k)$, and $C(k)$. The delay, $\tau (k)$, was given as input to ${\tau }_1$ of the sync timing controller on the FPGA of the external board, which generated a $SyncOut$ signal to trigger the HFR camera and match the phase ($P_c$) with the phase ($P_p$) of HFR projector. The proportional gain, $K_{d_p}$, was set to 0.01 to achieve synchronization in shorter duration with stability. Figure 7 shows that the calculated delay, $\tau (k)$, was 0.177 ms, and the HFR camera–projector system was synchronized in approximately 20 ms. As a result, in the synchronized system, the total brightness of the captured image from the HFR camera increased when a white image was projected, and decreased when a black image was projected.

HFR camera exposure times of 1/3015, 1/8000, and 1/12,500 were selected to evaluate the performance of our proposed algorithm, which can work with a very short exposure time of 1/12,500 and large exposure time of 1/3015, as shown in Figure 8. The projector frame rate was set to 3000 fps with the exposure time 331 μs, and the bit-plane images of the black and white patterns were projected alternately. The HFR camera frame rate was set to 3000 fps, and the exposure time was kept at 1/3015, 1/8000, and 1/12,500 under a constant room luminescence of 150 lx. Initially, the HFR camera and projector were not in sync for approximately 3 s, after which the synchronization algorithm was executed to calculate the delay, $\tau (k)$, depending on the total brightness of the image. Figure 8 shows that the HFR camera–projector was synchronized under different frame exposure times with different levels of total image brightness. The value of delay $\tau (k)$ was never constant for a particular frame exposure time, and it varied depending on either case-1 and case-2 as discussed above. From this experiment, we can deduce that the system works with a wider range of exposure times and provides system flexibility. Figure 8 shows that synchronization is achievable for all selected ranges, but the only difference is the total brightness of the image.

3. Real-Time Video Streaming Using Vlc System

In our previous study, we used real-time video streaming with the VLC system [57], which comprised an HFR projector (DLP LightCrafter 4500), HFR camera (monochrome Fastcam SA-X2 with an additional external board with FPGA), and personal computer (PC) as shown in Figure 9 with its concept. Figure 10 presents a block diagram of the transmitter and receiver section in detail. The transmitter section consists of a gray-coded color video sequence and header information that is fed to the HFR projector for bit-plane binary projection. The receiver section consists of a monochrome HFR camera that captures monochrome images sequentially, from which background subtraction is performed to achieve better thresholding of the bit plane images by combining them. The combined color image is a gray code image, which is further decoded to pure binary code to reconstruct the original image. With reference to our previous work, we added additional information to the header and introduced a method of background subtraction for each image.

3.1. Transmitter

The transmitter encoding system involves three stages, as shown in Figure 11, where the input image, $I_t(x,y)$, is first encoded from pure binary into gray code as $I_{gra{y}_t}(x,y)$, to which header information, $I_h(w,y)$, is added. Then, the encoded image, $I_{gra{y}_{rgb}}(m,n)$, as shown in Equation (5), is fed to the HFR projector for bit-plane projection.

$$I_h(w,y)+I_{gra{y}_t}(x,y)=I_{gra{y}_{rgb}}(m,n),$$

(5)

The header information is added to inform the receiver that it received the transmitted information, which consists of five blocks of pixels representing information about the current image as shown in Figure 12. The first block, S0, contains all pixel values set to a maximum value of 255 for an 8-bit pixel, which is used to determine the start of a new image. The next five blocks of pixels (i.e., F4, F3, F2, F1, and F0) in the header represent a 5-bit frame number, ranging from 0 to 31. Subsequently, two blocks of pixels contain 2-bit channel information C1 and C0 blocks to represent the red–green–blue (RGB) channels of an image. The next three blocks of pixels, represented as B2, B1, and B0), contain 3-bit information of 8-bit planes of a single channel. The C1, C0, B2, B1, and B0 blocks of pixels aid in determining the sequence of binary images for reconstruction, whereas the last block of pixel I0 represents the stream of video, webcam, or video sequence from two different PCs as input streams.

After combining the gray-code image with the respective header information, the image is fed to the HFR projector, where the spatio-temporal projection of an HFR projector is achieved by decomposing a given packed 24-bit $I_{gra{y}_{rgb}}(m,n)$ image into its equivalent twenty four 1-bit binary images. The projection pattern used in this system should have the total duration of exposure for all patterns less than or equal to the vsync duration in which a blank sequence is introduced to complete the vsync exposure time.

3.2. Receiver

The projected bit-plane images were captured by a monochrome HFR camera to reconstruct the transmitted 24-bit RGB image. The operation of the receiver section is shown in detail in Figure 13, where the captured images, $C_{in}(u,v)$, were collected sequentially according to the header information and reconstructed into a gray-code image, $C_{RGB}(u,v)$, which was converted back to pure-binary code, $I_{RGB}(u,v)$, to retrieve the transmitted image, $I_t(x,y)$.

The HFR camera was synchronized with an HFR projector, using a visual-feedback-based control algorithm. To retrieve the bit information accurately for each projected bit-plane, a background subtraction method was used to eliminate the noise introduced by ambient light on the projector screen in an indoor office environment. The thresholding method was used for background subtraction, where the reference image was subtracted from the input image. The reference image was estimated using the global thresholding method by projecting the maximum and minimum intensities through the HFR projector onto the screen. The threshold value, $thr(m,n)$, at $(m,n)$ was calculated using Equation (6), where $B(m,n)$ is the pixel value at $(m,n)$ of $C_{in}(u,v)$, captured after projecting its maximum brightness, and $D(m,n)$ is the pixel value at $(m,n)$ of $C_{in}(u,v)$, captured after projecting a black image.

$$thr(m,n)=\frac{B(m,n)}{2}+D(m,n),$$

(6)

The robustness offered by the background subtraction method is discussed in our previous work. We also introduced an additional background subtraction method in which the reference image is updated for each channel to maximize the efficiency of background subtraction. The bit plane decomposition of an 8-bit image is represented as eight 1-bit binary images with higher bit planes, containing more significant visual information, and the lower bit-plane containing more details. However, the intensity of the lowest bit hardly changes; therefore, we used the lowest bit of each channel to update the reference image, as shown in Figure 14.

Let the 1-bit image of the projected bit-plane image captured by the HFR camera be $C_{in}(u,v)$, and the reconstructed 8-bit image of three channels be combined to form a single 24-bit RGB color image, $C_{RGB}(u,v)$. The $C_{RGB}(u,v)$ image is an encoded gray-code image that is further decoded to a pure-binary-code-based image at the pixel level to obtain the reconstructed RGB color image, $I_{RGB}(u,v)$.

3.3. Evaluation Parameter of Image Quality

To assess the image quality of the reconstructed images, a full-reference metrics-based objective image quality index, such as the peak signal-to-noise ratio (PSNR) and multi-scale structural similarity index (MS-SSIM) [61] were used. PSNR compares images using different dynamic ranges, as expressed in Equation (7), where MSE is the mean-squared error, and $MA{X}_I$ is the dynamic range of the allowable pixel intensities. The value of the PSNR should be higher for better image quality.

$$PSNR=10·{\log }_{10}\left({\displaystyle \frac{MA{X}_I^2}{MSE}}\right).$$

(7)

The MS-SSIM, as shown in Equation (8), extracts the structural information from the field of view based on the human visual system (HVS) assumption. However, it is not very useful for blurred images, and the measured error lies between zero and one, where one represents the best image quality.

$$MS-SSIM(x,y)={\left[l_M(x,y)\right]}^{{\alpha }_M}·\prod _{j=1}^M{\left[c_j(x,y)\right]}^{{\beta }_j}·{\left[s_j(x,y)\right]}^{{\gamma }_j}.$$

(8)

To evaluate the efficiency of the reconstructed images, a 5-bit frame number in the header was used by assigning the frame number to each input frame ranging from 1 to 32, thereby creating a packet of 32 frames. The loss is calculated by counting the missing frames within the 32 frames at the receiver. The efficiency of the system was calculated using Equation (9), where $F_r$ is the frame reconstruction efficiency, and $S_r$ represents the successful frame reconstructed out of the total number of frames, $F_t$, within one packet of 32 frames. Thus, image quality metrics evaluate the quality of the reconstructed images at the receiver. Additionally, the frame reconstruction efficiency explains the number of frames being reconstructed at the receiver and those being lost because of the bandwidth of the system and luminescence of the HFR projector.

$$F_r[\%]={\displaystyle \frac{S_r}{F_t}}\times 100.$$

(9)

4. Experiments

To evaluate the performance of the system, we performed various experiments by streaming the saved video and real-time universal serial bus (USB) camera video and reconstructing it, using the VLC system. The HFR projector streamed 590 × 1080 video at 60 fps, which is a combination of 590 × 1060 gray-code images and 590 × 20 header information. This combined image is projected in a bit plane sequence as shown in Figure 15, where the duration of exposure for each bit-plane pattern is 331 μs. Therefore, the time required to project one frame is 24-bit × 331 μs, which is approximately 8000 μs or 8 ms. Therefore, in 1 s, approximately 125 frames can be projected using our system; however, owing to the limitation of the HFR projector, we can transmit 590 × 1080 images at maximum 60 fps. A 50-mm lens was mounted on the HFR camera, which was set to the same frame rate as that of the HFR projector (3000 fps). The experimental setup is shown in Figure 16, where the distance between the HFR projector and screen was 950 mm. The projection display onto the screen was 448 × 415 mm. The distance between the HFR camera and screen was 1130 mm to ensure that the overall area of the projected video on the screen was captured by the camera. For the proposed system, a stored video sequence was used for a single projector system, and live video streaming from two USB cameras was used to check the synchronization accuracy of the dual projector system. To check the robustness of the system, the indoor environment was illuminated with luminescence values of 0, 150, and 300 lx, using an external light source. The details of the experimental hardware are specified in Figure 17.

4.1. Synchronized Real-Time Video Reconstruction

To evaluate the operation of the VLC system after successful synchronization using the visual feedback algorithm, an experiment was performed in which a stored video sequence was streamed in real time. The selected video sequence was from the movie, Big Buck Bunny [62]. Initially, the pure-binary-code images of the 24-bit 1920 × 1080 RGB-color video sequence were gray coded and resized to 590 × 1060 alongside the addition of the 590 × 20 header information. The encoded resized image was projected in bit-plane or binary image sequences at 3000 fps, and the HFR camera captured 512 × 512 images to reconstruct the output image with a resolution of 510 × 459 by combining all bit planes of the 24-bit RGB image sequentially. Figure 18a shows a high definition input image of 1920 × 1080 at 60 fps. Figure 18b contains the reconstructed images 510 × 459, using gray code without background subtraction. Figure 18c depicts the reconstructed images 510 × 459, using gray code with background subtraction. From the reconstructed images, we can deduce that there were no artifacts present when gray-code encoding was used.

Next, an experiment was conducted to measure the image quality analysis and performance of the system by sending the saved video at 60 fps, which was projected at 3000 fps under different on-screen luminescence conditions of 0, 150, and 300 lx; images were captured at 3000 fps at different exposure times, i.e., 1/3015, 1/8000, and 1/12,500. The results of the image-quality analysis of hundreds of reconstructed images with respect to their original images, are shown in Figure 19 and Figure 20. From the graph shown in Figure 19 and Figure 20, the PSNR and MS-SSIM values indicate that the image quality was better when captured at an exposure of 1/8000 than those at 1/12,500 and 1/3015. However, the background subtraction performed at every frame helped improve the image quality at various HFR camera exposure times. Figure 21 shows the performance of the system based on the number of frames reconstructed at the receiver. A duration of 40 s was observed, and the number of frames was monitored, as shown in Figure 21. The frame reconstruction ratios were almost 100% for 0 lx, whereas for 150 and 300 lx, the frame reconstruction was nearly 100% with small losses. The experimental results indicate that the HFR projector and HFR cameras were synchronized; otherwise, the frame reconstruction would not be possible, and we would not be able to reconstruct the video sequence in real time at 60 fps.

4.2. Real-Time Video Reconstruction Using Two HFR Projectors

The experimental setup for the two projectors is shown in Figure 22, where the dual projectors are maintained such that the projection area overlaps. The distance between the screen and both HFR projectors was kept the same at 950 mm, and the HFR camera was set at a distance of 1130 mm with a 50-mm mounted lens. The experiment scene is shown in Figure 22. In this experiment, the input video sequence was streamed from two USB cameras (XIMEA, MQ003CG-CM) in 24-bit color with a resolution of 640 × 480 at 60 fps for transmission, and both cameras were connected to two PCs. The experiment scene comprised a person throwing a football on the floor. In this experiment, HFR projector 2 was set to 180° out of phase with respect to HFR projector 1, and both were set to 1500 fps. The bit-plane projection of the image sequence was the same as that in Figure 15. The HFR camera was kept at 3000 fps, that is, double the projection frame rate, which captured the images of each projector alternately and reconstructed both videos at 60 fps. In Figure 23, the two HFR projector input image sequences of 640 × 480 at 60 fps are shown. Figure 23b depicts the reconstructed 510 × 459 images, using gray code without background subtraction. Figure 23c shows the reconstructed images 510 × 459, using gray code with background subtraction. The sequences were reconstructed alternately from the image sequence projected, using HFR projectors 1 and 2.

The image quality analysis and performance of the two projector systems were evaluated by projecting a 60 fps video at 3000 fps under different on-screen luminescence conditions, i.e., 0, 150, and 300 lx, and the images were captured at 3000 fps with different exposure times, i.e., 1/3015, 1/8000, and 1/12,500. Figure 24, Figure 25, Figure 26 and Figure 27 show that the values of PSNRs and MS-SSIMs were similar, but the best result was observed for exposure 1/3015 at 0 lux. However, the values of MS-SSIM are more promising than the PSNR values because they represent the image quality of the system. Figure 28 shows the performance of the frames reconstructed from each projector system, reflecting the number of frames reconstructed at the receiver corresponding to each projector. Figure 28 shows that the live USB streaming led to few losses in the frame during reconstruction but were not very significant, and the system could reconstruct video sequence in real-time at 60 fps. The results indicate that multiple projector synchronization was possible with the HFR camera, and the overall bandwidth of the system was utilized.

5. Conclusions

In this article, we presented a novel HFR projector–camera synchronization algorithm using a visual feedback algorithm and evaluated its performance by streaming real-time video using the HFR projector–camera-based VLC system. The experimental results show that synchronization can be achieved at a high frame rate, and that the system is robust to ambient light and can work on a wide range of exposure times. The background subtraction method increased the image quality of the reconstructed image under different ambient light conditions. The experiments were conducted for real-time video streaming to evaluate the percentage of frames received at different frames-per-second and luxes, and it was observed that the frame loss was slightly increased with an increase in the frame rate and lux. The HFR camera and HFR projector system bandwidth were not fully utilized at 3000 fps when a single projector system was used because the system could reconstruct a 60 fps streaming video at nearly 60 fps. Therefore, the dual projector system proved promising, and a full bandwidth of approximately 120 fps was utilized as the dual projector system distributed the computational load of one PC to two. Overall, the images reconstructed using the dual projector had better quality, and the system can be expanded to multiple projectors. The only constraint of the dual projector system is that HFR projector 2 should be triggered by HFR projector 1.