Design of Streaming Data Transmission Using Rolling Shutter Camera-Based Optical Camera Communications

Abstract

This paper addresses a method to transmit streaming data via rolling shutter camera-based optical camera communications (OCC). Since the amount of data that can be contained within one frame is limited, and the continuity of received data cannot be guaranteed due to OCC environmental variations, we introduce the concept of dividing the streaming data into several fragmented sets that are transmitted sequentially. We propose a superframe to contain sequential packets of fragmented data and corresponding indices, so that sequential packets for streaming data can be continuously collected. When redundant frame transmission is considered, any packet lost due to OCC environmental conditions can be recovered. Experimental results show that the proposed method can be successfully used to transmit streaming data, with the number of redundant frames required to acquire all data packets based on image resolution. In addition, we describe how to identify missing packets from a network point of view to reduce the number of redundant frames needed to acquire all the data. This paper presents baseline results of communication performance when sending streaming data via a rolling shutter-based OCC.

1. Introduction

Recently, a lot of research related to optical camera communications (OCC), which is a subset of visible light communications (VLC), has been published by industry and academia [1,2,3,4,5]. A typical digital camera consisting of a lens and a two-dimensional image sensor is capable of receiving optical data and is more beneficial than a photo detector (PD) by providing spatial separation of different light sources from the optical lens. Thanks to widespread LED light sources and digital cameras, OCC technology can to be applied in various fields, such as vehicle-to-everything (V2X) [3], indoor communication and positioning [4,5], artificial reality (AR), and display field communications [6]. In addition, OCC can be used in intelligent services, such as unmanned factory automation and security systems in combination with artificial intelligence (AI) technology, and can be considered a key technology for implementing massive machine-type communications (mMTC), which will provide connections to large numbers of devices for 5G networks and beyond in the near future.

Among the non-flicker modulation series of OCC [7], the rolling shutter-based method [8,9,10,11] focuses on scenarios where the frame rate of a complementary metal oxide semiconductor (CMOS) image sensor is slower than the flickering speed of the light source that transmits the data. By turning an LED on and off frequently enough, data modulation is not perceived by the human eye. At the camera receiver, the information sent via the visible light spectrum appears in the form of a barcode on the captured image, and the patterns of the barcode contain the transmitted data. Recently, Teli et al. [12] carried out an experimental investigation of an indoor OCC system by employing a multi-channel transmitter and a Raspberry Pi camera as the Rx for IoT applications. Nguyen et al. [13] proposed a rolling shutter-based OFDM OCC system with multiple LEDs to improve the data rate performance. Eso et al. [14] increases the link span of a rolling shutter-based OCC system up to 400 m by reducing the spatial bandwidth of the camera. Liu et al. [15] introduces existing and new solutions to deal with the constraints of the OCC link with regard to data modulation, newly developed camera structures, and post-processing of sensed signals.

However, the rolling shutter-based OCC method inherently has two important constraints. First, for the receiving camera to fully utilize the rolling shutter effect, the distance between the transmitter and the receiver end must be short, and the pixel area of the light source must be sufficiently large in the received image. Second, when a continuous stream of data is sent, it is challenging for the receiver to acquire the streamed data as a single received image. In other words, the amount of continuous data sent by the transmitter and expressed in one image frame as a rolling shutter pattern is strictly limited [5]. Although streaming data delivery requires continuous reception, some missing data exist between temporally adjacent images, which results in distortions of the received information. Therefore, the application of transmitting stream data via a rolling shutter OCC link should consider how to utilize a continuous data reception mechanism. To the best of our knowledge, a solution for recovering all continuous data using rolling shutter-based OCC has not been studied in the literature.

This paper addresses the design of a streaming data communication mechanism using rolling shutter-based OCC. To support the continuity of the received data, we propose a superframe structure consisting of sequential packets of consecutive fragmented data and corresponding indices. To enhance the capability to recover lost packets, duplicate frames are transmitted, similar to the repetition coding scheme used in conventional communication systems. Therefore, even though part of the segmented packets may be lost or damaged while receiving a rolling shutter image, all the continuous data can be recovered through reception packet recombination using an index and duplicate superframes. The experimental results show that a data-streaming service can be provided through an OCC link with the help of the proposed superframe structure and its duplicate transmission mechanism. We presented the number of required duplicates based on the image resolution and the optical rate of the image. We believe this method provides a solution, allowing users to access smart content immediately using an OCC link, and the experimental guidelines can be used to design transmission and reception processes in implementing rolling shutter-based streaming data communication.

2. Superframe Structure for Streaming Data Communication

The system model for implementing the proposed streaming data transmission is an OCC downlink. When continuous data are transmitted by an LED light source in the form of visible light, they are broadcast to each OCC receiver equipped with a camera. Considering the CMOS image sensor at the receiving end, the rolling shutter effect appears as an on/off pattern and is analyzed to demodulate the signal. Due to the nature of rolling shutter-based OCC, the amount of data obtained in one received image contains a portion of the streaming data. Therefore, if the packet size of streaming data is too large to be received in one image, data must be aggregated using multiple images. A simple combination of consecutive images does not construct all the stream data owing to the OCC environmental characteristics, such as readout time delay, delay time variation between frames, etc. Therefore, in order to receive and collect all streaming data using a rolling shutter camera, transmitted data need to be fragmented and transmitted in small units.

Figure 1 presents the proposed superframe structure consisting of small consecutive packets. Note that the term superframe is used in different purposes in the 802.15.7 standard [16] and in this paper. While superframes in the standard are defined based on media access control (MAC) protocols such as contention access period (CAP), contention free period (CFP), etc., the superframe used in this paper is designed entirely for streaming data transmission by introducing indexing concept of fragmented packets. Each packet, X_i, has an associated preamble and the X_i-th fragmented data. The preamble contains synchronization bits and a packet index for X_i. Transmitting fragmented packet index is very important for recognizing the missing packets during the transmission of streaming data in OCC and make it possible to recover the missing packets from redundant (retransmitted) frames.

Figure 2 illustrates the process of data fragmentation and indexing. The original streaming data with k bits are fragmented into N small data units, b_gn, where n is the allocated bits per group, and N is the number of groups (or packets), such that 1 ≤ n ≤ N. Then, the indexing process combines the parts for timing synchronization, and the index indicates the sequence number of the corresponding packets. By combining fragmented data, b_gn, and an indexing packet, all N data packets are constructed. Then, consecutive data packets constitute a superframe, and data packets are sequentially transmitted via the LED. By doing this, it is possible to find a packet that gets lost or damaged in the middle of streaming data reception via an OCC link.

Figure 3 shows how to distinguish lost data packets when sending indexed data packets in superframe units. It is assumed that six packets can be captured in one image and received periodically according to the frame rate of the camera. If we assume that the streaming data to be transmitted comprises 20 packets, the index of lost packets forming the first superframe includes 7, 8, 9, 16, 17, and 18. After receiving the second duplicate superframe, the remaining lost packets are 7 and 16. When several duplicate superframes can be received consecutively, it is possible to recover all packets to construct the original superframe.

Due to interference or a noisy optical wireless channel and the capturing mechanism of the receiving camera, packets are lost with a certain probability. The proposed superframe structure is partitioned into multiple packets, and enables appropriate action to recover a loss. Using the packet index included in a data packet, the receiver can successfully recover lost packets by reconnecting to a streaming data set. In addition, the proposed scheme allows the OCC receiver to start capturing images from the transmitting LED starting at any time and smartly decoding them to recover all the streaming data, even in the presence of jitter in the frame rate [2].

3. Data Detection

Because the optical rate of an LED is much faster than the shutter speed of an image sensor, the LED changes its state multiple times during the image capture time. Therefore, the on/off states of the LED source are captured and shown as rolling shutter patterns in an image. Then, the transmitted data can be retrieved via the image processing and data decoding technique. Depending on the modulation scheme, one data bit can be presented by one or multiple light states. To mitigate the flicker effect of the LED transmitter, we use Manchester encoding for data modulation.

Although the commercial camera is assumed to operate at a fixed frame rate, such as 30 fps, unpredicted variations in the frame rate exist in practical operation. In between two neighboring image frames, the LED still blinks, but the camera cannot record it. Consequently, a data packet might be lost from the streaming data.

From a received image containing a pulse profile, the rolling shutter patterns can be converted to a binary array from which useful information can be extracted. As shown in Figure 4, the on/off patterns in the rolling shutter image present transmitted pulse profiles. The duration of the on/off patterns is presented as the pixel width of pulses, and is related to the optical rate of the transmitting LED and the camera capture rate. By adjusting these values, an array of images containing on/off patterns of different durations and intensities can be obtained. Using a predetermined threshold, pulse profiles are binarized into binary sequences.

As shown in Figure 5, each on/off pattern of binary sequences is composed of several pixel values. Because the camera cannot capture the transition strictly, a single pulse often occupies different pixel numbers. However, by measuring the average pixel width of on/off patterns, binary sequences are converted into pulse sequences. Considering Manchester encoding on the transmitter side, we can demodulate the bit data from the pulse sequence by comparing two consecutive pulse states. Additionally, it is also possible to demodulate data from binary sequences by using on/off transition patterns of bit sequences [5]. After the demodulation procedure is completed for each rolling shutter image, all demodulated data are merged to form the data as a whole. Then, if there are packets with lost data, we recover the transmitted streaming data by following the procedure explained in Section 2.

4. Simulation Results and Discussion

4.1. Simulation Results

A feasibility test of the streaming data transmission scheme was conducted in an indoor environment. In order to investigate the performance according to the length of the fragmented packet size, we use the streaming data of length 26 (for short data case) consisting of ASCII alphabet characters from A to Z. In addition, we also use the length 52 streaming data (for long data cases), including both lower- and upper-case letters. Based on the results of actual experiments in these two cases, the relationship between the number of packets and the number of additional transmissions can be verified. By transmitting streaming data composed of 52 and 26 fragmented data packets and by capturing a rolling shutter image, we measured the number of frames required to demodulate all the fragmented data. For the OCC-based experiment, a commercial LED luminary with an Atmega8A microcontroller was used to transmit visible light data at a 10 kHz optical rate. A 30-fps camera in a smartphone was used as a data receiver, and the sizes for captured images were set to 3264 × 2448 and 1600 × 1200. The images were taken at line-of-sight (LOS) distances, centered on the LED lights. In OCC, the Tx-Rx distance affects the resolution of the received image frame. Moreover, if we use the ‘zoom’ function of the camera, this affects the resolution of the image as well. This means that it is reasonable to consider the image resolution instead of the Tx-Rx link distance in evaluating the performance of OCC. To minimize the light blur effect in the rolling shutter image, the camera ISO, which amplifies or decreases the image sensors’ ability to read and gather light during an exposure, was set to 800. The fragmented data packets consisted of 20 bits with synchronization bits (6 bits), index bits (6 bits), and data bits (8 bits).

Table 1. Received streaming data consisting of 52 fragmented packets (sequential transmission).

Resolution	List	Results
3264 × 2448	Avg. Packets/Frame	8.78 (Max: 9, Min: 8)
	Avg. Pulses/Frame	389.59
	Avg. Bits/Frame	193.99
	Req. No. of Frame	12 Frame
1600 × 1200	Avg. Packets/Frame	5.58 (Max: 6, Min: 5)
	Avg. Pulses/Frame	262.98
	Avg. Bits/Frame	130.61
	Req. No. of Frame	11 Frame

shows the experimental results of receiving 52 fragmented data packets obtained from a rolling shutter image. Generally, the larger the number of pixels in the vertical direction, the greater the number of data packets included in one image frame. When we had a resolution of 3264 × 2448, we got 8.78 packets and 193.99 bits per image, whereas we got 5.58 packets and 130.61 bits at 1600 × 1200. The number of image frames required to receive all 52 pieces of streaming data when sequentially transmitting 52 pieces of fragmented data was 12 frames for 3264 × 2488 and 11 frames for 1600 × 1200.

Table 2. Received streaming data consisting of 26 fragmented packets (sequential transmission).

Resolution	List	Results
3264 × 2448	Avg. Packets/Frame	8.76 (Max: 9, Min: 8)
	Avg. Pulses/Frame	389.52
	Avg. Bits/Frame	193.92
	Req. No. of Frame	5 Frame
1600 × 1200	Avg. Packets/Frame	5.61 (Max: 6, Min: 5)
	Avg. Pulses/Frame	263.03
	Avg. Bits/Frame	130.67
	Req. No. of Frame	7 Frame

shows the results when sending 26 fragmented data packets, considering short streaming data. Since the total number of packets in the streaming data was halved, all the data were received within seven consecutive frames. From the experiments, we can observe that the number of redundant frame transmissions required to receive the entire streaming data is not significantly related to the camera resolution. Therefore, the transmitting end can design a duplicate frame transmission method without seriously considering the performance and specifications of the camera at the receiving end.

Figure 6 presents the average number of missing packets based on image resolution and the length of fragmented packets in the streaming data. When the number of packets was relatively large and a high-resolution image was used, we can see that the number of missing packets was drastically reduced in the consecutive received frames. However, if the number of packets was not large, the number of missing packets was consistently reduced, regardless of the image size. In addition, we can find that the average number of missing packets decreases as the number of redundant (or retransmitting) frames increases, and is saturated after enough redundant frames are transmitted. Note that this tendency is consistent regardless of image resolution and packet size. At the beginning, when redundant transmission is attempted, the probability of including the missing packet in the redundant frame gradually decreases. If more than a certain number of packets are received, the probability of including the missing packets in the redundant frame is low, so the reduction slope becomes gentle.

Note that the main purpose of this paper is to propose how to recover the missing packets of streaming data caused by frame-to-frame discontinuity property of OCC and investigate the feasibility of the proposed scheme. Therefore, data rate analysis according to the specifications of the transmitter and receiver of OCC are reserved for a future task.

4.2. Discussions

As shown in Figure 3 and Table 1 and Table 2, it is necessary to transmit duplicate frames to recover missing packets so all the data can be demodulated. Because duplicate frames should be transmitted, it is inefficient in terms of transmission rate. This problem can be solved efficiently by adopting the automatic repeat request (ARQ) concept. As shown in Figure 7: (1) an acknowledge signal containing a request to turn the LED on is transmitted from the receiver to the transmitter; (2) the transmitter begins sending information on ready status to communicate using the LED; (3) an acknowledge signal containing a request to send packets is transmitted from the receiver to the transmitter; (4) the transmitter begins sending streaming data and, afterwards, the index of lost packets is determined at the receiver; (5) the index information is sent to the transmitter; (6) the transmitter collects only the lost packets and transmits them. Note that this procedure significantly reduces the number of additional frames required to obtain complete information. For example, considering the rolling shutter image at 3264 × 2448 in Table 1 and Table 2 above, this ARQ approach requires, at most, two additional frames to demodulate all data. Therefore, this scheme ensures reliable data transmission while reducing the additional transmission overhead required to recover the lost packets.

5. Conclusions

This paper proposed a method to solve the packet loss problem when transmitting streaming data in rolling shutter-based OCC systems. Owing to OCC-based parameters such as the optical rate of the light source, the frame rate of the receiver, and camera performance, a single received frame does not contain all the streaming data because some of the data between the neighboring frames can be missed. By proposing a superframe structure consisting of index information and fragmented data packets, all data can be obtained from consecutive frames. Through experiments, we demonstrated the transmission of duplicate superframes that are required to obtain all the streaming data, based on image resolution and the optical rate. In addition, we suggested an ARQ type approach that can solve the packet loss problem efficiently with fewer duplicate frames.

The main purpose of this paper is to propose how to recover the missing packets of streaming data caused by the frame-to-frame discontinuity property of OCC and investigate the feasibility of the proposed scheme. Therefore, an analysis of the maximum data rate regarding the high-bandwidth LED transmitter and high-resolution camera receiver deviates from the scope of this paper. Data rate analysis, according to the specifications of the transmitter and receiver of OCC, can be considered as a future task. In addition, we will perform an evaluation of the overall network performance of the proposed scheme, considering the ARQ approach in transmitting the missing packets, in a future work.