Rolling Shutter-Based Underwater Optical Camera Communication (UWOCC) with Side Glow Optical Fiber (SGOF)

Abstract

Nowadays, a variety of underwater activities, such as underwater surveillance, marine monitoring, etc., are becoming crucial worldwide. Underwater sensors and autonomous underwater vehicles (AUVs) are widely adopted for underwater exploration. Underwater communication via radio frequency (RF) or acoustic wave suffers high transmission loss and limited bandwidth. In this work, we present and demonstrate a rolling shutter (RS)-based underwater optical camera communication (UWOCC) system utilizing a long short-term memory neural network (LSTM-NN) with side glow optical fiber (SGOF). SGOF is made of poly-methyl methacrylate (PMMA) SGOF. It is lightweight and flexibly bendable. Most importantly, SGOF is water resistant; hence, it can be installed in an underwater environment to provide 360° “omni-directional” uniform radial light emission around its circumference. This large FOV can fascinate the optical detection in underwater turbulent environments. The proposed LSTM-NN has the time-memorizing characteristics to enhance UWOCC signal decoding. The proposed LSTM-NN is also compared with other decoding methods in the literature, such as the PPB-NN. The experimental results demonstrated that the proposed LSTM-NN outperforms the PPB-NN in the UWOCC system. A data rate of 2.7 kbit/s can be achieved in UWOCC, satisfying the pre-forward error correction (FEC) condition (i.e., bit error rate, BER ≤ 3.8 × 10⁻³). We also found that thin fiber also allows performing spatial multiplexing to enhance transmission capacity.

1. Introduction

The demand for faster and more reliable internet access is growing continuously, and the capacity of the existing radio frequency (RF) communication spectrum has nearly reached its limits. Hence, researchers are exploring complementary or alternative solutions to provide high-speed wireless communication. This exploration led to the utilization of visible light communication (VLC) [1,2,3,4,5,6], which operates using a wavelength range from 380 nm to 750 nm as the carriers to carry the data. VLC belongs to the family of optical wireless communication (OWC) for providing promising and reliable wireless communication links in both outdoor and indoor environments [7,8,9]. VLC offers many advantages, such as providing an additional communication spectrum, immunity to electromagnetic interference (EMI), high transmission security and privacy, and no license requirement. Additionally, VLC can be cost and energy efficient by combining communication and illumination at the same time. One implementation of VLC, known as light fidelity (LiFi), not only provides communication and illumination but also supports movable users [10]. Furthermore, VLC is also regarded as one of the promising candidates for the future 6G mobile communication [11].

In addition to providing terrestrial transmissions, VLC can also provide underwater transmission [12,13,14]. Nowadays, a variety of underwater activities, such as underwater surveillance, marine monitoring, etc., are becoming crucial worldwide. Underwater sensors and autonomous underwater vehicles (AUVs) are widely adopted for underwater exploration. Although underwater communication can be realized using RF and acoustic waves [15,16,17], VLC could provide transmission merits. As shown in [18], visible light has the lowest absorption coefficient in water when compared with that in infrared (IR) and microwave. For example, visible light has about three orders of magnitude lower absorption coefficients in water than the 1.55 μm IR light used in traditional fiber optics communication. Additionally, visible light has about four orders of magnitude lower absorption coefficients than microwave signals.

Refs. [19,20] provide good reviews for the recent progress and the state of the art of underwater VLC. Recently, 3.075 Gbit/s and 1.2 m underwater VLC transmission were achieved using a blue light-emitting diode (LED) and PIN photodiode (PD) [21]. Additionally, 14.81 Gbit/s and 1.2 m VLC transmission were reported using five-color RGBYC LED and PIN PD [22]. Apart from using LED, a laser diode (LD) can also be used for underwater VLC. A 12.62 Gbit/s and 35 m underwater VLC transmission were demonstrated using 450 nm LD and PIN PD [23]. Additionally, 30 Gbit/s and 12.5 m underwater VLC transmission were achieved using 488 nm LD and avalanche photodiode (APD) [24]. Furthermore, in order to significantly enhance the receiver sensitivity, underwater VLC utilizing a single-photon detector (SPD) [25] or a single-photon avalanche diode (SPAD) [26] was also demonstrated. Apart from underwater communication, underwater sensing and localization have also attracted much attention recently [27,28].

One use of VLC or OWC is to employ a camera and image sensor to receive the optical signal, which is known as optical camera communication (OCC) [29,30]. Nowadays, nearly all mobile phones are embedded with a complementary metal oxide semiconductor (CMOS) camera; hence, the OCC deployment cost could be low since there is no need to have significant hardware modification on the VLC receiver (Rx) side. As discussed in [30], OCC systems can be classified into region-of-interest (ROI) and rolling shutter (RS) operations. The ROI-based OCC systems usually use high frame rates and high-resolution cameras to receive the optical signal. Spatial multiplexing can be utilized to receive different light sources simultaneously to enhance the data rate. Popular ROI-based OCC schemes include under-sampled frequency shift on–off keying (UFSOOK) modulation [31,32] and under-sampled phase shift on–off keying (UPSOOK) modulation [33,34]. The ROI-based OCC systems can support much longer transmission distances than in the RS-based OCC systems; however, they may need a heavy computational burden to trace and detect multiple light sources. Additionally, the data rate is usually much less than that in RS-based OCC systems. On the other hand, the RS-based OCC systems usually use a typical commercially available CMOS image sensor or camera, with a frame rate of about 30 or 60 frames per second (fps), and a resolution of 1920 × 1080 pixels. In the RS operation, the pixel rows in the CMOS camera are activated row by row. At the optical transmitter (Tx) side, when it is modulated faster than the frame rate of the camera, dark and bright fringes standing for the OCC signal “logic 0” and “logic 1” can be visualized in the image sensor RS pattern. Moreover, in order to enhance the applicability of OCC systems, both the Tx and Rx sides should have a wide field of view (FOV). Recently, wide FOV optical Txs based on light-diffusing fiber (LDF) or side glow optical fiber (SGOF) were proposed [35,36,37,38]. They can provide 360° uniform light emission around the LDF circumference. This means that the LDF can act as an “omni-directional optical antenna”. Moreover, it is lightweight and flexibly bendable. Previously, the LDF or SGOF could be installed in an unmanned aerial vehicle (UAV) to provide VLC [35]. Furthermore, the LDF or SGOF can also support sliding and multiple Rxs to obtain a VLC signal along the fiber [38]. Most importantly, SGOF is water resistant; hence, it can be installed in an underwater environment to provide “omni-directional” optical signals. Efficient RS thresholding methods are also required to decode the RS pattern correctly. Recently, effective algorithms have been utilized to enhance the performance of OCC, including the convolutional neural network (CNN), the pixel-per-bit labeling neural network (PPB-NN), the long short-term memory neural network (LSTM-NN), transition-based data decoding, etc. [39,40,41,42,43,44,45].

In this work, we present and demonstrate an RS-based underwater optical camera communication (UWOCC) system utilizing the LSTM-NN with SGOF. The optical Tx is a commercially available poly-methyl methacrylate (PMMA) SGOF. As discussed before, SGOF is lightweight and flexibly bendable. Most importantly, SGOF is water resistant; hence, it could be suitable for underwater communication. SGOF is a kind of polymer optical fiber (POF), which is popular for short-distance data communication, decoration, and sensing applications. Instead of sending an optical signal from one end of the fiber to the other end via total internal reflection (TIR), SGOF allows light to leak via the fiber cladding. The main challenge for SGOF is how to emit homogeneous light over the length of the fiber in all directions around its circumference. The SGOF used in the experiment is made of PMMA and is commercially available. Its radial side-emitting mechanism is based on bulk scattering [46]. Radial light emission (or light leakage) is achieved by doping the POF with scattering sites realized using different particles. By carefully controlling the size and concentration of the introduced particles, homogeneous radial light emission can be achieved. Different light-scattering particles, such as zinc oxide (ZnO) and titanium dioxide (TiO₂) [47], can be utilized. The size of these particles can vary from 50 to 2000 nm. In order to produce Rayleigh scattering, the scattering particle diameter should be smaller than the wavelength of the incident light [48]. In addition, in order to produce Mie scattering, the scattering particle diameter is equal to or larger than the wavelength of the incident light [48]. It can be installed in an underwater environment to provide 360° “omni-directional” uniform light emission around its circumference. This large FOV can fascinate the optical detection in underwater turbulent environments. The proposed LSTM-NN has the time-memorizing characteristics to enhance the OCC signal decoding. The proposed LSTM-NN is also compared with other decoding methods in the literature, such as the PPB-NN. The experimental results demonstrated that the proposed LSTM-NN outperforms the PPS-NN in the UWOCC system. A data rate of 2.7 kbit/s can be achieved in UWOCC, satisfying the pre-forward error correction (FEC) condition (i.e., bit error rate, BER ≤ 3.8 × 10⁻³). We also found that thin fiber also allows spatial multiplexing to enhance the transmission capacity.

The main contributions of this work are summarized as follows:

• Underwater VLC systems reported in the studies usually employ LD or LED Txs. Here, we employ SGOF Tx, which is lightweight, flexibly bendable, and water resistant. It can also provide 360° “omni-directional” uniform light emission around its circumference.
• Underwater VLC systems reported in the studies usually employ PD or APD Rxs. Here, we employ a CMOS image sensor-based OCC, which allows reliable and larger FOV detection.
• In this demonstration, both SGOF Tx and the CMOS image sensor are immersed in water to achieve more accurate experimental results.
• Here, RS decoding is realized by the proposed LSTM-NN, which has the time-memorizing characteristics to enhance the decoding of the OCC signal.
• Parallel SGOF Txs are also studied to achieve spatial multiplexing, enhancing the total transmission capacity.

2. Experimental Setup

The experiment setup of the proposed RS-based UWOC system with PMMA SGOF is shown in Figure 1. The on–off keying (OOK) signal at different data rates is produced by a low-speed arbitrary waveform generator (AWG, Tektronix^® AFG3252C, Beaverton, OR, USA). The AWG has a sampling rate of 2 GS/s and an analog bandwidth of 240 MHz. It is used to modulate a blue LD directly via an LD driver. The blue LD has an output optical power of 600 mW and a wavelength of 445 nm. Then, the blue OOK optical signal is launched into the PMMA SGOF, which is located inside a water tank with dimensions of 0.3 m × 0.73 m × 0.88 m. The PMMA SGOF has a diameter of 3 mm and a length of 2 m. The OCC Rx is a mobile phone (ASUS^® Zenfone3, Taiwan, China) with a CMOS camera with a 30 fps frame rate and a 1920 × 1440 pixel resolution. We select a shutter speed of 1/10,000 s and an ISO of 1600 in the experiment. As this mobile phone is not water resistant, it is put inside a waterproof transparent plastic bag, and then the Rx is placed inside the water tank. The Tx includes the PMMA SGOF and the blue LD module. The blue LD module consists of a blue LD with an output power of 600 mW and an LD driver. The driver DC voltage is 12 V, the modulation mode is analog, and the maximum modulation frequency is 10 kHz. An optical collimator is installed in front of the LD to produce the optical beam divergence < 3 mrad.

In order to increase the applicability of our proposed UWOCC system, we use off-the-shelf components and devices. Hence, a typical mobile-phone-embedded CMOS image sensor can be used. The SGOF is also an off-the-shelf product. There is no special requirement about the dimensions of the water tank, as long as the SGOF and the mobile phone can be immersed for the experiment. The use of a 445 nm wavelength blue LD is because it has low attenuation underwater, as shown in [18].

Here, the PMMA SGOF is utilized as the optical Tx. It is robust against bending and stretching. It is mainly used for signage lighting, such as advertising light boxes, swimming pools, tunnels, etc.

Table 1. Parameters of the SGOF.

Parameter	Value
Diameter	3 mm
Density	1.15 g/cm3
Length	2 m
Refractive index	1.39
Melting point	130–140 °C

shows the parameters of the SGOF used in the experiment. Figure 2a,b show, respectively, the photographs of the PMMA SGOF and the proof-of-concept experiment, illustrating the PMMA SGOF emitting blue light homogeneously around its circumference. It can act as a wide 360° large FOV “omni-directional optical Tx”. This large FOV can fascinate the optical detection in turbulent water environments. In the proof-of-concept demonstration, the distance between the mobile phone and the PMMA SGOF is about 20 cm.

The OCC Rx is a CMOS Image Sensor (Sony^® IMX298, Tokyo, Japan) embedded in a mobile phone (ASUS^® Zenfone3).

Table 2. Parameters of the CMOS image sensor.

Parameter	Feature
Resolution	1920 × 1440
Shutter type	Rolling shutter
Optical format	1/2.8″ optical format
Pixel size	1.12 μm × 1.12 μm
Frame rate	30 fps
Shutter speed	1/10,000 s

shows the parameters of the CMOS image sensor used in the experiment.

In the RS-based OCC system reported in this work, when the transmission distance between the SGOF and the camera increases, the occupied area of the SGOF image captured in the CMOS image sensor becomes smaller. As the RS is produced at the CMOS camera, the RS column pixel width is unchanged when the transmission distance increases. Hence, when the transmission distance increases, a complete OCC packet cannot be observed in an image frame, as illustrated in [32], and the BER measurement cannot be performed. In addition to the transmission distance limitation, UWOCC performance could also be affected by water clarity and water turbulence. This is because dirty water will block the light and reduce the received signal-to-noise ratio (SNR), while wavy water could distort the received RS pattern and increase the BER. We believe that it could be employed in real underwater conditions if the received SNR is enough.

3. AI/ML Algorithm

We now discuss the RS decoding algorithm. Figure 3a shows the RS pattern decoding flow diagram for the PMMA SGOF-based UWOCC system. First, different image frames obtained in the UWOCC system are divided into train images and test images, respectively. Then, these image frames are launched into the “image processing module”, as shown in Figure 3b. This module is used in both the training and testing processes. We will discuss the “image processing module” in detail later in this section. Then, the processed images will be input into the ML models for training and testing, respectively. It is worth noting that the images are different for the model training and testing. Finally, BER will be calculated based on ML model prediction.

We will now discuss the “image processing module”. The first step is the grayscale conversion. Because each pixel obtained in the CMOS image sensor consists of three colors, red, green, and blue, the images will be converted to grayscale format based on the equation grayscale value = 0.2989 × Red + 0.587 × Green + 0.114 × Blue. As shown in Figure 4, grayscale 0 means dark and 255 means bright. We can observe two RS patterns in Figure 4, in which the lower one is the actual SGOF fiber image and the upper one is generated by the reflection of the water tank. We only need one fiber image to decode the signal. It is worth mentioning that we convert the rolling shutter pattern into grayscale values (i.e., a grayscale value of 0 means complete darkness and a grayscale value of 255 means complete brightness) to facilitate the subsequent signal processing, such as signal enhancement (i.e., enhancing the grayscale value), and thresholding (i.e., identify logic 1 or logic 0) in this work since only blue light is utilized and modulated. We believe that only extracting the blue light from the signal processing could be good enough.

The image will be turned counterclockwise 90 degrees, and the actual SGOF RS pattern will be selected. There are several columns containing signals, so these columns will be added up, as shown in Figure 5. After this process, these columns will be averaged. The new column is the average column vector, as illustrated in Figure 6. The second step in Figure 3b is to find the average column vector. Then, we can locate the two headers from the average column vector; the OCC signal between headers is the data payload. After obtaining the payload, we can perform the last step in Figure 3b, which is the pixel-per-bit (PPB) calculation and re-sampling. This process is performed to ensure that each bit contains the same number of pixels. This means that if a label of the neural network is composed of several pixels, the number of pixels is the PPB. For instance, if the payload length is 340 pixels and the input signal is 30 bits, the payload will be re-sampled to 360 pixels (i.e., it is divisible by 30), and the PPB is equal to 12. The twelve pixels are used as one label for the model.

Figure 7a,b show the proposed architecture of the LSTM-NN and the LSTM cell, respectively. The first three layers are the LSTM layers, and the operation mechanism will be described later in this section. They have LSTM cell numbers of 30, 20, and 10, respectively. They are followed by another three fully connected (FC) dense layers with neural numbers of 32, 16, and 2, respectively. The dropout layer is used before the last layer to prevent overfitting. The activation functions for the first five layers are Relu, and the last layer is Softmax. The loss function is sparse categorical cross-entropy (SCCE) and the optimizer is Adam. The learning rate, epoch, and batch size are 0.01, 200, and 30, respectively.

Figure 7b shows the LSTM cell [49] used in decoding the UWOCC system. Variables in the LSTM cell include x_t, C_t−1, C_t, h_t−1, h_t, and y_t. They are the current input, cell state memory from the last cell, cell state memory from the newly updated cell, hidden state output of the last cell, hidden state output of the current cell, and current output, respectively. σ is the sigmoid activation. There are three gates in each LSTM cell. They are the forget gate, input gate, and output gate. The forget gate can decide whether the information should be stored or erased, the input gate determines what new information is stored, and the output gate produces output based on the LSTM cell state after different operations. First, we discuss the forget gate operation as shown in Equation (1).

$$f_t=\sigma (W_f[x_t,h_{t-1}]+b_f)$$

(1)

where W and b are the weight and base, respectively. The result in Equation (1) will be multiplied by the old LSTM cell state C_t−1 and added with the results from the input gate to produce the new LSTM cell state C_t, as illustrated in Equation (2).

$$C_t=f_t\times C_{t-1}+i_t\times g_t$$

(2)

Then, we discuss the input gate. Equation (2) shows the product of the two outputs from input gate i_t × g_t, and they can be expressed in Equation (3).

$$\begin{array}{l}{i}_t=\sigma (W_i[x_t,h_{t-1}]+b_i)\\ g_t=\tanh (W_g[x_t,h_{t-1}]+b_g)\end{array}$$

(3)

Finally, the result from the output gate can be expressed in Equation (4).

$$o_t=\sigma (W_o[x_t,h_{t-1}]+b_o)$$

(4)

Then, the updated hidden cell state obtained in Equation (4) can be multiplied with the tanh function of the previous LSTM cell state, as illustrated in Equation (5).

$$y_t=h_t=o_t\times \tanh (C_{t-1})$$

(5)

In the LSTM-NN, several time domain features should be extracted for training. In the experiment, the input OCC signal is divided into several groups, and the number of each group is the “PPB”. The parameter for time is “time-step”, which is equal to 4. This means that when time is equal to t, the signal from t − 4 to t − 1 will be considered, as illustrated in Figure 8.

Finally, BER measurement of the RS-based UWOC system with PMMA SGOF is performed. Here, in this proof-of-concept experiment, the LSTM-NN will be compared with the PPB-NN [40]. Additionally, the experiment of the UWOC system without water is also performed for comparison. Figure 9 shows the architecture of the PPB-NN used for the comparison. It consists of one input layer, two hidden layers, and one output layer. The activation functions are Relu and Softmax, respectively, as shown in Figure 9. It is similar to a typical fully connected deep neural network (DNN) but with a different input layer. Here, the PPB-NN input layer should match the pixel per bit in the experiment. If the pixel per bit is not a round number, it should be rounded off to the higher round number. For example, if the PPB is 4.1, it will be rounded off to 5; then, the number of neurons in the input layer is 5.

4. Results and Discussion

Figure 10 shows the BER measurements of the RS-based UWOC system with PMMA SGOF using the proposed LSTM-NN model and PPB-NN model in the underwater and without water (i.e., air) cases. It can be observed that the BER curves with the water case and without the water case are similar at low data rates < 1.8 kbit/s. Without the water case, all of the BER curves can satisfy the pre-FEC threshold (i.e., BER ≤ 3.8 × 10⁻³), and the data rate can achieve 3 kbit/s. In the underwater case, we can observe that the water significantly affects the OCC signal detection by the image sensor; hence, choosing a suitable model is important. When PPB-NN is used, the data rate > 2.1 kbit/s cannot satisfy the pre-FEC threshold. However, when the LSTM-NN is used, a data rate of 2.7 kbit/s can be achieved, satisfying the FEC threshold. In the experiment, tap water is used, and it is at room temperature. As shown in Figure 2b, the water is stored in a water tank, and the experiment is performed at the laboratory in the afternoon. We believe that as long as the water density variation will not distort the rolling shutter pattern, UWOCC performance will be unaffected.

In this experiment, because we need to handle the problem of signal classification, we use the sparse categorical cross-entropy function as the loss function. We can use the loss function based on the training data and obtain the corresponding loss value in each epoch. The smaller the loss value, the better the model prediction. Here, for every 300 image frames, the first 30 image frames are used for training, and the rest of the 270 image frames are used for testing. Of the thirty training image frames, six of them will be used for validation; hence, the validation rate is 0.2. As shown in Figure 11, when the number of training times increases, the loss value decreases and starts to converge after 75 epochs. Because the loss value decreases as the number of training times increases, and it completely converges after 200 epochs with a loss value of <10⁻³, the proposed LSTM-NN model has shown good accuracy. The learning rate, epoch, and batch size are 0.01, 200, and 30, respectively.

Here, we also investigate the error occurring using the PPB-NN model and the LSTM-NN mode, as illustrated in Figure 12a,b, respectively. The received data pattern and the predicted data pattern are also illustrated. At about the 250th pixel in the received signal, there are several higher-intensity peaks. When applying the PPB-NN model, the predicted result is incorrect, as illustrated in Figure 12a. When applying the LSTM-NN model, the correct result can be observed, showing that the LSTM-NN model has better accuracy than the PPB-NN in underwater conditions.

As the PMMA SGOF is thin and only occupies a few pixels in the RS scanning direction, we also study the applicability of performing spatial multiplexing for enhancing the transmission capacity. Here, two fibers are put side by side at different separation distances, and we study the crosstalk produced. Due to the limited AWG and SGOF available in the laboratory, we can only experimentally evaluate two fibers carrying different random OOK data. Figure 13 shows the BER measurements at different data rates and different fiber separations of 0.5 cm, 3 cm, and 5 cm, respectively. The decoding algorithm is based on the LSTM-NN model described in Section 3. In order to get rid of the fiber reflection by the water tank, they are studied in the air condition. We can observe that the crosstalk is negligible when the fiber separation is at 5 cm. By decreasing the fiber separation, the crosstalk between the two fibers increases. This is because when the fiber separation decreases, the occupied pixels in the RS pattern also decrease, as illustrated in

Table 3. Occupied pixels in the RS pattern at different fiber separations.

Fiber Separation (cm)	Occupied Pixel (Pixels)
0.5	20
3	113
5	184

. When the fiber separation is 0.5 cm, a total data rate of 6 kbit/s (i.e., two fibers case), satisfying the pre-FEC BER of 3.8 × 10⁻³, can be achieved. At this fiber separation of 0.5 cm, the occupied number of pixels between the two fibers is ~20 pixels. As our CMOS image sensor has a 1920 × 1440 pixel resolution, it can be estimated that 96 fibers can be captured in the image sensor. Hence, an aggregated data rate > 200 kbit/s (i.e., 96 fibers × 3 kbit/s) could be possible.

5. Conclusions

In this work, we presented and demonstrated a RS-based UWOCC system utilizing an LSTM-NN with SGOF. In order to enhance the applicability of OCC systems, both the Tx and Rx sides should have a wide FOV. Recently, wide FOV optical Txs based on SGOF were proposed. They can provide 360° uniform light emission around the fiber circumference. This means that SGOF can act as an “omni-directional optical antenna”. In the literature, SGOF was installed in a flying UAV to provide OCC transmission. Furthermore, SGOF was demonstrated to support multiple movable Rxs to obtain data signals along the fiber. SGOF is lightweight and flexibly bendable. Most importantly, SGOF is water resistant; hence, it can be installed in underwater environments. As SGOF Tx is lightweight, flexibly bendable, and water resistant, one potential application is that it can be easily installed on the surface of underwater AUVs for transmitting sensing or surveillance information to other nearby AUVs or divers equipped with CMOS image sensors for signal receiving. The proposed LSTM-NN has the time-memorizing characteristics to enhance OCC signal decoding. The proposed LSTM-NN is also compared with other decoding methods in the literature, such as the PPB-NN. The experimental results demonstrated that the proposed LSTM-NN outperforms the PPB-NN in the UWOCC system. A data rate of 2.7 kbit/s can be achieved in UWOCC, satisfying the pre-FEC condition (i.e., BER ≤ 3.8 × 10⁻³). We also studied that at a fiber separation of 0.5 cm, the occupied number of pixels between the two fibers was ~20 pixels. When using a CMOS image sensor with a 1920 × 1440 pixel resolution, it can be estimated that 96 fibers can be captured in the image sensor. Hence, an aggregated data rate > 200 kbit/s (i.e., 96 fibers × 3 kbit/s) could be possible.