Training a Dataset Simulated Using RGB Images for an End-to-End Event-Based DoLP Recovery Network

Abstract

Event cameras are bio-inspired neuromorphic sensors that have emerged in recent years, with advantages such as high temporal resolutions, high dynamic ranges, low latency, and low power consumption. Event cameras can be used to build event-based imaging polarimeters, overcoming the limited frame rates and low dynamic ranges of existing systems. Since events cannot provide absolute brightness intensity in different angles of polarization (AoPs), degree of linear polarization (DoLP) recovery in non-division-of-time (non-DoT) event-based imaging polarimeters is an ill-posed problem. Thus, we need a data-driven deep learning approach. Deep learning requires large amounts of data for training, and constructing a dataset for event-based non-DoT imaging polarimeters requires significant resources, scenarios, and time. We propose a method for generating datasets using simulated polarization distributions from existing red–green–blue images. Combined with event simulator V2E, the proposed method can easily construct large datasets for network training. We also propose an end-to-end event-based DoLP recovery network to solve the problem of DoLP recovery using event-based non-DoT imaging polarimeters. Finally, we construct a division-of-time event-based imaging polarimeter simulating an event-based four-channel non-DoT imaging polarimeter. Using real-world polarization events and DoLP ground truths, we demonstrate the effectiveness of the proposed simulation method and network.

1. Introduction

In recent years, various bio-inspired neuromorphic sensors have emerged, including event cameras, which have been widely studied. Compared with traditional frame-based cameras, event cameras do not have fixed frame rates or integration times and output asynchronous event stream information. Their advantages include high temporal resolutions, high dynamic ranges, low latency, and low power consumption [1]. As a new form of optical sensors, they are widely used in many research fields, such as visual navigation [2,3,4], image reconstruction [5,6,7,8], optical flow [9,10,11], and some optical applications [12,13,14]. However, optical information is not only about intensity; it also concerns polarization. Traditionally, imaging polarimeters are used for such purposes as target detection [15,16,17], 3D reconstruction [18,19,20], underwater image recovery [21,22], and remote sensing [23,24,25,26]. However, traditional frame-based imaging polarimeters have limited frame rates and dynamic ranges [27].

Therefore, researchers have developed division-of-time (DoT) and division-of-focal-plane (DoFP) event-based imaging polarimeters [27,28,29,30]. Theoretical models and methods are used to calculate polarization information based on event-based DoT imaging polarimeters. However, since an event only represents a relative change in the brightness intensity and does not include the absolute brightness intensity, recovering the degree of linear polarization (DoLP) of event-based non-DoT imaging polarimeters is an ill-posed problem [30].

Due to this lack of intensity information, deep learning methods are used in event processing for the DoLP recovery of non-DoT polarization events. Data-driven neural networks require large amounts of data to train network models for better results and generalization ability. However, due to the limited development of event cameras, significant amounts of resources are needed to construct event-based non-DoT imaging polarimeters. Moreover, collecting thousands of event sequences in different scenes with DoLP ground truths for training requires considerable resources and time. Thus, for the training of a DoLP recovery network, we propose a simulation method for generating polarization event datasets. Using a single red–green–blue (RGB) image combined with an existing event simulator, we simulate a sequence of polarization events and DoLP ground truths for training. This approach can greatly reduce the time and resource consumption of building a training dataset and can convert existing RGB image datasets into polarization event training sets. Finally, we build an event-based DoT imaging polarimeter to verify the proposed simulated dataset and DoLP recovery network. We obtain real non-DoT polarization events and intensities by sampling the same motion of the same target at different angles of polarization (AoPs) and offline calibrating times. The experiment under real conditions demonstrates the effectiveness of the proposed simulated dataset and DoLP recovery network.

In summary, (1) we propose a dataset simulation method for event-based DoLP recovery networks. This approach can construct large-scale polarization event datasets using existing RGB images, thus reducing time consumption, cost, and research barriers and promoting the development of polarization event processing. (2) We propose an end-to-end event-based DoLP recovery network model that can effectively restore the DoLPs of targets from polarization events at four AoPs. (3) We build an event-based DoT imaging polarimeter and provide a time calibration method to obtain non-DoT polarization events and DoLP ground truths under real-world conditions.

2. Background and Related Work

This section provides an overview of event cameras and polarization imaging and presents recent research and development efforts related to polarization events.

2.1. Event Camera

Event cameras are bio-inspired neuromorphic sensors that are also called dynamic vision sensors (DVSs). Each pixel passively detects the change of log brightness intensity and outputs an event when it exceeds a threshold, as shown in Figure 1. Each event contains coordinates, a timestamp, and polarity (positive or negative). Thus, the asynchronous event stream does not have a fixed frame rate or integration time. The outputs of traditional frame-based cameras and event cameras are shown in Figure 2. The red and blue points correspond to positive and negative events, respectively. In most cases, events are caused by the movement of the target or the camera. The difference in reflectivity between the target and the background causes a change in the brightness intensity received by a certain pixel during motion, which then triggers an event. An individual pixel of an event camera can detect and output brightness intensity changes at the microsecond level. Compared with traditional frame-based cameras, event cameras have high temporal resolutions, low latency, and low power consumption. In addition, because the differential detector used by event cameras for event triggering detects changes in the log brightness intensity, event cameras have higher dynamic ranges than traditional frame-based cameras.

Event cameras originated from the study of Mahowald and Mead at the California Institute of Technology from 1986 to 1992 [31]. Since the first commercial event camera appeared in 2008 [32], these devices have gradually become a widely researched area of computer vision and visual navigation. Early event cameras, such as DVSs [33,34], only output events, whereas subsequent devices, such as asynchronous time-based image sensors [35,36] and dynamic and active-pixel vision sensors [37,38], can output both events and intensities through different circuits. After more than a decade of development, the resolution of event cameras has improved (reaching mainstream levels for frame-based cameras) [39,40,41], providing further possibilities for related applications.

Figure 2. Comparison of output between a frame-based camera and an event camera. Data come from [42].

Figure 2. Comparison of output between a frame-based camera and an event camera. Data come from [42].

2.2. Polarization

Polarization is a fundamental parameter in optics. Polarization imaging systems can be mainly divided into DoT systems [43,44] and non-DoT systems. A DoT system mainly consists of a rotatable polarizer and an imaging sensor. Through the rotation of the polarizer, the sensor can obtain the brightness intensities at various AoPs at different times. A non-DoT system mainly uses methods such as the division of amplitude (DoAm) [45,46], division of aperture (DoAp) [47,48], and DoFP [49,50] for the simultaneous acquisition of multiple sets of brightness intensities at different AoPs for the same scene. The polarization of the scene can be calculated using a Stokes vector ($S_0$, $S_1$, $S_2$, and $S_3$, where $S_3$ represents circular polarization states; this element is approximately equal to 0 in natural scenes and is, therefore, not discussed in our work). For instance, commonly used DoFP cameras generally obtain brightness intensities at AoPs of 0°, 45°, 90°, and 135°. In this scenario, the DoLP and AoP of the scene are calculated using Equations (1)–(3), where $I_0$, $I_{45}$, $I_{90}$, and $I_{135}$ are the brightness intensities at four different AoPs and $S_0$, $S_1$, and $S_2$ are the first three components of the Stokes vector [51]:

$$\left\{\begin{array}{c}{S}_0=I_0+I_{90}\hfill \\ S_1=I_0-I_{90}\hfill \\ S_2=I_{45}-I_{135}\hfill \end{array}\right\}$$

(1)

$$\mathrm{DoLP}=\frac{\sqrt{S_1^2+S_2^2}}{S_0}$$

(2)

$$\mathrm{AoP}=\frac{1}{2}arctan\left(\frac{S_2}{S_1}\right)$$

(3)

2.3. Polarization Events

Event-based imaging polarimeters can also be divided into DoT systems and non-DoT systems. Lu et al. [28,29] and Hawks and Dexter [27] established event-based DoT imaging polarimeters in 2021 and 2022, respectively. For stationary scenes, the events caused by the rotating polarizer, which changes the intensity of the passing brightness, can be obtained. For an event-based DoT imaging polarimeter, the DoLP and AoP can be calculated directly through the event rate. Muglikar et al. [52] used such systems for 3D reconstruction and verified the feasibility of using event-based systems for polarization applications. Haessig et al. [30] have developed a polarization event camera that couples a micropolarization array to the image plane of the event camera, enabling the simultaneous acquisition of events at four AoPs. This realizes the application of polarization events in dynamic scenes such as polarized target detection.

The literature on polarization events remains limited. First, due to the short history of event cameras, they are expensive, and building an event-based imaging polarimeter requires substantial resources, especially for non-DoT systems, which may require multiple event cameras. Second, for DoT systems, the DoLP and AoP of a scene can be directly calculated using the events. However, for non-DoT systems, the DoLP cannot be directly calculated without the reconstruction of intensity images at four AoPs. The application scenarios of various systems are summarized and compared in

Table 1. Comparison of imaging polarimeters.

Type	DoLP	AoP	Application Scenarios
DoT	✓	✓	Static scene in standard dynamic range (SDR)
Non-DoT	✓	✓	Static scene and slow motion in SDR
Event-based DoT	✓	✓	Static scene in high dynamic range (HDR)
Event-based non-DoT	✗	✓	Fast and slow motion in HDR

.

3. Method

This section presents the dataset simulation method and the network structure, loss function, and training method of the end-to-end DoLP recovery network.

3.1. Simulated Dataset

Training an event-based DoLP recovery network requires a large dataset. As building an event-based non-DoT imaging polarimeter needs large amounts of resources, setting up and shooting the dataset scenes are time consuming. Inspired by studies on event-based image reconstruction, we propose a method for generating simulated datasets. Two requirements should be fulfilled to simulate polarization events and DoLP ground truths. First, there must be a large number of scenes or targets with different polarization distributions. Second, there must be brightness intensity changes that can trigger events (for an event-based non-DoT imaging polarimeter, such changes are mainly caused by the motion of the target). We obtain simulated scene polarization distributions from existing RGB image datasets as ground truths and then generate events at multiple AoPs using the simulated motion. This method can acquire adequate training data while saving resources and time.

We are inspired by the visualization method commonly used in polarization image displays. In Figure 3b, shows the raw data captured by a DoFP camera (intensities at four AoPs) and Figure 3c shows the DoLP and AoP of the scene. By mapping the raw data to the hue–saturation–value (HSV) color space, as shown in Figure 3a, we obtain an image similar to that in Figure 3c.

We can also convert color images from the HSV color space into corresponding polarization values, as shown in Figure 4. This does not represent the true polarization characteristics of the scene, but we can obtain a large number of false polarization characteristic distributions that can be used for network training. Using Equation (4), we obtain simulated polarized images at different $\theta$, such as 0°, 45°, 90°, and 135°:

$$I_{\theta }=\frac{1}{2}{S}_0(1-\mathrm{DoLP})+S_0\mathrm{DoLP}{cos}^2(\theta -\mathrm{AoP}).$$

(4)

To visualize the polarization images, we only need to determine the DoLP and AoP. Therefore, only two HSV channels are used, and the S channel can be set to a constant value of 1. In the simulation process, as indicated by Equation (4), we need to input three variables, namely, $S_0$, AoP, and DoLP, which correspond exactly to the three HSV channels. The overall structure of the simulation is shown in Figure 5. By converting an RGB image to the HSV color space and mapping the three HSV channels to AoP, DoLP, and $S_0$, we obtain the polarization distribution of a scene. Then, by simulating motion, we obtain four image sequences with continuous changes in brightness intensities at the four AoPs. Finally, we can simulate the events at these AoPs from these sequences using existing event simulators [53,54,55] (we use V2E [54] in our experiments), and pairs of ground truths and polarization events are obtained for supervised network training.

The network can correctly learn to recover DoLPs from polarization events using the dataset simulated using RGB images. For an event-based non-DoT imaging polarimeter, the network should learn to recover the DoLPs of scenes from events at different AoPs. We initially reconstruct intensity images at various AoPs and then calculate the DoLPs from these intensity images using Equations (1) and (2). We use an end-to-end network to fit these two processes. In the dataset simulation, we reverse this process. We map RGB images to HSV space and then to AoP, DoLP, and $S_0$. This process is used solely to generate a large number of random initial scene polarization spatial distributions and does not imply that we obtain real-world physical polarization distributions. Through this large set of generated polarization spatial distributions, we can obtain intensity spatial distributions at different AoPs. All in all, simulating the intensity images at different AoPs is consistent with the DoLP formula, and simulating the motion and events is consistent with the event camera model. Therefore, for the network, whether the original source of data is real or simulated polarization distributions is irrelevant; the abovementioned consistency of the simulations with the DoLP formula and the event camera model is the only requirement.

3.2. Input and Output of the Network

The input and output of the end-to-end DoLP recovery network are the four channels of polarization events and DoLP ground truths, respectively. However, the event camera outputs asynchronous sparse 3D spatiotemporal data, which we need to compress into a 2D representation, such as a time surface, to make them correspond to the ground truths for supervised training. We gather events within a time window and assign different values to them based on their timestamps. The formula is as follows:

$$\mathrm{Event}\mathrm{tensor}=\sum _{i=1}^n\frac{p_i\left(t_i-t_0\right)}{\Delta t},$$

(5)

where n is the total number of events within the time window, $\Delta t$ is the time window, $p_i$ is the polarity of event i, $t_i$ is the timestamp of event i, and $t_0$ is the timestamp of the first event in time window. The four-channel event tensor input to the network and the single-channel DoLP output are shown in Figure 6.

3.3. Network Architecture

Our network model uses a U-Net [56]-like architecture combined with a residual block [57], a swin transformer block [58], and a ConvGRU [59]. The overall network structure is shown in Figure 6. C1 expands the input four-channel event tensor to 32 channels. In S1 and S2, the downscaling factor is 2 and the hidden dimensions are 64 and 128, respectively. The input dimensions of G1 and G2 are 64 and 128, respectively. R1 and R2 use 128 input and output channels, stride-1 3 × 3 convolutions, and ReLU activation. P1 is the final prediction layer (with a 1 × 1 convolution), and the output channel is 1.

3.4. Loss

The total loss function of our network is a combination of three loss functions. First, it includes the mean square error (MSE) loss, which is commonly used to evaluate the pixel-level error between a prediction and the ground truth. The use of this loss function ensures that the DoLP trained by the network is close to the ground truth, as shown in Equation (6):

$$L_2={(y-f\left(x\right))}^2,$$

(6)

where $L_2$ is the MSE loss, y is the ground truth, and $f\left(x\right)$ is the network prediction. Second, we use structural similarity (SSIM), which includes a similarity comparison of three dimensions: image brightness, contrast, and SSIM. This loss function is commonly used to ensure SSIM between predictions and true values. As a larger SSIM indicates closer images, we implement some adjustments when using it as a loss function, as shown in Equation (7):

$$L_S=1-SSIM(y,f\left(x\right)),$$

(7)

where $L_S$ is the SSIM loss and $SSIM$ is the SSIM loss function. The SSIM parameters are set to default values. Considering that the abovementioned pixel-level loss may lead to blurry reconstruction results [60], we add the perceptual loss function LPIPS [61]. By feeding the network prediction and ground truth into a VGG network pretrained on ImageNet, we calculate their feature distance in the VGG’s feature layers. By minimizing this loss function, we make the network prediction and ground truth closer to natural statistics, as shown in Equation (8).

$$L_L=LPIPS(y,f\left(x\right)),$$

(8)

where $L_L$ is the LPIPS loss and $LPIPS$ is the LPIPS distance. Thus, our total loss function is as follows:

$$L={\lambda }_1\sum _{i=1}^n{L}_2+{\lambda }_2\sum _{i=1}^n{L}_S+{\lambda }_3\sum _{i=1}^n{L}_L,$$

(9)

where n is the number of samples in training dataset and ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are the weighting coefficients of the three loss function terms and are used to balance the value range of each loss function.

3.5. Training

Using the simulation method described in Section 3.1, we simulate 1000 sequences of 1 s (30 Hz) from an existing RGB dataset (MIT-Adobe 5K [62]). Each sequence includes events at four AoPs and the DoLP ground truths. The original size of the images is 400 pixels × 600 pixels, and the window size we select for motion simulation is 120 pixels × 180 pixels. In the event simulator, the trigger threshold for positive and negative events is set to 0.2, and all other items are set to their default values. Using an Intel Xeon Gold 6226R CPU (@2.90GHz) and a NVIDIA GeForce RTX 3090 GPU, we can generate 1000 sequences in two hours and fifteen minutes, by running five simulation programs in parallel.

We implement our network using PyTorch [63] and use ADAM [64] with a learning rate of 0.0001. The batch size is 2. We train for a total of 250 epochs. In the loss function, ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are set to 1, 1, and 10, respectively.

4. Experiment Setup

To validate the dataset simulation approach and end-to-end DoLP recovery network, we collect polarization events and DoLPs from actual scenes. Due to our current experimental conditions, we cannot construct a real event-based non-DoT imaging polarimeter system. Therefore, we use a DoT system as a substitute and obtain events and intensity images at four AoPs by rotating the polarizer multiple times. By controlling the target to perform the same motion each time and using markers for time calibration, we align the events and intensity images obtained from multiple acquisitions using their timestamps, achieving the same effect as that of a non-DoT system for experimental validation. The experimental setup is shown in Figure 7.

In the experiment, we use a CeleX5 event camera, which can switch output modes (intensity frames or events). Due to the camera’s resolution (800 pixels × 1280 pixels), we crop the central area to 480 pixels × 720 pixels and resize it to 120 pixels × 180 pixels for the subsequent calculations. We collect a total of 32 sets of targets, with each set being captured eight times (four sets for events and four sets for intensities). By detecting the appearance time of the landmark in the images and events, we determine the starting time points of multiple sets of data for the same scene. Therefore, we select an area in the upper right corner of the pixel plane for detection. The area size is 45 pixels × 2 pixels. For intensity sets, when the average value in the region is less than 30, we determine that timestamp as the time starting point of the set. For event sets, when 45 negative events accumulate in the area, we record that timestamp as the time starting point of the set. In most of the collected data, calibration can be successfully completed, but there are a few scenes that require further manual fine-tuning of the time starting point. The consistent rotation speed of the turntable ensures one-to-one correspondence between subsequent moments. An example set after calibration is shown in Figure 8. The DoLPs in Figure 8 are calculated using the intensity images obtained at the four AoPs.

5. Results and Discussion

We compare the effectiveness of the proposed dataset simulation approach and end-to-end DoLP recovery network model with that of existing methods. As non-DoT polarization event systems cannot directly calculate DoLPs, existing methods first need to recover the intensity distributions at the four AoPs and then use the Stokes formula for DoLP calculation. We select two classic recovery algorithms for event stream intensities: Complementary Filter (CF) [5] is a traditional event stream intensity recovery algorithm, and Events to Video (E2V) [6] is a classic deep learning algorithm. Both algorithms input only events to reconstruct intensity at the four AoPs. In the experiment, both are set to their default parameters and intensity images are recovered at four AoPs to calculate DoLPs using the Stokes formula Equations (1) and (2).

The experimental results indicate that the network model trained on the simulated datasets can accurately recover polarization in real-world scenarios. This verifies the effectiveness of the dataset simulated using RGB data. As shown in Figure 9, DoLP recovery using CF and E2V is limited by the image reconstruction results and a noticeable trailing phenomenon occurs in the motion direction of the target, as shown by the red boxes numbered 1 and 5. The DoLP image estimated by the E2V method is affected by background noise events, resulting in errors in the background. CF is also affected by noise events, resulting in errors in the target, as shown in the red box numbered 5 in Figure 9a. The proposed method has clearer outlines, has a clean background, and is not affected by event trailing. As shown in the red boxes numbered 2 and 3, the DoLP estimated by CF is low, and the DoLP estimated by E2V is high but contains nonsmooth mutations. The red boxes numbered 4 suggest that when the target DoLP is low, CF cannot easily calculate the edge of the polarized target correctly; E2V can calculate it, but the edges are blurred. By contrast, our method can complete the edge well.

In summary, the real-scene experiments indicate that the results of CF and E2V estimation are affected by the quality of the reconstructed intensity image. Moreover, the DoLP of the target is difficult to estimate correctly, and it is affected by noise events, whether in the background or the target area. The proposed method, trained using the RGB simulation dataset, can work in real scenes and achieve better results than existing approaches.

Due to the limitations of the constructed DoT system, only simple object movements can be performed. For further evaluation of the DoLP recovery effect under complex motion, we use simulated polarization events for comparison. We use the same parameter settings as those for the simulation training set and select 50 sets of images that are not in the training set for simulation experiments. As shown in Figure 10, correctly recovering the DoLP distribution of the scene using CF in complex scenes and movements is difficult. Compared with the proposed method, although E2V can also recover the DoLP of the scene, its error is larger. Overall, the proposed network model can better recover the DoLP of the scene and exhibits better performance in complex scenes and motions. To evaluate DoLP recovery objectively, we use the MSE, peak signal-to-noise ratio (PSNR), SSIM, and LPIPS for metric comparison, as shown in

Table 2. Experimental results.

Method	Real Experiment				Simulated Experiment
	MSE ↓	PSNR ↑	SSIM ↑	LPIPS ↓	MSE ↓	PSNR ↑	SSIM ↑	LPIPS ↓
CF	0.0087	23.17	0.4116	0.3342	0.1371	10.60	0.1667	0.5890
E2V	0.0134	20.02	0.3819	0.3350	0.0778	12.74	0.3068	0.4140
Ours	0.0026	27.48	0.5189	0.1369	0.0355	15.56	0.3947	0.3295

. Our method outperforms the existing approaches in both the real and simulated scenes.

To further assess how many events are needed for a non-DoT event-based imaging polarimeters to work effectively, we conducted additional experiments. As shown in Figure 11, t refers to the timestamp of the last event in the input event tensor, and n refers to the total number of events contained in the input event tensor (average number of events at the four AoPs). For example, In Figure 11a, when t = 24 ms and n = 79,191, it means that the input event tensors at the four AoPs contain 79,191 events on average within the time range of 0–24 ms. In Figure 11b, when t = 35 ms and n = 100,000, it means that the input event tensor at each AoP contains 100,000 events with the last timestamp less than 35 ms.

From Figure 11a, it can be observed that at the beginning of the motion, there are very few events, which are insufficient to effectively recover the target’s DoLP. As the motion continues, the number of input events increases, and the recovery effect of the DoLP gradually improves, eventually stabilizing. In our experiment, we find that when the number of input events reaches about 100,000, the recovery of DoLP can achieve good results. Considering that the target occupies approximately a 240 pixels × 400 pixels area (before resizing), the event occurrence rate is 1.04 (events per pixel). In Figure 11b, it shows that it is possible to recover DoLP with a very high temporal resolution, by overlapping some events during continuous motion as input. At this point, it can be considered that the time resolution of DoLP is close to the event camera itself, at the sub-millisecond level.

6. Conclusions

We propose a training dataset simulation method and an end-to-end DoLP recovery network. We simulate events and DoLP ground truths at multiple AoPs from existing RGB image datasets for network training. This effectively reduces the resources and time required for system setup and training set collection. Since the DoLP is difficult to calculate directly for event-based non-DoT polarimeters, we propose the end-to-end U-Net-like network structure combining a swin transformer block, a residual block, and a ConvGRU. This can directly recover DoLPs from polarization events. Finally, we build an event-based DoT imaging polarimeter by controlling the motion of the target in the scene to collect events at different AoPs multiple times and achieve the effect of a non-DoT system. Multiple sets of data are collected to verify the effectiveness of the proposed array training set and end-to-end network. In future studies, we will build a non-DoT polarization event system to experiment with the advantages and application effects of using an event camera in polarimetric imaging systems.