Polarization-Based Two-Stage Image Dehazing in a Low-Light Environment

Abstract

Fog, as a common weather condition, severely affects the visual quality of images. Polarization-based dehazing techniques can effectively produce clear results by utilizing the atmospheric polarization transmission model. However, current polarization-based dehazing methods are only suitable for scenes with strong illumination, such as daytime scenes, and cannot be applied to low-light scenes. Due to the insufficient illumination at night and the differences in polarization characteristics between it and sunlight, polarization images captured in a low-light environment can suffer from loss of polarization and intensity information. Therefore, this paper proposes a two-stage low-light image dehazing method based on polarization. We firstly construct a polarization-based low-light enhancement module to remove noise interference in polarization images and improve image brightness. Then, we design a low-light polarization dehazing module, which combines the polarization characteristics of the scene and objects to remove fog, thereby restoring the intensity and polarization information of the scene and improving image contrast. For network training, we generate a simulation dataset for low-light polarization dehazing. We also collect a low-light polarization hazy dataset to test the performance of our method. Experimental results indicate that our proposed method can achieve the best dehazing effect.

1. Introduction

In natural environments, images captured in long-distance observation or low-light conditions often contain a significant amount of noise and blur due to factors such as dust, haze, and pollution. Traditional image enhancement methods often struggle to address these issues, highlighting the need for the development of more efficient and accurate techniques to handle images in such environments.

Utilizing polarization-based dehazing technology has become a vital image enhancement technique, receiving considerable attention in the fields of computer vision and image processing [1,2,3,4]. Polarization-based dehazing leverages the polarization properties of light to restore scene details concealed by haze, thereby improving image clarity and contrast. In comparison to traditional single-image dehazing methods, polarization dehazing techniques provide unique advantages and distinctions. Conventional single-image dehazing methods mainly rely on image processing techniques to estimate haze density and grayscale distribution in images, aiming to recover details concealed by haze [5,6,7,8,9,10,11]. However, these methods often have difficulty accurately estimating the extent of haze interference, especially in intricate lighting scenarios. Additionally, they frequently introduce artifacts and distortion, resulting in dim and blurred images. In contrast, polarization dehazing techniques utilize the polarization properties of light to address haze-related issues in images, providing advantages in terms of physical mechanisms. By utilizing polarization information, polarization dehazing techniques can separate haze components and precisely model them, thereby restoring scene details concealed by haze.

Even though polarization-based dehazing techniques perform well in bright scenes, they cannot be used for dark environments. This is because low-light scenarios pose unique challenges and difficulties [12,13,14].

First, low-light scenarios typically involve very low light intensity [15,16]. Conversely, traditional polarization dehazing algorithms often assume sufficient illumination, relying on relatively bright lighting conditions to separate haze and scene information accurately. However, in low-light situations, the intensity of light is greatly reduced, rendering conventional polarization dehazing algorithms unable to accurately evaluate the impact of haze. This results in inaccurate restoration.

Secondly, low-light scenarios often involve high-level noise interference [17,18,19]. Traditional polarization dehazing algorithms typically assume low image noise, which can be removed through simple filtering methods. However, in low-light scenarios, noise interference is significant and can easily be confused with haze phenomena. This makes traditional filtering methods ineffective in noise removal, consequently affecting dehazing results.

Furthermore, the polarization characteristics in low-light scenarios differ from those in daytime scenarios [20,21]. The outdoor light source during the day is primarily sunlight, which exhibits strong polarization characteristics after being scattered by the atmosphere. In low-light scenarios, the main light sources are night sky illumination and urban glow. While night sky illumination also possesses a certain degree of polarization after atmospheric scattering, the polarization characteristics of urban glow are relatively weak. Consequently, atmospheric polarization transmission models developed for daytime are not suitable for low-light scenarios, making conventional polarization dehazing algorithms incapable of restoring images in low-light conditions.

To overcome these challenges, this paper proposes a novel two-stage approach based on low-light polarization dehazing, aiming to address the issues faced by traditional dehazing methods in low-light environments and enhance image quality. Regarding the issues of image brightness and noise, we first design a polarization-based image enhancement module that improves the overall visual effect of the image while preserving its polarization characteristics. Secondly, to remove the interference of haze in the image, we propose a polarization-based dehazing module that extracts the features of the image from both intensity and polarization perspectives. Both modules employ the same feature extraction module but are trained using different data and loss functions. The polarization image enhancement module is trained using data captured by the camera, allowing for restoration of the imaging quality of that specific camera. However, it is impossible to achieve a large number of data for the polarization-based dehazing module. Therefore, in this paper, we generate a simulated polarization haze dataset. Overall, the main contributions of this paper are as follows:

• We propose a two-stage image dehazing method based on polarization to recover the image quality of hazy images from a low-light environment;
• We propose a low-light polarization dehazing dataset which provides an integrated quality evaluation framework for low-light polarization image dehazing;
• The experimental results demonstrate that the proposed method achieves good dehazing results and exhibits significant improvements in terms of image quality, detail restoration, and contrast enhancement.

2. Proposed Method

In this section, we will describe the proposed two-stage polarization image dehazing model shown in Figure 1 in detail. Due to the limited sensitivity of polarization imaging equipment, there are issues with insufficient brightness and low signal-to-noise ratio in low-light environments. We design a polarization-based image enhancement module that includes a multi-scale feature extraction module (MSFEM) and a reconstruction module. The multi-scale feature extraction module extracts multi-modal information on intensity and polarization based on the input polarized images and performs feature learning on different scales of feature expression dimensions. The enhanced image information will be recovered based on the reconstruction module. To address the issue of images being disturbed by scattering media, a polarization-based image dehazing module is introduced, which includes a multi-scale feature extraction module, polarization feature extraction module (PFEM), and multi-modal reconstruction module (MRM). The polarization feature extraction module performs targeted feature extraction based on the polarization degree and angle information of the input image and combines the feature expressions output by the multi-scale feature extraction module to jointly achieve the reconstruction of multi-modal images. The whole model results in the output of images with dehazed polarization as well as intensity.

2.1. Polarization-Based Image Enhancement Module

In comparison with the conventional image enhancement models, polarization-based image enhancement modules need to simultaneously consider both the intensity information and the polarization information of the image. We aim to recover the image using the four-channel DoFP image. Inspired by the structures of [19], we design a cross-scale network for the enhancement model, as shown in Figure 2. Firstly, the four-channel inputs are fed into the head module, which maps them into a high-dimensional feature space. Then, the features are sent to the cross-scale module. To extract local and global features from images, we not only utilize parallel branches to represent features at different levels, but also enhance the interaction of multi-scale image features. At the end of the module, we add two convolutional layers as the reconstruction module to reconstruct the recovered image. The central component of the enhancement module is the residual attention module, which harnesses the benefits of attention blocks and the skip connection operator. The attention block captures important feature information from the initial feature map, while the skip connection operator helps overcome network training challenges.

2.2. Low-Light Polarization Dehazing Module

In this part, we will describe the polarization dehazing module in detail. After obtaining the enhanced noise-free image from the image enhancement module, we need to remove the haze interference. As shown in Figure 3, the low-light polarization dehazing module takes the output of the image enhancement module as input and outputs intensity, polarization angle, and polarization degree images. It can be divided into three parts: the MSFEM, PFEM, and MRM. Here, the structure of the MSFEM is consistent with that of the image enhancement module, both aiming to extract intensity features from different angle-polarized images. Since the polarization dehazing module needs to achieve the conversion of information between different modalities, i.e., intensity information and polarization information, in order to provide more guidance from polarization modalities to the network, this model proposes a PFEM. Firstly, the input image is transformed into intensity, polarization degree, and polarization angle images, which are then input into the PFEM for polarization modality information extraction. The PFEM consists of three RAM modules. The extracted features will be combined with the features from the MSFEM and enter the MRM for the restoration of dehazed images. The main task of the MRM is to convert the features extracted from the MSFEM and the PFEM into the dehazed polarization image. The MRM is divided into three branches based on different outputs: the intensity branch, polarization degree branch, and polarization angle branch. Each branch takes the corresponding dimensional features and reconstructs the image. Each branch is composed of three RAM modules.

3. Loss Function

Since different modules have different requirements, it is necessary to assign different loss functions to constrain the optimization of module parameters. To ensure the intensity and polarization characteristic of the images, we set two different loss functions for the polarization-based image enhancement module: intensity loss and polarization loss. The intensity loss is used to constrain the differences between the generated image of the polarization-based image enhancement module and the label image in the four polarization channels. It can be formulated as

$$L_{int}={∥y_0-x_0∥}_1+{∥y_{45}-x_{45}∥}_1+{∥y_{90}-x_{90}∥}_1+{∥y_{135}-x_{135}∥}_1$$

(1)

where $y_{0,45,90,135}$ represents the label image, and $x_{0,45,90,135}$ represents the enhanced image.

The polarization loss aims to recover the degree of polarization (DoLP) and angle of polarization (AoP), which can be formulated as

$$L_{polar}={∥y_{DoLP}-x_{DoLP}∥}_1+{∥y_{AoP}-x_{AoP}∥}_1$$

(2)

where $y_{DoLP,AoP}$ represents the label image, and $x_{DoLP,AoP}$ represents the enhanced image. To obtain the DoLP and AoP of the images, we first need to convert the DoFP to Stokes vectors. Afterwards, the DoLP and AoP can be calculated using the Stokes vectors.

To ensure the performance of the polarization dehazing module, the loss function consists of four parts: $s_0$ loss, DoLP loss, AoP loss, and gradient loss.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill L_{s_0}=& {∥y_{s_0}-x_{s_0}∥}_1\hfill \\ \hfill L_{DoLP}=& {∥y_{DoLP}-x_{DoLP}∥}_1\hfill \\ \hfill L_{AoP}=& {∥y_{AoP}-x_{AoP}∥}_1\hfill \\ \hfill L_{grad}=& {∥grad\left(y_{s_0}\right)-grad\left(x_{s_0}\right)∥}_1\hfill \\ \hfill & +{∥grad\left(y_{DoLP}\right)-grad\left(x_{DoLP}\right)∥}_1\hfill \end{array}\end{array}$$

(3)

where $x_{s_0}$, $x_{DoLP}$, and $x_{AoP}$ are the output of the polarization dehazing module, and the $y_{s_0}$, $y_{DoLP}$, and $y_{AoP}$ are the ground truth of the corresponding image. We choose the L1-norm loss function for its capacity to generate sparse and resilient outcomes. To retain more detailed information, we further include a gradient loss function. Nonetheless, this technique is unsuitable for AoP images because of the inherent noise. In the end, the complete loss function is defined as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill L_{all}& =L_{enhance}+L_{dehaze}\hfill \\ & =L_{int}+L_{polar}+L_{s_0}+L_{DoLP}+L_{AoP}+L_{grad}\ast 10.\hfill \end{array}\end{array}$$

(4)

4. Data Preparation

Due to the different requirements of the various modules in the algorithm presented in this paper, the corresponding datasets also have different requirements. In this section, we will introduce the datasets for the different modules separately. The dataset for the image enhancement module is based on images captured by a polarimetric camera, while the dataset for the polarization dehazing module is derived from the simulation of existing polarization images.

4.1. Data for Polarization-Based Image Enhancement Module

In our paper, we employ the FLIR BFS-U3-51S5P-C as the camera (Teledyne FLIR, Portland, OR, USA). In order to obtain the clean images in the low-light environment, we capture bursts of still images and apply the frame integral algorithm to obtain clean results for each sequence of scenes. We capture video sequences of 50 still scenes, each consisting of 300 frames. While these scenes have the same core content, they exhibit unique patterns of noise. By utilizing an integral algorithm for each frame, we derive a clear image from every video. As a result, this dataset comprises 50 clean images and 15,000 noisy images. To train our model, we randomly select 30 video sequences, reserving the remaining 20 for evaluation. Figure 4 shows examples of the noisy images and the clean images. As shown in the figure, the images basically exclude noise components, and the detailed information of the image itself is well preserved, so we employ frame integral images as ground truth images.

However, the illumination of the images is very low; it is not suitable for the image dehazing tasks. Therefore, here, we also apply a unique image enhancement model to the clean images for training.

$$\left\{\begin{array}{c}\begin{array}{cc}\hfill & L_{wmax}=Max\left(I\right)\hfill \\ \hfill & L_w=exp(\frac{1}{N}\sum log(I+\delta ))\hfill \\ \hfill & L_g=\frac{log(\frac{I}{L_w}+1)}{log(\frac{L_{wmax}}{L_w}+1)}\hfill \end{array}\hfill \end{array}\right.$$

(5)

where I represents the input clean image, $L_{wmax}$ represents the maximum value of I, $L_w$ is the “world” luminance, and $\delta$ is employed as a small value to avoid singularity in scenarios where there are black pixels within the image. $L_g$ is the enhanced image. This model is a classic tone mapping operator but it is also suitable for image enhancement in this context. As shown in Figure 5, the details of the dark image are successfully restored. In this paper, we set these enhanced noise-free images as our training data for the polarization-based image enhancement model.

4.2. Data for Low-Light Polarization Dehazing Module

Due to the lack of a large number of real data to support the polarization dehazing model, this paper considers using simulated data as training data. Based on our previous work [22], we propose a low-light polarization dehazing simulation dataset. This dataset takes a large number of polarization images as input and uses Stokes vectors as basic elements. The haze is simulated as a set of Mueller matrices, simulating fog with different distances and concentrations. The whole process is shown in Algorithm 1.

$S_T$ represents the original Stokes vector of the DoFP image, $S_A$ is the randomly generated Stokes vector of night sky radiance, ${\alpha }_A$ and ${\alpha }_T$ are the proportionality coefficients controlling the contribution of the airlight and transmitted light to scattering during each iteration, $M_{\theta }$ is the Mueller matrix that we use to simulate hazy scattering, $\beta$ is the scattering coefficient, z is the distance from the scene.

Based on the simulation method mentioned above, we present some simulated foggy weather polarized images in Figure 6. The polarized images used in this paper are from LLCP. In this paper, we use the original image as the label and the generated foggy image as the training sample.

Algorithm 1 Low-light hazy polarization simulation

1: initialize $S_A$, ${\alpha }_A$, ${\alpha }_T$, $M_{\theta }$, $\beta$, z 2: input DoFP image 3: $S_T\leftarrow I_0,I_{45},I_{90},I_{135}$ 4: procedure Simulation($S_A$, $S_T$,${\alpha }_A$, ${\alpha }_T$, $M_{\theta }$, $\beta$, z) 5:     $i\leftarrow 0$ 6:     while $i<z$ do 7:          $S_A^{\prime }\leftarrow S_A,{\alpha }_A,M_{\theta },\beta$ 8:          $S_T^{\prime }\leftarrow S_T,{\alpha }_T,M_{\theta },\beta$ 9:          $S_{total}^{\prime }\leftarrow S_A^{\prime }+S_T^{\prime }$ 10:         $I_0,I_{45},I_{90},I_{135}\leftarrow S_{total}^{\prime }$ 11:         $i\leftarrow i+1$ 12:         return $I_0,I_{45},I_{90},I_{135}$ 13:     end while 14: end procedure

4.3. Data for Low-Light Image Dehazing Benchmark

In order to test the performance of the polarization image dehazing algorithm under low-light conditions, this paper also collects a set of low-light polarized hazy image datasets. We construct an artificial enclosed scene, recreate the hazy conditions using a fog machine, and capture hazy images of different targets with a polarimetric camera. The scene is as shown in the Figure 7. This dataset consists of 20 different sets of targets; for each set, low-light hazy images as well as normal-light no-haze images are captured as paired test data. Partial example images are shown in Figure 8.

5. Experimental Setup and Results

5.1. Implementation Details

We implement our framework in PyTorch 1.10. During training, we optimize the parameters using the Adam optimization method with ${\beta }_1=0.9$ and ${\beta }_2=0.999$. The initial learning rate is set to $1\times {10}^{-4}$. We adopt the CosineAnnealing function to adjust the variation of the learning rate, ensuring that it gradually decreases throughout the entire iterative process. The training process comprises a total of 1000 epochs, with the performance on the test set being validated every 20 epochs, and the optimal parameters selected as the final outcome. All experiments are conducted using two NVIDIA GeForce RTX 3090 GPUs.

5.2. Results and Discussion

Since there is no existing work on polarization-based dehazing algorithms for low-light images, we selected some algorithms for low-light (polarization) image enhancement and (polarization-based) image dehazing for comparison with our method, which include the low-light polarized image enhancement method PALLIE, the low-light image enhancement method Zero-DCE, the polarization-based image dehazing method LDP, and single-image dehazing methods C2PNet and Dehazeformer. Specifically, the low-light image enhancement method and the single-image dehazing methods take the intensity image $S_0$ as input, and the polarization-based methods take the polarization images DoFP as input.

In order to test the effectiveness of our proposed polarization-based image enhancement module, we compare it with the low-light polarized image enhancement method PALLIE and low-light image enhancement method Zero-DCE. As shown in Figure 9, all the methods can improve the illumination of the input images. However, the results of Zero-DCE appear to have too high saturation, leading to low image contrast and unclear details. Additionally, the image noise is quite severe. The results from PALLIE look too dark, with significant image noise interference. In contrast, our model not only restores image brightness and contrast, but also removes noise interference, yielding the best overall visual effect. We also conduct quantitative comparison to verify the effectiveness of our method. We use the collected data from the polarization camera in a low-light environment for evaluation. As shown in

Table 1. Quantitative comparison with different enhancement methods. Bold indicates the best.

Method	Amplified	Zero-DCE	PALLIE	Ours
PSNR	21.05	22.26	23.59	37.72
SSIM	0.5234	0.5338	0.7861	0.9429

, our polarization-based image enhancement model achieves the highest score among all methods.

As the existing image dehazing algorithms are all based on images under normal illumination conditions, we compare the effects of our polarization dehazing models with two-step approaches. That is, existing low-light image enhancement algorithms are randomly combined with polarization dehazing algorithms. Zero-DCE takes intensity images as input; hence, we combine it with two single-image dehazing algorithms, C2PNet and Dehazeformer. PALLIE takes polarized images as input, so we combine it with the polarization dehazing algorithm LDP. As shown in Figure 10, the effects of the single-image dehazing algorithms are rather poor. C2Pnet can hardly improve the overall contrast of the image, while Dehazeformer amplifies the noise in the image, resulting in a very poor visual effect. The polarization-based dehazing algorithm LDP does enhance the contrast of the image to some extent, but the dehazing effect is uneven across the image, producing many artifacts, which is not a satisfactory result. Our method is able to restore the texture structures of targets under foggy conditions rather well, enhancing the overall contrast of the image and removing the interference caused by fog. To validate the effectiveness of the proposed dehazing algorithm, we also adopt quantitative evaluation metrics to compare the dehazing effects of different algorithms. As can be seen from the results in

Table 2. Quantitative comparison with different two-step dehazing methods. Here, DCE means Zero-DCE and D-former means Dehazeformer. Bold indicates the best.

	DCE + C2Pnet	DCE + D-Former	PALLIE + LDP	Ours
PSNR	21.07	11.62	19.21	25.38
SSIM	0.4596	0.0563	0.5833	0.8358

, our method shows quite apparent advantages in all metrics.

6. Conclusions

In this paper, we proposed a novel two-stage model for polarization-based image dehazing in low-light environments which contains a polarization-based image enhancement module and a low-light polarization dehazing module. For the image enhancement module, we designed a multi-scale interactive network model and collected a polarization noise dataset in low-light scenarios. For the image dehazing module, we first produced a large number of data samples based on a polarization image simulation algorithm. Subsequently, we designed a structure with multiple sub-modules to realize the extraction and transformation of features from different modalities. The experimental results show that our method has strong generalization and robustness. With the continuous development of the autonomous driving field, the requirements for image processing technologies in special scenarios are getting higher. The algorithm proposed in this paper provides solutions to technical problems encountered in night-time and foggy weather conditions, which has strong application prospects.