Night Vision Anti-Halation Algorithm Based on Different-Source Image Fusion Combining Visual Saliency with YUV-FNSCT

Abstract

In order to address driver’s dazzle caused by the abuse of high beams when vehicles meet at night, a night vision anti-halation algorithm based on image fusion combining visual saliency with YUV-FNSCT is proposed. Improved Frequency-turned (FT) visual saliency detection is proposed to quickly lock on the objects of interest, such as vehicles and pedestrians, so as to improve the salient features of fusion images. The high- and low-frequency sub-bands of infrared saliency images and visible luminance components can quickly be obtained using fast non-subsampled contourlet transform (FNSCT), which has the characteristics of multi-direction, multi-scale, and shift-invariance. According to the halation degree in the visible image, the nonlinear adaptive fusion strategy of low-frequency weight reasonably eliminates halation while retaining useful information from the original image to the maximum extent. The statistical matching feature fusion strategy distinguishes the common and unique edge information from the high-frequency sub-bands by mutual matching so as to obtain more effective details of the original images such as the edges and contours. Only the luminance Y decomposed by YUV transform is involved in image fusion, which not only avoids color shift of the fusion image but also reduces the amount of computation. Considering the night driving environment and the degree of halation, the visible images and infrared images were collected for anti-halation fusion in six typical halation scenes on three types of roads covering most night driving conditions. The fused images obtained by the proposed algorithm demonstrate complete halation elimination, rich color details, and obvious salient features and have the best comprehensive index in each halation scene. The experimental results and analysis show that the proposed algorithm has advantages in halation elimination and visual saliency and has good universality for different night vision halation scenes, which help drivers to observe the road ahead and improve the safety of night driving. It also has certain applicability to rainy, foggy, smoggy, and other complex weather.

1. Introduction

When vehicles meet at night, the abuse of high beams will dazzle the oncoming driver. As a result, drivers cannot see the road ahead for a short period of time, and their judgment regarding their vehicle from both sides and pedestrians between vehicles as well as from behind the oncoming vehicle is decreased, which may easily induce traffic accidents. According to statistics, 30–40 percent of nighttime traffic accidents are related to the abuse of high beam lights, and this proportion is still rising [1].

To prevent traffic accidents caused by the halation phenomenon, setting up green barriers or installing anti-glare plates in the middle of a two-way lane can effectively eliminate halation, but they are limited by the road and other conditions. They are generally only used in urban trunk roads and highways.

To improve the safety of night driving and enhance the visibility of effective information in night vision halation images, night vision anti-halation technology has attracted extensive attention from the research community. Reference [2] installs a polarization film on the front windshield of vehicles, which can effectively block the strong light from high beams. The method is simple and easy but also leads to a sharp reduction in light intensity, making it difficult for the driver to observe information in the dark area. An infrared night vision device [3] is installed on the vehicle to obtain infrared images without halation on road conditions ahead. However, the infrared images are grayscale images with low resolution, a serious lack of detail information, and the visual effects are not ideal. Reference [4] fuses the images collected synchronously using two visible cameras with different exposure apertures in the same road conditions, which expands the image’s dynamic range, effectively weakening the halation phenomenon. However, there are some problems such as incomplete halation elimination and weak information complementarity between visible images.

The different-source image fusion method eliminates halation by taking advantage of the strong complementarity of visible and infrared image information, resulting in a good visual effect of the fusion image. However, Wavelet transform [5] is not anisotropic, so it cannot accurately express the orientation of an image boundary, resulting in a certain degree of blurring of the edge details in the fusion image. Curvelet transform [6] overcomes the shortcomings of Wavelet transform and improves the clarity of the fusion image. However, important information such as pedestrians and vehicles is not significant in the fusion image. Non-subsample Shearlet transform (NSST) [7] decomposes images efficiently and reduces the calculation amount. However, it requires high registration for source images. The Laplacian Pyramid based on sparse representation (LP-SR) [8] highlights the high-frequency information in the image and has some resistance to noise. However, when the source image is too complex, it is easy to cause loss of image detail information. Non-subsampled contourlet transform (NSCT) [9] fully extracts the orientation information from the images. However, the decomposition process is complicated, and the operation efficiency is low. Subspace-based image fusion methods include principal component analysis [10] and independent component analysis [11], which project a high-dimensional source image into a low-dimensional space to improve fusion efficiency. However, the noise of the fusion image is obvious.

The main contributions to the work are as follows: (1) A different-source image fusion algorithm based on visual saliency and YUV-FNSCT is introduced. (2) An improved FT visual saliency detection algorithm is proposed. It quickly locks the salient information in infrared images to improve the salient features of fusion images. (3) A visible and infrared image fusion method combining YUV and FNSCT is designed, where the contour features of images are effectively extracted by FNSCT, and only the luminance component decomposed by YUV transform is involved in image fusion to improve the processing efficiency and avoid the color shift of fusion images. (4) An experimental study is conducted to show the effectiveness and universality of the proposed algorithm. The proposed algorithm is compared with five state-of-the-art algorithms in six typical halation scenes on three types of roads. The experimental outcomes illustrate that the proposed algorithm has advantages in halation elimination and visual saliency and has good universality for different night vision halation scenes.

The remainder of the article is arranged as follows. Section 2 presents the night vision anti-halation principle of infrared and visible image fusion. Section 3 describes the step-by-step design and implementation of the anti-halo fusion algorithm. Section 4 gives the experiential results and discussion. Lastly, Section 5 represents the conclusion.

2. Principle of Night Vision Anti-Halation Based on Different-Source Image Fusion

In a typical nighttime halation scene, the visible image has rich colors. However, the brightness of the halation area is too high, which not only makes the effective information drowned out by strong light but also leads to lower brightness in the non-halation area so that it is difficult to observe important information in the dark area. The infrared image can better reflect thermal properties and is not affected by the halation, but it is a grayscale image with low resolution, so the features of scenes such as details and textures are often lost [12]. Based on the strong complementarity of two different-source images [13], a night vision anti-halation algorithm based on image fusion is proposed. The original images and the fusion image in the halation scene are shown in Figure 1.

In order to highlight important information such as pedestrians and vehicles, the visual saliency detection of infrared images is carried out to quickly lock the area of interest to human eyes and extract complete salient contours so as to improve the salient features of night vision anti-halation fusion images.

To effectively retain the colors of the visible image, it is decomposed into luminance Y and chrominance U and V by YUV transform. Since the halation information in the visible image is mainly distributed in the luminance Y [14], it is necessary to perform the anti-halation fusion process only for the luminance Y and the infrared salient images. In this way, it not only reduces the amount of computation but also avoids the color distortion caused by the chrominance U and V participating in the fusion [15].

In order to make a fusion image with richer spatial details, higher definition, and less spectral distortion, the fast non-subsampled contourlet transform (FNSCT) is used to decompose and reconstruct the luminance Y and the infrared salient image. An FNSCT that has good multi-direction, multi-scale, and shift-invariance can effectively extract the image contour features and further improve the sharpness of the fused image. The low-frequency decomposition sub-band contains the main contour and halation of the image. The nonlinear adaptive fusion strategy of low-frequency weight is designed to eliminate halation while preserving the contours of two original images. The high-frequency sub-bands reflect image details such as edges and textures. The statistical matching feature strategy is used to fuse high-frequency sub-bands, which can distinguish between the common and unique edge information by mutual matching, so as to obtain more and more effective information from the original image. The overall implementation block diagram is shown in Figure 2.

3. Realization of Image Fusion Combining Visual Saliency with YUV-FNSCT

According to the above principle, the image fusion algorithm combining visual saliency with YUV-FNSCT is mainly divided into visual saliency detection of an infrared image, YUV transformation, FNSCT decomposition and reconstruction, and low-frequency and high-frequency fusion strategies.

3.1. Visual Saliency Detection of Infrared Images

Based on the human visual system, visual saliency detection quantifies the interest degree of each region to a gray level to obtain the visual saliency image. Therefore, it can accurately identify the visual region of interest in the image [16]. The improved Frequency-turned (FT) algorithm proposed in this paper is used to detect the visual saliency of infrared images and extract the complete contours and edges.

As a spatial frequency domain saliency algorithm, the FT algorithm first performs Gaussian low-pass filtering, and then calculates the Euclidean distance between each pixel and the average value of all pixels in the CIE Lab color space as the significance value [17]. This algorithm is simple and efficient and can output full resolution images while highlighting salient objects. However, the segmentation of salient information boundaries is not clear, so the difference between the generated salient area and the background gray value is small, resulting in an unsatisfactory effect of detecting salient targets.

To improve the detection accuracy of salient targets in images, the improved FT algorithm firstly extracts the edge features of an infrared image, and then normalizes and weighs each feature value to obtain the significant image. As a result, the algorithm can effectively improve the saliency value, suppress background noise, and enrich the edge details for the infrared image.

The Gaussian low-pass filter is used to truncate the high-frequency information from the infrared image I (x, y) and retain the low-frequency part reflecting the significant features. The infrared smoothed image I_whc(x, y) after filtering can be expressed as

$$I_{w_{hc}}(x,y)=h(x,y)\ast I(x,y)$$

(1)

where h (x, y) is the two-dimensional Gaussian function, (x, y) is the coordinate of the image pixel, and ‘⁎’ is the convolution operator. h (x, y) is expressed as

$$h(x,y)=e^{-\frac{x^2+y^2}{2{\sigma }^2}}$$

(2)

The Sobel operator with accurate edge localization and high noise immunity is used for edge extraction of infrared images. The weight of a pixel can be determined according to the distances from its neighboring pixels to achieve edge sharpening and detection [18]. The horizontal gradient g_x(x, y) and the vertical gradient g_y(x, y) of the image are expressed as

$$\{\begin{array}{l}{g}_x(x,y)=m_x\times I_{w_{hc}}(x,y)\\ g_y(x,y)=m_y\times I_{w_{hc}}(x,y)\end{array}$$

(3)

where m_x, m_y are the convolution kernels of edge detection in the horizontal and vertical directions, respectively, and are taken as

$$m_x=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right], m_y=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right]$$

(4)

The gradient image g(x, y) reflecting edge features of the Gaussian smoothed image is expressed as

$$g(x,y)=\left|g_x(x,y)\right|+\left|g_y(x,y)\right|$$

(5)

Since the infrared Gaussian smoothed image I_whc(x, y) is a grayscale image, the pixel values of the color components a and b are constant in the CIE Lab color space. Therefore, only the luminance L(x, y) is decomposed and expressed as [19]

$$L(x,y)=\{\begin{array}{l}116I_{whc}{(x,y)}^{\frac{1}{3}}-16\begin{array}{cc},& I_{whc}(x,y)>{(\frac{6}{29})}^3\end{array}\\ \frac{24389}{27}{I}_{whc}(x,y),\begin{array}{cc}& \end{array}\mathrm{others}\end{array}$$

(6)

The luminance L and the gradient g have different value ranges (L∈[0, 100], g∈[0, 255]), resulting in more prominent gradient features of the image. Therefore, L and g must be normalized [20]. The normalized luminance L^⁎(x, y) and gradient g^⁎(x, y) are expressed as

$$\{\begin{array}{l}{L}^{\ast }(x,y)=\frac{L(x,y)-L_{\min }}{L_{\max }-L_{\min }}\\ g^{\ast }(x,y)=\frac{g(x,y)-g_{\min }}{g_{\max }-g_{\min }}\end{array}$$

(7)

where L_min and L_max are the minimum and maximum of the luminance L, respectively. g_min and g_max are the minimum and maximum of the gradient g, respectively.

The infrared visual saliency image S(x, y) obtained by the improved FT algorithm is

$$S(x,y)=\sqrt{w_L{(L(x,y)-L_u)}^2+w_g{(g(x,y)-g_u)}^2}$$

(8)

where L_u and g_u are the arithmetic means of the normalized luminance L^⁎ and gradient g^⁎, respectively. w_L and w_g are the weight coefficients of the luminance L and gradient g, respectively.

The luminance L has a greater influence on the visual saliency image and should be assigned a larger weight. In this paper, w_L and w_g are set to 0.7 and 0.3 by calculation and optimization.

The processing flow of the improved FT algorithm to obtain infrared visual salient images is shown in Figure 3.

Taking the infrared image using Figure 1b as an example, the comparison of the visual saliency detection process between the improved FT and the traditional FT is shown in Figure 4.

3.2. FNSCT Decomposition

The visible image is transformed from RGB to a YUV color space, and the luminance Y and the chrominance U and V are obtained [14]. The color space model transformation from RGB to YUV is defined as

$$\left[\begin{array}{l}Y\\ U\\ V\end{array}\right]=\left[\begin{array}{ccc}0.299& 0.587& 0.114\\ -0.169& -0.331& 0.5\\ 0.5& -0.419& -0.081\end{array}\right]\left[\begin{array}{l}R\\ G\\ B\end{array}\right]+\left[\begin{array}{l}0\\ 128\\ 128\end{array}\right]$$

(9)

The luminance Y and the infrared saliency image S are decomposed by FNSCT to obtain the corresponding low-frequency and high-frequency sub-band images.

FNSCT is a simple and fast NSCT in the frequency domain, which replaces the tree-composed filter bank of NSCT with a multi-channel filter bank, and can effectively eliminate the pseudo-Gibbs phenomenon and solve the frequency aliasing of the contourlet transform. Therefore, it can retain the important information from the illumination Y and the infrared saliency image S such as edges, textures, and orientations.

NSCT is composed of the non-subsampled pyramid (NSP) and the non-subsampled directional filter bank (NSDFB) [21,22]. Image decomposition needs to be processed by NSP and NSDFB step by step. This iterative filtering has a large amount of computation and low efficiency. FNSCT reconstructs the tree filter banks of NSCT into the multi-channel filter banks R(z). The low-frequency channel filters $R_{i,0}^L$(z) are defined as

$$R_{i,0}^L(z)=H_0(z)\ast H_0(z^2)\ast \cdot \cdot \cdot \ast H_0(z^{2^{i-1}})$$

(10)

where H₀($z^{2^{i-1}}$) denotes the low-pass filter in the NSP and i is the decomposition level of FNSCT.

The high-frequency channel filters $R_{i,j}^H$(z) are defined as

$$R_{i,j}^H(z)=H(z)\ast H_1(z^{2^m})\ast U(z)$$

(11)

where H(z) and U(z) are denoted as

$$H(z)=\{\begin{array}{l}1 , m=0\\ H_0(z)\ast H_0(z^2)\ast \cdot \cdot \cdot \ast H_0(z^{2^{m-1}}), m\ge 1\end{array}$$

(12)

$$U(z)=U_v(z)\ast U_v(z^2)\ast \cdot \cdot \cdot \ast U_v(z^{2^{i-m-1}})$$

(13)

where H₁(z^2m) denotes the high-pass filter in NSP. U_ν($z^{2^{i-m-1}}$) denotes the dual-channel fan filter in NSDFB and satisfies U₁($z^{2^{i-m-1}}$) =1 − U₀($z^{2^{i-m-1}}$). Subscript v is 0 or 1. m = 0, 1, 2, …, i − 1. j is the direction number of high-frequency decomposition and is expressed as [23]

$$j=2^{i-m}+{\displaystyle \sum _{c=0}^{i-m-1}v\times 2^{i-m-1-c}}-1$$

(14)

The multi-channel filter bank R(z) convolves multiple filters in each channel of NSCT into one filter, which reduces the computational complexity and improves the decomposition efficiency. The sampling frequency and the size of the frequency domain filtering kernel are fixed in FNSCT, and the number of filters is reduced from 2ⁱ⁺² − 4 to 2ⁱ⁺¹ − 1. Figure 5 shows the sub-band image $D_{2,1}^H$(x, y) is decomposed in the first direction of the high-frequency channel by two-level FNSCT. The blue region denotes the filter passband, and the orange region denotes the decomposed sub-band images. ‘⁎’ represented as convolution operator.

The FNSCT filters are formed by the convolution of the NSCT filters; thus, the filter passband of each channel in FNSCT is an intersection of the frequency bands of the corresponding channel filters in NSCT.

A comparison of the decomposition of two-level NSCT and FNSCT in each direction is shown in Figure 6.

From Figure 6, filter convolution is faster and more efficient than multiple filters. The FNSCT filter has the same frequency domain and filtering characteristics as the NSCT filter. Let ψ be the passband of filter bank R(z). It is satisfied that

$${\displaystyle \sum _{n=1}^{2^{j+1}-1}\left|{\psi }_n\right|}=1$$

(15)

The luminance Y and the infrared saliency image S are decomposed by multi-channel filters R(z) to obtain the corresponding low-frequency sub-bands $Y_0^L$(x, y) and $S_0^L$(x, y) and the high-frequency sub-bands $Y_{i,j}^H$(x, y) and $S_{i,j}^H$(x, y). Each sub-band image can be obtained by filtering only once, which improves the decomposition efficiency.

3.3. Nonlinear Adaptive Fusion Strategy of Low Frequency

The low-frequency sub-bands in the luminance Y contain most of the halation information, and the gray value of the halation areas is significantly higher than that of the non-halation areas [24]. In order to eliminate halation completely, we designed a nonlinear adaptive fusion strategy with a low-frequency weight.

The information from halation areas in the low-frequency fusion image is mainly taken from the infrared saliency image S, while that of the non-halation areas is from the synthesis of Y and S [25,26]. The constructed infrared low-frequency coefficient weight α(x, y) is defined as

$$\alpha (x,y)=\frac{1}{2\pi }{\tan }^{-1}2\left[\frac{c}{255}{Y}_0^L(x,y)-c_H\right]+c_{IL}$$

(16)

where $Y_0^L$(x, y) is the low-frequency coefficient of the luminance Y. c_H is the critical halation value in the low-frequency coefficient matrix of Y. c_IL is the low-frequency coefficient of the infrared salient image S at the critical value. Y maps to the interval [a, b], and c is the length of the interval. After calculating and optimizing many times, the optimal fusion result was reached when c_H was 3.5 and c_IL was 0.75. The interval [a, b] is taken as [0, 6], and the corresponding relation curve between $Y_0^L$(x, y) and α(x, y) is shown in Figure 7.

According to Figure 7, in the interval [0, 255], the weight α increases with the increase in $Y_0^L$, indicating that the higher the pixel brightness in the low-frequency sub-band of the luminance Y, the greater the weight of the infrared visual saliency image. In the halation, $Y_0^L$(x, y) is greater than the critical grayscale value of halation, and the weight α is close to 1. That is, the information from halation areas in the low-frequency sub-band fusion image is mainly taken from the infrared saliency image S, avoiding the high-brightness halation in the visible image to participate in the fusion, so as to achieve a more complete elimination of the halation. In the non-halation areas, S is given about half of the weight. The information from non-halation areas is from the synthesis of the detail information of Y and S. At the halation boundary, the smooth curve with nonlinear continuous change is used to transition from the halation area to the non-halation area so as to avoid the light and dark fusion boundary in the low-frequency fusion image.

The nonlinear adaptive fusion strategy of low-frequency weight is defined as

$$F_0^L(x,y)=\alpha (x,y)S_0^L(x,y)+[1-\alpha (x,y)]Y_0^L(x,y)$$

(17)

where $F_0^L$(x, y) is the low-frequency coefficient of the fusion image.

The nonlinear adjustment strategy of low-frequency weight not only eliminates the halation completely but also retains the useful information from the two original images to the maximum extent.

3.4. Statistical Matching Feature Fusion Strategy of High Frequency

The high-frequency sub-bands reflect the image detail information such as the edges and textures [27]. The high-frequency sub-bands of the luminance Y are quite different from that of the infrared saliency image S; thus, the statistical matching feature strategy is adopted to enhance the textures and other details of the fusion images [28]. The high-frequency sub-band fusion is expressed as

$$F_{i,j}^H(x,y)=\{\begin{array}{l}{Y}_{i,j}^H(x,y),\begin{array}{c}\end{array}{Y}_{i,j}^H(x,y)>S_{i,j}^H(x,y)\\ S_{i,j}^H(x,y),\begin{array}{c}\end{array}{S}_{i,j}^H(x,y)>S_{i,j}^H(x,y)\end{array}$$

(18)

where $F_{i,j}^H$(x, y), $Y_{i,j}^H$(x, y), and $S_{i,j}^H$(x, y) are the high-frequency sub-bands of the fusion image, the luminance, and the infrared saliency image, respectively.

The statistical matching feature strategy can distinguish between the common and the unique edge information by mutual matching to obtain more effective details from Y and S.

3.5. FNSCT Reconstruction

Similar to FNSCT decomposition, the reconstruction filters are designed for each channel to form the reconstruction multi-channel filter bank r(z). The fused high- and low-frequency sub-bands were convolved with the corresponding reconstructed multi-channel filters, respectively, and then summed to obtain the reconstructed image. The reconstructed new luminance Y′ is

$$Y^{\prime }=F_0^L(x,y)\ast r_0^L(z)+{\displaystyle \sum _{n=2}^{2^{i+1}-1}{F}_{i,j}^H(x,y)\ast r_{i,j}^H}(z)$$

(19)

where $r_0^L$(z) and $r_{i,j}^H$(z) are the low-frequency and the high-frequency reconstructed multi-channel filters, respectively.

In order to completely reconstruct the decomposition sub-bands of FNSCT, the decomposition multi-channel filters and the reconstruction multi-channel filters should satisfy the following relation:

$$R(z)+r(z)=1$$

(20)

The new luminance Y′ and the original chromaticity U and V generate an RGB color image conforming to human visuals by YUV inverse transform [14]. The color space model transformation from YUV to RGB is defined as

$$\left[\begin{array}{l}R\\ G\\ B\end{array}\right]=\left[\begin{array}{ccc}1& 0& 1.402\\ 1& -0.34414& -0.71414\\ 1& 1.772& 0\end{array}\right]\left[\begin{array}{l}Y\\ U\\ V\end{array}\right]$$

(21)

The new luminance Y′ and the night vision anti-halation fusion image obtained from the original image in Figure 1 is shown in Figure 8.

4. Experimental Results and Analysis

Considering both the night driving environment and the degree of halation, the effectiveness and universality of the proposed algorithm are verified. We collected infrared and visible videos on three types of roads covering most night driving, i.e., urban trunk road, residential road, and suburban road. In addition, the videos include vehicle fast-speed videos and vehicle slow-speed videos of each road. The halation areas in the videos change from small to large and then smaller as the vehicles come closer. We obtained more than 6200 original infrared and visible images from the videos collected.

The proposed algorithm is compared with Wavelet [5], Curvelet [6], NSST [7], LPSR [8], and FNSCT [23]. The quality of the fusion images obtained by the six algorithms above is evaluated by the adaptive partition quality evaluation method of night vision anti-halation fusion images [29]. The degree of halation elimination D_HE is used to evaluate the anti-halation effect in the halation areas. The greater the value, the more complete the halation elimination. In non-halation areas, seven objective indexes, including mean µ, standard deviation σ, average gradient AG, spatial frequency SF, cross-entropy CE_FU-VI and CE_FU-IR, and edge preservation Q^AB/F, were used to evaluate the quality of fusion images in terms of clarity, color richness, and visual saliency, respectively.

μ represents the average gray of pixels, and, the greater the value, the higher the average brightness. σ indicates the dispersion of the image grayscale relative to μ, and, the higher the σ, the larger the contrast. AG represents the change rate of the image detail contrast. The higher the AG, the clearer the image. SF reflects the overall activity of the image in the spatial domain. The larger the SF, the better the fusion effect. CE_FU-VI and CE_FU-IR reflect the differences of the corresponding pixels between the fusion image and the original visible as well as that between the fusion image and infrared images, respectively. The lower the value, the more information the fusion image obtains from the original image. Q^AB/F reflects the degree to which the fusion image maintains the edge of the original image. The greater the value, the more useful information such as contour and texture can be extracted from the original image.

4.1. Urban Trunk Road

There are streetlamps and scattered light from surrounding buildings on the urban trunk road, and the oncoming car is running with low beams from far to near, forming small halation scenes (Scene 1) and large halation scenes (Scene 2). In the visible image, the roads near the halation areas can be seen, but the pedestrians and road conditions in dark areas are difficult to observe. In the infrared image, the outlines of the car and pedestrians can be seen, but the color information is missing.

The original images and the fusion images of each algorithm in Scene 1 and Scene 2 are shown in Figure 9 and Figure 10, respectively.

Table 1. Objective indexes of fusion images in Scene 1.

Algorithm	DHE	μ	σ	AG	SF	CEFU-VI	CEFU-IR	QAB/F
Wavelet	0.491	42.632	15.169	2.769	7.465	0.691	3.408	0.373
Curvelet	0.535	43.354	17.693	5.585	15.040	0.720	3.701	0.305
NSST	0.517	44.108	15.399	3.116	13.091	0.614	1.650	0.376
LPSR	0.532	45.055	19.172	5.262	15.385	0.532	1.995	0.314
FNSCT	0.588	70.094	18.458	5.255	15.618	0.652	1.802	0.292
Ours	0.637	76.007	19.499	5.631	15.642	0.524	0.981	0.492

and

Table 2. Objective indexes of fusion images in Scene 2.

Algorithm	DHE	μ	σ	AG	SF	CEFU-VI	CEFU-IR	QAB/F
Wavelet	0.811	48.978	24.259	2.972	9.884	0.635	2.728	0.362
Curvelet	0.819	51.982	25.849	4.761	15.754	0.552	2.491	0.386
NSST	0.785	53.044	23.471	3.109	9.779	0.490	3.234	0.405
LPSR	0.833	57.572	31.197	7.105	23.809	0.483	2.137	0.179
FNSCT	0.804	54.926	25.837	5.510	18.517	0.468	2.464	0.543
Ours	0.866	64.109	34.338	9.526	28.204	0.330	1.814	0.684

show the objective evaluation indexes of the fusion images in Scene 1 and Scene 2, respectively.

From Figure 9, Wavelets and NSST algorithms cannot eliminate halation completely. The fusion image is dark overall, and the details of salient targets such as vehicles and pedestrians are fuzzy. Curvelet and LPSR algorithms improve the image sharpness, but the halation elimination is still incomplete. The FNSCT algorithm enhances the brightness and contrast of the background of the fusion image, and the halation elimination is more complete. However, the fusion traces between halation and non-halation areas are obvious. The proposed algorithm completely eliminates the halation formed by the headlights, and the color of the fusion image is more natural and bright.

From Figure 10, the Wavelet algorithm eliminates most of the halation. However, local halation still exists, the color shift is serious, and the edge of the vehicle and the background is fuzzy. The Curvelet algorithm improves the clarity of the fusion image. However, there are obvious fusion traces at the junction of the halation and non-halation areas. The NSST algorithm enhances the contour details of vehicles and pedestrians but enlarges halation range, and the non-halation area is dark. The LPSR algorithm enhances the brightness of salient targets and eliminates halation around headlights. However, the road surface has obvious light–dark alternating shadows. The FNSCT fusion image has better halation elimination effects. However, the details of trees and background buildings are lost in non-halation areas. The proposed algorithm eliminates halation completely. The textures of vehicles, pedestrians, lane lines, and road edges are clear, and the visibility of background buildings and trees is greatly improved, which is more suitable for human visual perception.

It can be seen from Table 1 and Table 2 that the D_HE of the proposed algorithm is the highest, indicating that the halation elimination is the most thorough in the fusion images. µ, σ, AG, SF, and Q^AB/F are the largest, and CE_FU-VI and CE_FU-IR are the smallest, indicating that the overall visual effect is best in the non-halation areas of the fusion image.

In conclusion, the proposed algorithm not only eliminates the high-brightness halation information but also moderately enhances the visual effect of the non-halation area.

4.2. Residential Road

There is weak scattered light from buildings on the residential road. The overall road environment is dark, and the cars are running with low beams. When the cars are far away, the halation area is small in the visible image (Scene 3). There is little information about pedestrians and road conditions next to cars in the dark areas. The infrared image is not influenced by halation, and the outlines of vehicles and pedestrians are clear. However, the details of the background and both sides of the road are missing. When multiple oncoming cars are close, there are more halation parts in the visible image, and the halation area is larger (Scene 4). The license plate of the previous car and the road conditions ahead can be seen clearly. However, the large-area high-brightness halation makes the information in the dark area more difficult to observe, and it is almost impossible to see the pedestrians and road conditions near the cars. In the infrared image, cars and pedestrians can be seen clearly. The original images and the fusion images of each algorithm in Scene 3 and Scene 4 are shown in Figure 11 and Figure 12, respectively.

Table 3. Objective indexes of fusion images in Scene 3.

Algorithm	DHE	μ	σ	AG	SF	CEFU-VI	CEFU-IR	QAB/F
Wavelet	0.530	52.692	18.824	1.573	5.436	0.758	5.462	0.289
Curvelet	0.527	70.969	21.333	3.490	10.794	0.680	5.568	0.324
NSST	0.756	67.185	14.377	2.903	7.989	0.607	2.913	0.324
LPSR	0.523	91.858	20.825	4.296	13.082	0.630	4.779	0.270
FNSCT	0.590	77.938	19.953	3.486	11.085	0.480	3.814	0.336
Ours	0.654	119.116	22.430	4.740	14.016	0.535	2.502	0.532

and

Table 4. Objective indexes of fusion images in Scene 4.

Algorithm	DHE	μ	σ	AG	SF	CEFU-VI	CEFU-IR	QAB/F
Wavelet	0.576	60.143	24.594	4.073	10.911	0.730	3.799	0.297
Curvelet	0.629	70.588	25.611	6.240	17.944	0.500	3.486	0.261
NSST	0.563	80.203	32.329	3.324	9.277	1.418	5.480	0.341
LPSR	0.570	83.572	32.746	6.290	18.371	1.441	5.157	0.260
FNSCT	0.648	42.094	22.744	5.552	17.131	0.943	2.549	0.379
Ours	0.702	71.566	32.964	7.643	21.273	0.446	1.616	0.442

show the objective evaluation indexes of the fusion images in Scene 3 and Scene 4, respectively.

From Figure 11, the Wavelet fusion image has serious halation, a dim background, and blurred contours. In the Curvelet fusion image, the edges of vehicles and pedestrians are enhanced, but the halation is enlarged. The NSST algorithm reduces the halation range. However, the color information is severely lost, and the noise is obvious. The LPSR algorithm enhances background brightness, but the halation elimination is poor. The FNSCT algorithm eliminates most of the halation. However, the detailed information such as the buildings in the background and the trees along the road is lost. The proposed algorithm eliminates halation more thoroughly. The brightness of the non-halation area is improved significantly, and the features of important information such as vehicles and pedestrians are remarkable. The visual effect is more desirable.

As can be seen from Table 3, the CE_FU-VI of the proposed algorithm is only slightly higher than that of the FNSCT, but the CE_FU-IR is much lower than that of the FNSCT. The reason is that the algorithm in this paper retains more details and textures in the infrared image so as to completely eliminate halation of the visible image. The actual visual effect of the fusion image is better. The remaining indexes are better than that of the other algorithms.

From Figure 12, the Wavelet and Curvelet fusion images still have halation. NSST and LPSR algorithms improve the color saliency and clarity of fusion images, yet the halation is still obvious. The FNSCT fusion image has better halation elimination and enhanced details. However, the overall brightness is low. The algorithm in this paper eliminates the halation completely. The license plate is clear, and the information such as pedestrians and background buildings are enhanced moderately.

Table 4 shows that the µ of the proposed algorithm is lower than that of the NSST and the LPSR. This is because the NSST and LPSR enlarge the halation range, which enhances the brightness of the non-halation area, resulting in the smallest D_HE of them all. The remaining indexes are better than that of the other algorithms.

Comparing the results of Scene 3 and Scene 4, it can be seen that as the halation area of the visible image becomes larger, the halation degree becomes more serious, and the proposed algorithm has better halation elimination, more significant color and details, and a better overall visual effect.

4.3. Suburban Road

There are almost no other light sources on the suburban roads except for the headlights, and the overall illumination is very low. The car is running with high beams. When the car is far away, the halation area is small (Scene 5). When the car is close, the halation area is large and the brightness is high (Scene 6), and it is easy to make the driver opposite blinded, resulting in traffic accidents. In the infrared images, the edge contours of the car and pedestrians can be seen, yet the clarity is low.

The original images and the fusion images of each algorithm in Scene 5 and Scene 6 are shown in Figure 13 and Figure 14, respectively.

Table 5. Objective indexes of fusion images in Scene 5.

Algorithm	DHE	μ	σ	AG	SF	CEFU-VI	CEFU-IR	QAB/F
Wavelet	0.494	49.252	18.716	2.691	7.185	1.802	2.937	0.377
Curvelet	0.548	50.143	19.719	4.575	11.248	1.294	2.819	0.298
NSST	0.562	49.327	19.345	2.528	6.787	1.452	2.993	0.383
LPSR	0.631	52.091	20.468	4.812	9.243	1.941	1.635	0.322
FNSCT	0.651	49.772	22.366	4.419	11.141	1.232	1.477	0.312
Ours	0.758	54.959	24.688	5.585	12.251	1.227	1.151	0.480

and

Table 6. Objective indexes of fusion images in Scene 6.

Algorithm	DHE	μ	σ	AG	SF	CEFU-VI	CEFU-IR	QAB/F
Wavelet	0.609	103.388	26.888	1.572	7.345	1.769	1.387	0.463
Curvelet	0.605	94.801	25.097	5.286	13.112	1.017	1.472	0.513
NSST	0.694	84.880	22.453	3.087	10.562	1.383	1.383	0.469
LPSR	0.738	89.825	21.522	4.575	18.267	1.474	1.270	0.406
FNSCT	0.716	99.707	25.520	3.102	13.056	0.774	2.542	0.539
Ours	0.886	106.406	28.204	6.380	21.783	0.744	0.879	0.678

show the objective evaluation indexes of the fusion images in Scene 5 and Scene 6, respectively.

From Figure 13 and Figure 14 and Table 5 and Table 6, the car and pedestrians in the Wavelet fusion image are blurred, and the halation is obvious in both scenes. In Scene 5, the Curvelet and LPSR fusion images have a great amount of background noise. The license plate is fuzzy in the NSST fusion image. The pedestrian details are missing in the FNSCT fusion image. The proposed algorithm eliminates halation thoroughly, and the details of the road surface and background are richer in the fusion image. In Scene 6, the Curvelet and NSST algorithms improve the clarity of the fusion images, yet the halation elimination is insufficient. The LPSR and FNSCT algorithms further eliminate the halation. However, the background of the LPSR fusion image is dark, and the FNSCT fusion image is blurred overall. The algorithm in this paper eliminates not only the strong halation of the headlights but also the reflected light of the road surface. The obtained fusion image has significant contour features, richer color and texture, and the best visual effect.

4.4. Discussion

From the above experimental results and analysis, it can be seen that there are significant differences in the evaluation indexes of the fusion images obtained by different algorithms, indicating that the adopted evaluation system can reflect the quality of night vision anti-halation fusion images from different aspects and the advantages and disadvantages of different algorithms.

To evaluate the quality of fusion images with more confidence and validity, the eight indexes are normalized using Min-Max Normalization, and then summed to obtain a comprehensive index for comparative evaluation. The normalized evaluation index I^* is expressed as

$$I^{\ast }=\frac{I-I_{\min }}{I_{\max }-I_{\min }}$$

(22)

where I is the original evaluation index and I_max and I_min are the maximum and minimum of the same index for different algorithms. For the convenience of data analysis, I_min is taken as 0.

The histogram of normalized indexes of the fusion images in six scenes is shown in Figure 15.

The comprehensive index reflects the overall quality of the fusion image. The higher the bars, the better the image fusion quality and the more consistent with human visual perception.

From Figure 15, Wavelet and NSST algorithms have a low ability to process halation images in each scene. The Curvelet algorithm is more suitable for processing images in a large halation scene. The LPSR and FNSCT algorithms have almost the same comprehensive index, indicating that the fusion effect of the two algorithms is similar. The algorithm in this paper has the best comprehensive index in each halation scene, indicating that the proposed algorithm has advantages in halation elimination and visual saliency and has good universality for different night vision halation scenes.

The urban trunk road has scattered light from streetlamps and surrounding buildings, which can enhance the visibility of the whole fusion image. So, the comprehensive index of the fusion image of the urban trunk road is higher than that of the other two roads. On the residential road, the road environment is dim. When the cars and pedestrians are far away, the halation area is small and the salient targets are inconspicuous, so the brightness is low on the non-halation area of the fusion image, and the comprehensive index of each algorithm is lower than that of the urban trunk road. The overall illumination is very low on the suburban road, and the comprehensive index of each algorithm is the lowest.

On the same road, as the oncoming cars approach, the halation area becomes larger, and the large halation scene can enhance visibility surrounding the halation. Therefore, the comprehensive index of the large halation scene is higher than that of the small halation scene.

In conclusion, both the nighttime ambient illumination and the degree of halation will affect the visual effect of the fusion image.

5. Conclusions

To solve the problem of drivers being blinded by nighttime halation, this paper proposes a night vision anti-halation algorithm-based different-source image fusion. The improved FT saliency detection algorithm is used to extract the salient information from infrared images, which effectively highlights the objects of interest in fusion images, such as cars and pedestrians. Only the luminance is involved in anti-halation fusion by YUV transform, which improves the efficiency of the algorithm and avoids color distortion of the fusion image. FNSCT decomposition and reconstruction improves clarity and details in the fusion image and reduces spectral distortion. The experimental results show that the fusion images of the algorithm in this paper have complete halation elimination, rich color and details, and prominent saliency information. The proposed algorithm has good universality for different halation scenes and is helpful for night driving safety. It also has certain applicability to rainy, foggy, smoggy, and other complex weather.