An Effective Infrared and Visible Image Fusion Approach via Rolling Guidance Filtering and Gradient Saliency Map

Abstract

To solve problems of brightness and detail information loss in infrared and visible image fusion, an effective infrared and visible image fusion method using rolling guidance filtering and gradient saliency map is proposed in this paper. The rolling guidance filtering is used to decompose the input images into approximate layers and residual layers; the energy attribute fusion model is used to fuse the approximate layers; the gradient saliency map is introduced and the corresponding weight matrices are constructed to perform on residual layers. The fusion image is generated by reconstructing the fused approximate layer sub-image and residual layer sub-images. Experimental results demonstrate the superiority of the proposed infrared and visible image fusion method.

1. Introduction

Infrared and visible light sensors are two kinds of commonly used imaging sensors. The infrared sensor recognizes the target by detecting the thermal radiation difference between the target and the background, and has the ability to identify camouflage, but it is not sensitive to the brightness changes of the scene [1]. Visible light imaging sensors are sensitive to the reflection of the target scene, and the acquired image is usually clear. It can accurately provide the details of the scene where the target is located, but it is vulnerable to light, weather, occlusion, and other factors. According to the respective characteristics of infrared and visible images, the fusion of the two images can make full use of information complementarity, expand the space-time coverage of system target detection, and improve the spatial resolution and target detection capability of the system [2].

There are many methods for image fusion, including multi-resolution analysis (MRA), sparse representation, deep learning, and edge-preserving filtering, etc. In terms of multi-resolution analysis-based methods [3], the Laplacian pyramid, steerable pyramid, DWT, DTCWT, ridgelet transform, contourlet transform, and shearlet transform are widely used in image fusion [4,5,6,7]. Mohan et al. [8] introduced the Laplacian pyramid into the multi-modal image fusion utilizing the quarter shift DTCWT and modified principal component analysis. The Laplacian pyramid is used to decompose the source images into low- and high-components, the quarter shift DTCWT is used to fuse the sub-bands, and the final fused image is obtained through modified principal component analysis. Vivone et al. [9] introduced the pansharpening method via the Laplacian pyramid. Liu et al. [10] introduced the early work on image fusion based on the steerable pyramid. Liu et al. [11] constructed image fusion work using DWT and stationary wavelet transform (SWT). Sulaiman et al. [12] introduced the contourlet transform into pansharpening with kernel principal component analysis and improved sum-modified Laplacian fusion rules. Huang et al. [13] introduced another pansharpening method using the contourlet transform and multiple deep neural networks. Qi et al. [14] introduced the co-occurrence analysis shearlet transform and latent low rank representation for infrared and visible image fusion. Feng et al. [15] introduced the intensity transfer and phase congruency into the shearlet domain for infrared and visible image fusion. Although these algorithms have achieved certain image fusion results, it is easy to lose detail information, and the implementation process of algorithms such as contourlet and shearlet transforms are relatively time-consuming.

Sparse representation-based methods also perform well in image fusion [16,17]. Nejati et al. [18] proposed an image fusion algorithm utilizing dictionary-based sparse representation and Markov random field optimization. Zhang et al. [19] introduced the joint sparse model with coupled dictionary learning for image fusion. The source images were presented with the common sparse component and innovation sparse comments by the over-completed coupled dictionaries; the designed new fusion rule is applied for fusing the sparse coefficients, and the fused image is obtained by using the fused coefficients and coupled dictionaries. Wang et al. [20] introduced the joint patch clustering-based adaptive dictionary and sparse representation for multi-modal image fusion, and this method is robust for dealing with infrared and visible image fusion. These image fusion algorithms utilizing sparse representation have achieved remarkable fusion results in multi-modal image processing.

Deep-learning-based image fusion algorithms have gained unprecedented development and application in recent years. Li et al. [21] introduced a meta learning-based deep framework for infrared and visible image fusion; that framework can accept the source images of different resolutions and generate a fused image of arbitrary resolution just with a single learned model. This method generates the state-of-the-art image fusion results. Cheng et al. [22] introduced a general unsupervised image fusion network based on memory units. This algorithm is applied to infrared and visible image fusion, multi-focus image fusion, multi-exposure image fusion, and multi-modal medical image fusion; qualitative and quantitative experiments on four image fusion subtasks have shown that this method has advantages over the most advanced methods. Zhang et al. [23] proposed an infrared and visible image fusion method using an entropy-based adaptive fusion module and a mask-guided convolutional neural network, and this method generates good performance in qualitative and quantitative assessments. Sun et al. [24] introduced the multiscale network for infrared and visible image fusion. This method generates state-of-the-art fusion performance. Xiong et al. [25] proposed cross-domain frequency information learning via the inception transformer for infrared and visible image fusion, and this method outperforms other fusion approaches in subject and object assessments.

In recent years, base-detail decomposition-based approaches have been introduced into image fusion by some scholars, achieving extraordinary fusion results. These methods decompose the source images into base layers and detail layers, and different fusion rules are performed on the decomposed sub-images [26,27,28]. Guided image filtering was introduced into image fusion using a weighted average strategy by Li et al. in 2013 [29]. Liu et al. [30] introduced the convolutional neural network (CNN) and nuclear norm minimization for image fusion via a rolling guidance filter (RGF). The source images are decomposed by RGF into base- and detail-information, the nuclear norm minimization-based fusion rule is applied for the detail components, and the CNN-based model is used to fuse the base components. Zou et al. [31] introduced guided filter and side window guided filter for image fusion; the visible image is enhanced by an adaptive light adjustment algorithm. This method improves the overall contrast, highlights salient targets, and retains weak details.

According to the above description, the rolling guidance filter has good application prospects in image fusion. In this paper, a novel and effective infrared and visible image fusion algorithm using rolling guidance filtering and gradient saliency map is proposed. We mainly focus on the design of fusion rules for both the approximate and residual layers. The main contributions of the proposed fusion approach are outlined as follows.

(1)

The rolling guidance filter is introduced as the decomposition structure, and the approximate and residual layers of the source images are generated.

(2)

The approximate layers contain most of the background and energy information of the source images, and the energy attribute (EA) fusion strategy is applied to fuse the approximate layers.

(3)

The residual layers contain small gradient textures and noise; the gradient saliency map and corresponding weight matrices are constructed to fuse the residual layers.

(4)

This method is superior to most fusion algorithms and provides an important approach for assisting target detection.

The rest of this paper is arranged as follows. The rolling guidance filtering is described in Section 2. The proposed method is shown in Section 3. The experimental results and discussions are given in Section 4. Finally, the conclusions and future works are presented in Section 5.

2. Rolling Guidance Filtering

Rolling guidance filtering (RGF) is an effective edge-preserving filter with fast convergence [32]. It is widely used in image fusion and image enhancement. Goyal et al. [33] proposed the multi-modal image fusion method using cross bilateral filter and rolling guidance filter. This method can generate good fusion results both visually and quantitatively. Prema et al. [34] introduced multi-scale multi-layer rolling guidance filtering for infrared and visible image fusion. The images are decomposed into micro-scale, macro-scale, and base layers; the phase-congruency-based fusion rule is used to fuse the micro-scale layers, the absolute maximum-based consistency verification fusion rule is used to fuse the macro-scale layers, and the weighted energy related fusion is used to fuse the base layers. This method can visually preserve the background and target information from the source images without pseudo and blurred edges compared to state-of-the-art fusion approaches. Chen et al. [35] proposed an image fusion technique via rolling guidance filtering and the Laplacian pyramid. The source images are separated into structural components and detail components, then the Laplacian pyramid-based model and sum-modified-Laplacian-based model are employed to fuse the structural components and detail components, respectively. This fusion method is superior to other fusion algorithms. Lin et al. [36] introduced rolling guidance filtering and saliency detection for adaptive infrared and visible image fusion technique; this method can improve the contrast and maintain details.

Firstly, the Gaussian filter is utilized to remove the small structures. The input image and output image are defined as I and G, respectively. ${\sigma }_s$ presents the standard deviation of Gaussian filter, and the filter is described as follows [32]:

$$G\left(p\right)=\frac{1}{K_p}{\displaystyle \sum _{q\in N\left(p\right)}\exp \left(-\frac{{‖p-q‖}^2}{2{\sigma }_s^2}\right)}I\left(q\right)$$

(1)

where $p$ and $q$ denote the pixel coordinates in the image, and $K_p$ is given by

$$K_p={\displaystyle {\sum }_{q\in N\left(p\right)}\exp \left(-\frac{{‖p-q‖}^2}{2{\sigma }_s^2}\right)}$$

(2)

where $N\left(p\right)$ denotes the set of neighboring pixels of $p$.

Secondly, the joint bilateral filter is adopted to recover the edge iteratively. The $J^1$ denotes the output of the Gaussian filter, 0$J^{t+1}$ shows the output of the t-th iteration by the joint bilateral filter with the input I and previous iteration value $J^t$. The corresponding formulas are given by:

$$J^{t+1}\left(p\right)=\frac{1}{K_p}{\displaystyle \sum _{q\in N\left(p\right)}\exp \left(-\frac{{‖p-q‖}^2}{2{\sigma }_s^2}-\frac{{‖J^t\left(p\right)-J^t\left(q\right)‖}^2}{2{\sigma }_r^2}\right)}I\left(q\right)$$

(3)

$$K_p={\displaystyle \sum _{q\in N\left(p\right)}\exp \left(-\frac{{‖p-q‖}^2}{2{\sigma }_s^2}-\frac{{‖J^t\left(p\right)-J^t\left(q\right)‖}^2}{2{\sigma }_r^2}\right)}$$

(4)

where $K_p$ is for normalization, ${\sigma }_s$ and ${\sigma }_r$ control the spatial and range weights, respectively.

Finally, the aforementioned two steps can be combined into one by starting rolling guidance simply from a constant-value image. Supposing that $J^t$ is constant C, then the $J^{t+1}$ is updated by the following:

$$J^{t+1}\left(p\right)=\frac{1}{K_p}{\displaystyle \sum _{q\in N\left(p\right)}\exp \left(-\frac{{‖p-q‖}^2}{2{\sigma }_s^2}\right)}I\left(q\right)$$

(5)

In this paper, the filtered image $U$ is defined as follows:

$$U=RGF\left(I,{\sigma }_s,{\sigma }_r,T\right)$$

(6)

where $T$ denotes the number of iterations.

3. The Proposed Method

In this section, a novel infrared and visible image fusion technique based on rolling guidance filtering and gradient saliency map is proposed, and the structure of the proposed algorithm can be divided into four steps: image decomposition, approximate layer fusion, residual layer fusion, and image reconstruction. The flow-process diagram of the proposed method is shown in Figure 1.

• Step 1. Image decomposition

The source images $S_n$ are decomposed by the rolling guidance filtering, and the approximate layers $A_n$ and residual layers $R_n$ can be given by:

$$A_n=RGF\left(I,{\sigma }_s,{\sigma }_r,T\right)$$

(7)

$$R_n=S_n-A_n$$

(8)

where $n\in \left[1,2\right]$ denotes the n-th image.

• Step 2. The approximate layer fusion

The approximate layers present the brightness, energy information, and contrast information of the source images [37,38]. In this section, an energy attribute (EA) fusion model is applied to the approximate layers. The intrinsic property values of the approximate layers are calculated by:

$$I{P}_{A1}=({\mu }_{A1}+M{e}_{A1})/2$$

(9)

$$I{P}_{A2}=({\mu }_{A2}+M{e}_{A2})/2$$

(10)

where $\mu$ and $Me$ show the mean value and the median value of $A_1$ and $A_2$, respectively.

The EA function $E_{A1}$ and $E_{A2}$ are achieved by:

$$E_{A1}\left(x,y\right)=\exp \left(a\left|A_1\left(x,y\right)-I{P}_{A1}\right|\right)$$

(11)

$$E_{A2}\left(x,y\right)=\exp \left(a\left|A_2\left(x,y\right)-I{P}_{A2}\right|\right)$$

(12)

where $\exp$ shows the exponential operator, and $a$ shows the modulation parameter.

The weight maps $W_{A1}$ and $W_{A2}$ are computed by:

$$W_{A1}=\frac{E_{A1}}{E_{A1}+E_{A2}}$$

(13)

$$W_{A2}=\frac{E_{A2}}{E_{A1}+E_{A2}}$$

(14)

The fused approximate layer is computed by the weighted mean:

$$F_A=W_{A1}\times A_1+W_{A2}\times A_2$$

(15)

where $F_A$ represents the fused approximate layer.

• Step 3. The residual layer fusion

The residual layers contain the texture information and some noise of the input images, reflecting the changes of the small gradient so that the gradient saliency maps of residual layers based on the image gradient can be introduced; then, the multi-scale morphological gradient (MSMG) [39] is utilized to construct gradient maps. Compared to residual layers, the input source images has a smaller proportion of noise, so the multi-scale morphological gradient is adopted to obtain gradient features from the input images $S_n$, generating the gradient graphs $MSM{G}_n$. It can preserve the texture information, and the effect of noise can be reduced effectively. The detail steps of MSMG are as follows:

Firstly, the multi-scale structuring elements are computed by:

$$S{E}_j=\underset{j}{\underbrace{S{E}_1\oplus S{E}_1\cdots \oplus S{E}_1}},\hspace{1em}j\in \left\{1,2\cdots n\right\}$$

(16)

where $S{E}_1$ shows the basic structuring element with radius $r$, and $n$ shows the number of scales.

Secondly, the gradient features $G_j$ can be calculated by the morphological gradient operator from image $S$, and the corresponding equation is defined as follows:

$$G_j\left(x,y\right)=S\left(x,y\right)\oplus S{E}_j-S\left(x,y\right)\mathrm{\Theta }S{E}_j,\hspace{1em}j\in \left\{1,2\cdots n\right\}$$

(17)

where $\oplus$ and $\mathrm{\Theta }$ present the morphological dilation and erosion operators, respectively. They are defined as follows:

$$S\oplus SE=\underset{\left(u,v\right)}{\max }\left(S\left(x-u,y-v\right)+SE\left(u,v\right)\right)$$

(18)

$$S\mathrm{\Theta }SE=\underset{\left(u,v\right)}{\min }\left(S\left(x+u,y+v\right)-SE\left(u,v\right)\right)$$

(19)

where $\left(x,y\right)$ and $\left(u,v\right)$ show the pixel coordinate in the image and structuring element, respectively.

Thirdly, MSMG is generated by integrating gradients of all scales, and it is defined as follows:

$$MSMG\left(x,y\right)={\displaystyle \sum _{j=1}^n{w}_j\cdot G_j\left(x,y\right)}$$

(20)

where $w_j$ denotes the weight for the gradients at scale $j$, and it is defined as follows:

$$w_j=\frac{1}{2\times j+1}$$

(21)

In order to expand the influence range of the gradient on weight and achieve the final gradient saliency maps $S_{R_n}$, the Gaussian filter is utilized to diffuse the generated gradient images $MSM{G}_n$. The corresponding equation is defined as follows:

$$S_{R_n}=Gaussian\left(MSM{G}_n,\delta ,r\right)$$

(22)

where $Gaussian\left(\cdot \right)$ denotes the Gaussian filter; $\delta$ and $r$ present the standard deviation and the radius of Gaussian template, respectively. In this section, $\delta$ and $r$ are set to 5.

For the saliency maps $S_{R_n}$ of residual layers, the weight maps $W_{R_1}$ and $W_{R_2}$ of residual layers are calculated by the following:

$$W_{R_1}=\frac{S_{R_1}}{S_{R_1}+S_{R_2}}$$

(23)

$$W_{R_2}=\frac{S_{R_2}}{S_{R_1}+S_{R_2}}$$

(24)

The fused residual layers are computed by the following:

$$F_R=W_{R_1}\times R_1+W_{R_2}\times R_2$$

(25)

where $F_R$ represents the fused residual layer.

• Step 4. Image reconstruction

The fused image $F$ is generated by integrating the fused approximate and residual layers, and the corresponding equation is given by:

$$F=F_A+F_R$$

(26)

4. Experimental Results and Discussion

In order to verify the effectiveness of the proposed infrared and visible image fusion approach in this paper, we selected eight groups of infrared and visible images collected by Liu et al. [40] to test, and the corresponding datasets as shown in Figure 2. The eight image fusion algorithms are selected to compare with our method, which includes DWT [40], DTCWT [40], CVT [40], CSR [41], WLS [42], CNN [43], CSMCA [44], and TEMST [45]. In our framework, the parameters ${\sigma }_s$, ${\sigma }_r$ and $T$ are set to 1, 0.05, and 3, respectively. All the experiments were run on Matlab2018b.

We used subjective and objective evaluation assessments to analyze the fusion results. In terms of the objective evaluation, the following six evaluation metrics are used, which includes mutual information (Q_MI) [46,47,48], the human perception inspired metric (Q_CB) [49], nonlinear correlation information entropy (Q_NCIE) [50], the sum of the correlations of differences (Q_SCD) [51], average pixel intensity (Q_API) [52,53], and standard deviation (Q_SD) [53,54]. The larger the values of the six metrics, the better fusion performance there will be. The experiment results and metrics data are shown in Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7 and

Table 1. The average metrics data with different methods in Figure 8.

	QMI	QCB	QNCIE	QSCD	QAPI	QSD
DWT	2.6059	0.5066	0.8065	1.5492	78.9207	37.1654
DTCWT	2.5294	0.5141	0.8062	1.5716	78.8871	35.4248
CVT	2.3700	0.5119	0.8058	1.5624	78.9175	36.1980
CSR	2.8744	0.5299	0.8073	1.5986	78.9665	32.5813
WLS	2.7274	0.5326	0.8068	1.5844	78.4454	32.8518
CNN	3.0436	0.5365	0.8080	1.5817	84.8334	46.8968
CSMCA	2.7472	0.5230	0.8069	1.5939	79.5881	36.8009
TEMST	2.5803	0.4952	0.8062	1.5664	78.9659	44.7601
Proposed	3.4094	0.5418	0.8093	1.6035	83.8083	38.8459

, respectively.

4.1. Subjective Evaluation

In this section, we will compare the subjective visual effect of the fused images, and Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7 depict the fusion results. Figure 3 denotes the fused images of different methods simulated on the first group of images in Figure 2. From the results, we can notice that all fused images can depict the basic information of the scene, but the DWT method causes the blocking effect. The DTCWT and CVT methods are similar, but the surroundings of the target person in the scene are dark; the image fusion effects of the CSR and WLS methods are poor, and some information is lost. The CNN and CSMCA methods perform well in brightness preservation, and the TEMST method also achieves a certain fusion effect. Our method has an obvious effect on preserving the human body contour and the scene’s details.

Figure 4 shows the fusion images of different approaches simulated on the second group of images in Figure 2. From the fused images, we can see that all fusion image methods can basically express the basic information of the scene, including people, cars, street buildings, etc. The DWT algorithm generates the blocking effect and seriously affects information acquisition in the fused image; the fusion effects of the DTCWT and CVT algorithms are similar, but there is obviously incomplete fusion at the street lamp and the fork. The images generated by the CSR and WLS algorithms are dark, and some brightness information is lost; there are some shadows around people in the image generated by the CNN method. The image generated by the CSMCA algorithm is very dark, and the heat source object in the scene is not obvious; although the brightness information of the heat source object is maintained in the image computed by the TEMST, black blocks appear in some areas of the scene, and some important information is lost. The proposed method has a moderate brightness and clear texture, it is easy to observe the feature information, and the scene information is easy to interpret.

Figure 5 depicts the fused results of different methods simulated on the third group of images in Figure 2. From the results, we can notice that the DWT has a distortion and blocking effect. The DTCWT, CVT, CSR, and CNN generate artifacts, especially at the upper edge of the trees in the scene. The WLS has multiple black spots, poor visual effect and serious information loss. The CSMCA generates a dark fusion image, and it is difficult to capture and observe the person and object information in the scene. The TEMST produces some artifacts, and some areas are dark. Our method has moderate brightness, clear textures, and an easier access to information in the scene.

Figure 6 depicts the fused results of different methods simulated on the fourth group of images in Figure 2. From the results, we can notice that the DWT, DTCWT, and CVT generate the blocking effect and dark regions; the CSR, WLS, CNN, and CSMCA are similar, however, there are still some dark areas that are not easy to observe in detail. The image generated by TEMST is generally dark, including information such as leaves that are difficult to observe, and some information is severely lost. Our method has moderate brightness, clear textures, and an easier access to information in the scene.

Figure 7 depicts the experimental results of different fusion approaches on other infrared and visible images in Figure 2. From top to bottom, the fusion results are DWT, DTCWT, CVT, CSR, WLS, CNN, CSMCA, TEMST, and the proposed method. The experimental results demonstrate that the proposed fusion model has the advantages of maintaining brightness and detailed information.

4.2. Objective Evaluation

In this section, the six metrics are used to evaluate the fusion effects objectively. For each indicator, the indicator scores obtained from different source images using the same fusion model simulation experiment are connected to generate a curve, and the average index value is given on the right side of the legend. From Figure 8, we can conclude that different methods have basically the same change trend in the given measurement. From Table 1, we can conclude that the average value of the metrics Q_MI, Q_CB, Q_NCIE, and Q_SCD computed by the proposed infrared and visible image fusion approach are the best, and that the average value of the metrics Q_API and Q_SD achieved by the proposed method also gains obvious results compared to other state-of-the-art algorithms.

4.3. Application on RGB and Near-Infrared Fusion

In this section, we will extend the proposed method to fuse the RGB and near-infrared images, and the flow chart of the algorithm is shown in Figure 9. The color space conversion between RGB and YUV is applied here. Some experimental results generated by the proposed method are shown in Figure 10. The source images are provided by Vanmali et al., in reference [55]. From the results, we can denote that the proposed method can improve scene visibility, contrast and color perception.

5. Conclusions

This paper introduces an effective infrared and visible image fusion algorithm based on rolling guidance filtering and multi-scale morphological gradients. We constructed the energy attribute fusion model for approximate layers, and the gradient saliency map and weight matrices are constructed to perform on residual layers. In the end, the fusion image can be generated by superposing the fused approximate and residual layers. The experimental results demonstrate good performance and effectiveness compared to other state-of-the art fusion algorithms. We also extended this algorithm to fuse the RGB and near-infrared images demonstrating the effectiveness of improving contrast and color perception. In future work, we will expand the application of this algorithm to multi-focus image fusion, multi-exposure fusion, as well as medical image fusion [56,57,58]. Additionally, due to the good application of convolutional neural networks [59] in image fusion, we will consider the combination of edge-preserving filtering and deep learning in multi-modal image fusion.