DAGANFuse: Infrared and Visible Image Fusion Based on Differential Features Attention Generative Adversarial Networks

Abstract

The purpose of multi-modal visual information fusion is to integrate the data of multi-sensors to generate an image with higher quality, more information, and greater clarity so that it contains more complementary information and fewer redundant features. Infrared sensors detect thermal radiation emitted by objects, which is related to their temperature, whereas visible light sensors generate images by capturing the light that interacts with objects, including reflection, diffusion, and transmission. However, due to the different principles of infrared and visible light sensors, there is a large similarity difference between the generated infrared and visible images, which makes it difficult to extract complementary information. Existing methods generally use simple splicing or addition methods to fuse features at the fusion layer without considering the intrinsic features of different modal images and the interaction of features between different scales. Moreover, only correlation is considered. On the contrary, the image fusion task needs to pay more attention to their complementarity. For this reason, we introduce a cross-scale differential features attention generative adversarial fusion network, namely DAGANFuse. In the generator, we designed a cross-modal differential features attention module to fuse the intrinsic content of different modal images. We proposed a parallel path calculation of differential features and fusion features for attention weights and performed parallel spatial and channel attention weight calculations on the two paths. In the discriminator, a dual discriminator was used to maintain the information balance between different modalities and avoid common problems such as information blurring and loss of texture details. Experimental results show that our DAGANFuse has state-of-the-art (SOTA) performance and is superior to existing methods in terms of fusion performance.

1. Introduction

Infrared sensors generate images by capturing the thermal radiation information of objects. Even under extreme conditions, harsh weather, and partial occlusion, they can effectively highlight significant targets. However, infrared images cannot provide sufficient environmental information such as texture details and environmental illumination. In contrast, visible light sensors generate images by capturing the light that interacts with objects, including reflection, diffusion, and transmission. This interaction results in rich texture detail information, making visible images more consistent with human visual perception. The fusion of infrared and visible light images not only solves the problems of blurred backgrounds and missing details of infrared sensors but also overcomes the dependence of visible light sensors on environmental conditions, constructing images with both thermal radiation sources and multispectral information. The complementary characteristics of infrared images and visible images have enabled the fusion technology of these two types of images to be widely applied in engineering technology and scientific research [1].

Traditional image fusion algorithms typically perform activity level measurements in the spatial or transform domain and manually design fusion rules to achieve image fusion. Traditional image fusion frameworks mainly include fusion frameworks based on multi-scale transformation [2], sparse representation [3], subspace [4], saliency [5], and variational models [6]. Although existing traditional image fusion algorithms can produce relatively satisfactory results in most cases, there are still some problems that hinder their further development. First, existing methods usually use the same transformation or representation to extract features from source images but do not consider the inherent feature differences of multi-modal images. Second, manually designed activity level measurements and fusion rules cannot adapt to complex fusion scenarios. In order to pursue better fusion performance, the design of activity level measurements and fusion rules becomes more and more complex, limiting the practical application of traditional methods [7].

Consequently, image fusion algorithms based on deep learning have emerged, highlighting their numerous advantages over traditional algorithms. Existing deep learning-based image fusion algorithms primarily aim to address three critical issues in image fusion: feature extraction, feature fusion, and image reconstruction. To tackle these key problems and improve upon the shortcomings of traditional algorithms, researchers have designed various deep learning networks to achieve higher efficiency and better performance. Depending on the network architecture adopted, these algorithms can be categorized into three types: image fusion frameworks based on autoencoders (AE), convolutional neural networks (CNN), and generative adversarial networks (GAN). In AE-based methods, networks pre-trained on large datasets are commonly used to address the issues of feature extraction and image reconstruction. However, the fusion strategies for integrating deep features extracted from different modal images are still manually designed, which restricts the performance of these methods [8]. Compared to AE-based methods, CNN-based methods achieve end-to-end feature extraction, feature fusion, and image reconstruction by designing loss functions and network structures. Nevertheless, these methods often employ a simple serial structure and fail to consider the inherent features of different modal images [9]. GAN-based methods model the image fusion problem as an adversarial game between the generator and the discriminator. The generator aims to blend the underlying distributions of the source images into the fused image, while the discriminator determines whether the input is a source image or a fused image. This process enables the generator to produce a fused image that contains more features of the source images. Although GAN-based methods can achieve better fusion results, problems have gradually emerged during research. Existing methods only perform feature fusion at a single layer and do not consider the interaction of information between different scales, which to some extent limits the fusion performance of their networks. To address the limitations of existing methods, we propose a cross-scale differential features attention generative adversarial network for infrared and visible light image fusion. This method is an end-to-end model based on Wasserstein generative adversarial networks(WGAN) [10] and dual discriminator conditional generative adversarial network (DDcGAN) [11]. Firstly, unlike existing simple fusion strategies, we designed a cross-modal attention module to measure the corresponding activity levels at different scales separately. Secondly, to enable the interaction of feature information across different scales, we designed a cross-scale attention decoder. This decoder is capable of fusing feature information across different scales in a sequential manner from deep features to shallow features. We proposed a parallel path of differential features and fusion features to calculate attention weights and perform parallel spatial and channel attention weight calculations on the two paths so that the fused image can contain more complementary information and continuously optimize the activity levels of different modal images. DAGANFuse effectively fuses the cross-scale features of multi-modal images; extracts more complementary information from features of different scales and different modalities; highlights the infrared target perception and visible light texture details; and makes the fused image of higher quality, with more information, and clearer. The main contributions of this paper are summarized as follows:

• We designed a cross-modal differential features attention module to measure the activity level at the same scale. The proposed module increases the information and details contained in the fused image by effectively integrating the intermediate features of different modal images.
• We proposed a new cross-scale attentional decoder, which can enable cross-modal features of different scales to interact. Through the parallel path of differential features and fusion features, it is beneficial to extract more complementary information on the basis of maintaining detailed information.
• We proposed an end-to-end dual discriminator adversarial fusion network based on WGAN [10]. The experimental outcomes demonstrate that the proposed method achieves the SOTA performance. Compared with other conventional fusion techniques, it exhibits superiority in both qualitative visual depiction and quantitative index assessment, thereby offering a more efficient and robust approach for multi-modal image fusion tasks.

The remainder of this paper is organized as follows. Section 2 provides a comprehensive review of deep learning-based infrared and visible image fusion work, based on the adopted deep learning network framework. Section 3 elaborates on the proposed fusion framework in detail. Section 4 presents the experimental design, results, and analysis. Finally, Section 5 concludes the paper.

2. Related Work

Existing deep learning-based image fusion algorithms primarily adopt three architectural paradigms: AE-based frameworks, CNN-based frameworks and GAN-based frameworks. While these paradigms may overlap, this categorization reflects their dominant design principles and training objectives.

2.1. AE-Based Fusion Methods

Addressing the limitations and challenges inherent in traditional image fusion methods, researchers have increasingly embraced neural network-based approaches, which are highly regarded for their superior nonlinear fitting capabilities [12]. These advanced techniques have significantly enhanced the quality and efficiency of image fusion. Some researchers have proposed a variety of infrared and visible light image fusion methods utilizing autoencoders (AE). During the training phase, the autoencoder network is pre-trained using a large-scale dataset. The encoder in the network is specifically designed for feature extraction, while the decoder is responsible for image reconstruction. In the testing phase, the features extracted by the encoder are fused through manually designed fusion strategies to produce the final fused image. For instance, Li et al. [13] introduced DenseFuse, where dense connections were proposed in the encoder network to produce a powerful ability for feature extraction. Deep learning methods based on filters or optimization in the decomposition stage have certain limitations. To solve this problem, Zhao et al. designed the first deep image decomposition model under the AE network model, named DIDFuse [14]. Both the fusion and decomposition tasks are completed by the AE network. To alleviate the domain difference problem, Zhao et al. proposed a self-supervised IVIF feature adaptive network framework: SFA-Fuse [15]. This network includes a self-supervised feature adaptive network and an enhanced network coordinated by edge details and contrast-based loss functions.

To further improve the fusion performance, some researchers have introduced attention mechanisms into autoencoders. Li et al. [8] proposed the NestFuse fusion method, introducing a nested connection architecture and a spatial/channel attention model into the AE network. Subsequently, Jiang et al. proposed a symmetric encoder–decoder network with residual blocks, named SEDRFuse [16]. They designed a feature fusion strategy based on attention maps, thereby preserving more information from different modality images. Wang et al. [17] proposed a fusion network based on dense Res2net and dual non-local attention models. Thereafter, they introduced UNFusion [18]. Dense skip connections are introduced into the AE network to extract the feature correlation of the intermediate layer. The problem of insufficient fine features and coarse-scale features is solved by designing a multi-scale network structure. The Lp normalized attention model is used as the fusion strategy. These methods propose attention-based fusion strategies and use these attention models to refine and integrate features.

2.2. CNN-Based Fusion Methods

Liu et al. [19] proposed the first CNN-based IVIF method. This method uses Siamese CNN to obtain the weight map of infrared and visible light pixel activity information; uses image pyramids to fuse images at multiple scales; and uses a strategy based on local similarity to adjust the decomposition coefficients of the fusion mode automatically. Inspired by this, researchers have successively proposed a variety of CNN-based methods. Hou et al. proposed an end-to-end unsupervised CNN-based model called VIF-Net [20]. This method introduced a dual-channel dense network to achieve feature extraction and realized feature fusion by directly connecting deep features. Long et al. introduced an IVIF method—RXDNFuse [21]—using an aggregated residual dense network in the literature. This network combines the network structure advantages of ResNeXt [22] and DenseNet [23], effectively avoiding the limitations of manually designed fusion rules. Considering that lighting factors are important for the effects they have on image fusion quality, Tang et al. designed a light perception-based progressive image fusion network PIAFusion [24]. A light perception subnetwork is used to calculate light distribution and probability; in addition, a progressive feature extractor is developed, which contains a cross-modal differential perception fusion module and a mid-way fusion strategy to extract and fuse complementary information in multi-modal images adaptively. Tang et al. proposed a semantic-aware real-time infrared and visible image fusion network, SeAFusion [25]. This method is the first framework to integrate high-level vision tasks with image fusion. By introducing a semantic loss and a gradient residual dense block, SeAFusion significantly enhances the semantic information in the fused images, thereby improving visual quality and detail preservation while substantially boosting performance in high-level vision tasks. Additionally, it achieves efficient real-time processing. Xu et al. proposed an unsupervised end-to-end image fusion network, U2Fusion [26]. This method addresses the multi-modal image fusion problem by automatically evaluating the importance of corresponding source images to determine the adaptive degree of information retention.

Although these methods have adopted some strategies to replace the ground truth label, they have greatly improved the quality of fused images. However, due to the absence of a standard ground truth label, in the image fusion task, the potential performance of CNNs still cannot be maximally presented.

2.3. GAN-Based Fusion Methods

In 2019, Ma et al. [27] first introduced GAN into the infrared and visible image fusion (IVIF) task, namely FusionGAN. Although FusionGAN has achieved certain results, there are still some shortcomings: on the one hand, relying only on adversarial training to add additional detailed information may lead to the loss of some information; on the other hand, it ignores the edge information in infrared images, resulting in relatively blurred target edges in the fusion result. To solve these problems, Ma et al. proposed Detail-GAN [28] and designed detail loss and target edge enhancement loss, respectively. However, these two methods failed to adequately consider the texture information of infrared images and the contrast information of visible light images, resulting in the generated fused images being more similar to sharpened infrared images. Subsequently, Ma et al. proposed a fusion network that employs a multiple classifier as discriminators, namely GANMcC [29], thereby endowing the fused images with abundant texture information and significant contrast. Fu et al. proposed Perception-GAN [30], which uses dense blocks in the generator to connect the shallow layer and source images with rich detail features to the deeper layer. Le et al. proposed an unsupervised continuous learning generative adversarial network UIFGAN [31] for training a single model with memory to achieve multiple image fusion tasks.

The GAN network with a single discriminator cannot well preserve both infrared pixel intensity information and visible light texture detail information at the same time, and the training is not stable enough, which has certain deficiencies. Therefore, Ma et al. proposed the use of dual discriminators in DDcGAN [11] to increase the evaluation and feedback of the generated image, thereby enhancing the stability of training. The two discriminators act on the generator respectively, enabling the fused image to retain features from both modalities. While preserving the texture information of visible light, it also retains the thermal radiation information of infrared images. Building on this, Li et al. introduced a multi-scale attention mechanism into GAN, proposing AttentionFGAN [32]. Meanwhile, Zhou et al. proposed a semantic-supervised dual discriminator generative adversarial network, SDDGAN [33], which guides the fusion process through an information quantity discriminator block.

Distinguished from the methods previously mentioned, our DAGANFuse calculates attention weights in both the spatial and channel dimensions on two parallel paths of differential features and fused features, respectively, to measure the activity level of the source images at the same scale. It then effectively integrates cross-scale features through a cross-scale attention mechanism, thereby obtaining high-quality infrared and visible light fused images.

3. Methods

3.1. The Architecture of the Fusion Network

The principle of our DAGANFuse is presented in Figure 1, which includes a generator and dual discriminators.

3.1.1. Generator Architecture

In the feature extraction phase, two encoders respectively process infrared and visible light images. Both encoders contain four multi-scale convolution blocks (MSCB). Each MSCB has two convolution layers with a 3 × 3 kernel, and strides set to 1 and 2. The filter banks for different scales are all set to 16 × l. When source images are input into these blocks, four-scale features of each source image are obtained, represented as ${\Phi }_{ir}^l$ and ${\Phi }_{vis}^l$, l = 1, 2, 3 and 4. In the feature reconstruction phase, a cross-scale attentional decoder (CSAD) is built, as shown in Figure 2. Features of different scales and modalities are fed into the decoder. Use the proposed cross-modal differential features attention module (CDAM) to reconstruct features from deep to shallow scales. The upsampled output of each CDAM, combined with the next-layer features, becomes the input for the next CDAM.

3.1.2. Cross-Modal Differential Features Attention Module (CDAM)

To integrate the inherent features of different modal images, we propose a cross-modal differential features attention module. The architecture of the CDAM module is shown in the Figure 3.

Different from CBAM [34], we use two convolutional layers to replace the multi-layer perceptron (MLP) in CBAM. Based on this, we propose an attention framework using differential features path module (DFPM) and fusion features path module (FFPM) in parallel. On the two parallel paths, the initial fusion feature ${\Phi }_f^l\in R^{H\times W\times C}$—where H and W represent the height and width of the source image, and C represents the number of channels—is used to compute its attention maps from the spatial and channel dimensions, respectively. Combine the results from the two paths and allocate the resulting attention weights to measure the activity levels of different modalities of images at the same scale. In DFPM, the differential feature ${\Phi }_d^l$ is the concatenation of ${\Phi }_{d1}^l$ and ${\Phi }_{d2}^l$, which are defined as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\Phi }_{d1}^l={\Phi }_{ir}^l-{\Phi }_{vis}^l\end{array}\end{array}$$

(1)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\Phi }_{d2}^l={\Phi }_{vis}^l-{\Phi }_{ir}^l\end{array}\end{array}$$

(2)

The generated intermediate fused features, termed ${\Phi }_f^{l-1}$, will be seen as the initial fused features of the next CDAM. Particularly, at the fourth scale layer, the initial fusion feature is a concatenation of ${\Phi }_{ir}^l$ and ${\Phi }_{vis}^l$.

In the spatial-independent path, we utilize the maximum and average pooling operations to obtain their initial spatial attentional matrices. Then, we concatenate them into a convolution layer to generate the spatial attentional matrix ${\Phi }_f^{SA,l}\in R^{1\times H\times W}$ and ${\Phi }_d^{SA,l}\in R^{1\times H\times W}$, which are formulated as follows:

$${\Phi }_f^{\mathrm{SA},l}(i,j)=\mathrm{Conv}([\mathrm{MP}({\Phi }_f^l(c,i,j)),\mathrm{AP}({\Phi }_f^l(c,i,j))])$$

(3)

$${\Phi }_d^{\mathrm{SA},l}(i,j)=\mathrm{Conv}\left([\mathrm{MP}\left({\Phi }_d^l(c,i,j)\right),\mathrm{AP}\left({\Phi }_d^l(c,i,j)\right)]\right)$$

(4)

where $MP(·)$ and $AP(·)$ denote the global maximum and average pooling operations. Conv represents the convolution.

Meanwhile, in the channel-independent path, we first use the maximum pooling and average pooling operations. Then, the obtained initial channel attention vectors pass through two convolution layers and a PReLU activation layer, and are concatenated and fed into a convolutional layer to obtain the channel attention vector ${\Phi }_f^{CA,l}\left(c\right)\in R^{C\times 1\times 1}$ and ${\Phi }_d^{CA,l}\left(c\right)\in R^{C\times 1\times 1}$, which are formulated as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\Phi }_f^{\mathrm{CA},l}\left(c\right)=\mathrm{Conv}\left([\mathrm{Conv}\left(\mathrm{PReLU}\left(\mathrm{Conv}\left(\mathrm{MP}\left({\Phi }_f^l(c,i,j)\right)\right)\right)\right),\mathrm{Conv}\left(\mathrm{PReLU}\left(\mathrm{Conv}\left(\mathrm{AP}\left({\Phi }_f^l(c,i,j)\right)\right)\right)\right)]\right)\end{array}\end{array}$$

(5)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\Phi }_d^{\mathrm{CA},l}\left(c\right)=\mathrm{Conv}(\mathrm{Conv}\left(\mathrm{PReLU}\left(\mathrm{Conv}\left(\mathrm{MP}\left({\Phi }_d^l(c,i,j)\right)\right)\right)\right),\mathrm{Conv}\left(\mathrm{PReLU}\left(\mathrm{Conv}\left(\mathrm{AP}\left({\Phi }_d^l(c,i,j)\right)\right)\right)\right))\end{array}\end{array}$$

(6)

where PReLU represents PReLU activation operations.

Then, we element-wise multiply the obtained two pairs of spatial attention matrices and channel attention vectors. After that, we attempt to normalize them using the sigmoid activation function to obtain the corresponding attention weights. The attentional weights ${\alpha }_f^{ir,l}$/${\alpha }_d^{ir,l}$ for infrared path and ${\alpha }_f^{vis,l}$/${\alpha }_d^{vis,l}$ for the visible path are represented by the following formulas:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\alpha }_f^{ir,l}(c,i,j)=\sigma ({\Phi }_f^{\mathrm{CA},l}\left(c\right)\times {\Phi }_f^{\mathrm{SA},l}(i,j))\end{array}\end{array}$$

(7)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\alpha }_d^{ir,l}(c,i,j)=\sigma ({\Phi }_d^{\mathrm{CA},l}\left(c\right)\times {\Phi }_d^{\mathrm{SA},l}(i,j))\end{array}\end{array}$$

(8)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\alpha }_f^{vis,l}(c,i,j)=1-{\alpha }_f^{ir,l}(c,i,j)=1-\sigma ({\Phi }_f^{\mathrm{CA},l}\left(c\right)\times {\Phi }_f^{\mathrm{SA},l}(i,j))\end{array}\end{array}$$

(9)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\alpha }_d^{vis,l}(c,i,j)=1-{\alpha }_d^{ir,l}(c,i,j)=1-\sigma ({\Phi }_d^{\mathrm{CA},l}\left(c\right)\times {\Phi }_d^{\mathrm{SA},l}(i,j))\end{array}\end{array}$$

(10)

where $\sigma (·)$ denotes the sigmoid activation function.

Finally, the intermediate fused features ${\Phi }_f^{l-1}\in R^{C\times H\times W}$ are show as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\Phi }_f^{l-1}(c,i,j)=[{\alpha }_f^{ir,l}\times {\Phi }_{ir}^l+{\alpha }_d^{ir,l}\times {\Phi }_{d1}^l,{\alpha }_f^{vis,l}\times {\Phi }_{vis}^l+{\alpha }_d^{vis,l}\times {\Phi }_{d2}^l]\end{array}\end{array}$$

(11)

3.1.3. Discriminator Architecture

In the discriminator, we adopt two discriminators, $D_{ir}$ and $D_{vis}$, to enable a more balanced adversarial game with the generator. The architecture of the discriminators is shown in the Figure 4. The first discriminator $D_{ir}$ is used to distinguish the fused results from the infrared images, and the second discriminator $D_{vis}$ is used to distinguish the fused results from the visible images. The two discriminators have the same structure, but their parameters are not shared. They consist of four convolutional layers, and the convolution kernel size and stride are set to $3\times 3$ and 2, respectively. The output channels are 16, 32, 64, and 128 in sequence. The activation function of the last layer is the Tanh function, and the remaining three layers are all LeakyReLU functions [35].

3.1.4. Loss Function

To generate high-quality fused images due to GAN features, we added intensity and texture loss functions to the generator loss. So, the generator loss function can be expressed as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{\mathrm{generator}}=L_{\mathrm{adversarial}}+{\lambda }_1{L}_{\mathrm{intensity}}+{\lambda }_2{L}_{\mathrm{texture}}\end{array}\end{array}$$

(12)

where $L_{generator}$ denotes the total generator loss function, while $L_{adversarial}$ and $L_{content}$ represent the adversarial and content loss functions, respectively. The parameter ${\lambda }_1$ and ${\lambda }_2$ is the factor to control the balance between the three terms.

To make generated images contain more intensity information from infrared and visible light images, we define the intensity loss function as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{\mathrm{intensity}}={\displaystyle \frac{1}{HW}}({∥I_f-I_{\mathrm{ir}}∥}_2+{\lambda }_3{∥I_f-I_{\mathrm{vis}}∥}_1)\end{array}\end{array}$$

(13)

where $I_f$, $I_{ir}$ and $I_{vis}$ represent the fusion image, infrared image, and visible image. ${∥·∥}_1$ and ${∥·∥}_2$ denote the L_1-norm and L_2-norm, respectively. The parameter ${\lambda }_3$ is a weighted coefficient.

To make the fusion image retain more texture information, we introduce texture loss to assist intensity loss. The texture loss is shown as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{\mathrm{texture}}={\displaystyle \frac{1}{HW}}\left({∥\left|\nabla I_f\right|-max\left(\left|\nabla I_{ir}\right|,\left|\nabla I_{vis}\right|\right)∥}_1\right)\end{array}\end{array}$$

(14)

where ∇ represents the gradient operator.

In DAGANFuse, the adversarial loss consists of two parts: the adversarial loss between the generator and each of the two discriminators, $D_{ir}$ and $D_{vis}$. The adversarial loss is formulated as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{\mathrm{adversarial}}=-\mathbb{E}\left[D_{ir}\left(I_f\right)\right]-\mathbb{E}\left[D_{vis}\left(I_f\right)\right]\end{array}\end{array}$$

(15)

In addition, we define the loss functions of the two discriminators based on the Wasserstein distance:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{D_{\mathrm{ir}}/D_{\mathrm{vis}}}=\mathbb{E}\left[D_{\mathrm{ir}/\mathrm{vis}}\left(I_f\right)+{\lambda }_4{\left({∥\nabla D_{\mathrm{ir}/\mathrm{vis}}\left({\tilde{I}}_{\mathrm{ir}/\mathrm{vis}}\right)∥}_2-1\right)}^2\right]\end{array}\end{array}$$

(16)

In the formula, the first part represents the Wasserstein distance estimation, and the second part is the gradient penalty. ${\lambda }_4$ is defined as the regularization parameter.

4. Experiments and Discussion

In this section, we first briefly introduce the experimental details. Next, we design ablation experiments to analyze the impact of each part in the proposed attention module. Finally, we evaluate and analyze the advantages of our method through comparative experiments and multiple performance indicators. Our network is implemented on the NVIDIA GPU (GTX 4060) using PyTorch 1.13.1 as a programming environment in PyCharm.

4.1. Experimental Settings

During data preprocessing, to create a dataset capable of training a good model, 25 pairs of infrared and visible images are selected from the TNO Image Fusion Dataset [36] and cropped into $256\times 256$ sub-images with a sliding step size of 12, yielding 16,986 image pairs. This practice can split some local features into smaller ones, thereby enhancing detail capture, improving robustness to variations, and increasing data diversity for better model training.

In the training phase, we trained the discriminators twice, then the generator once using the Adam optimizer. The epoch and batch size were set to 14 and 4. The learning rates for the generator and discriminator are set to $1\times {10}^{-5}$ and $1\times {10}^{-4}$. The weighted parameters ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, and ${\lambda }_4$ are set to 1, 10, 1, and 10, respectively.

In the testing phase, we utilize the Image Fusion Dataset [36] and Roadscene Dataset [37]. The TNO Image Fusion Dataset contains multispectral nighttime imagery of different military relevant scenarios. The Roadscene Dataset contains 221 aligned visible and infrared image pairs with rich scenes like roads, vehicles, pedestrians, and so on. These images are highly representative of FLIR video. Moreover, seven representative methods—such as FusionGAN [27], DDcGAN [11], MDLatLRR [38], DenseFuse [13], Res2Fusion [17], MFEIF [39], and SDDGAN [33]—are proposed to compare with the DAGANFuse.

To comprehensively assess the quality of fused results, we conduct evaluations in both qualitative and quantitative dimensions. For quantitative verification, we use entropy (EN) [40], standard deviation (SD) [41], average gradient (AG), mutual information (MI), visual information fidelity (VIF) [42], and spatial frequency (SF) [43].

4.2. Ablation Study

In this section, to verify the effectiveness of our proposed attention mechanism, we conductablation experiments on four comparable models and perform analysis. The four ablation experiments include replacing CDAM with two convolutional layers, termed Without_CDAM; replacing CDAM with CBAM, termed CBAM; without the differential feature path, termed Without_DFPM; and only containing the differential feature path, termed Without_FFPM.

• Qualitative analysis: As can be seen from the results in Figure 5, in the absence of CDAM, the Without_CDAM model not only reduces the brightness of infrared targets but also has the problem of blurred contours. In contrast, adding the CBAM model enables the fused image to retain some infrared target information but loses some texture details. In addition, the results of the Without_DFPM model, the Without_FFPM model, and our model are similar, with no obvious differences. All of them better retain the brightness information of infrared targets and the texture information of visible images.
• Quantitative analysis: The quantitative analysis results are shown in

  Table 1. The objective results of ablation studies.

  	EN	SD	AG	MI	VIF	SF
  Without_CDAM	6.9969	37.1377	1.7484	1.0673	0.1998	3.1063
  CBAM	7.0451	40.5360	2.1700	1.6596	0.4171	4.1848
  Without_DFPM	7.1019	41.5828	3.9443	2.3943	0.5145	8.2949
  Without_FFPM	7.0958	41.6206	3.9841	2.8534	0.5373	8.4275
  ours	7.1253	41.8938	4.0462	2.6519	0.5367	8.5184

  . The red bold and underlined values indicate the optimal and sub-optimal values respectively. Compared with other models, our model obtains the optimal values under the evaluation indicators of entropy (EN), standard deviation (SD), average gradient (AG), and spatial frequency (SF), and obtains the sub-optimal values under the evaluation indicators of mutual information (MI) and visual information fidelity (VIF). In summary, every component of the differential features attention module proposed is effective.

4.3. Fusion Results on TNO Dataset

To test the performance of our DAGANFuse, we first employ the TNO Image Fusion Dataset. We select four pairs of typical images for intuitive qualitative comparison, as shown in Figure 6. The infrared targets and visible details are marked with red and blue frames, respectively. Judging from the fusion results, FusionGAN and DDcGAN inherit more infrared target information, but there are problems of blurred contours and lack of texture information, while MDLatRR and DenseFuse retain more texture information but lose some infrared target information. In contrast, Res2Fusion, MFEIF, and SDDGAN are superior to the previous four methods in comprehensive performance. In general, from the comparison of the results of four typical image pairs, our method can retain more visible light texture information and infrared target information.

To objectively evaluate the quality of fused images, we also conduct quantitative analysis. The index values are shown in

Table 2. The average metrics values obtained by the existing fusion methods and the proposed network on TNO.

	EN	SD	AG	MI	VIF	SF
FusionGAN	6.4493	27.2618	3.0096	2.1065	0.2846	5.9125
DDcGAN	6.4783	41.6072	3.9393	1.2044	0.2432	8.1566
MDLatLRR	6.3292	24.0935	3.5845	1.9596	0.3669	7.0322
DenseFuse	6.2892	23.2344	3.1913	2.0259	0.3424	6.0231
MFEIF	6.6492	31.5080	3.4791	2.3943	0.3842	6.7707
Res2Fusion	6.9569	39.2702	4.0239	2.7047	0.4579	8.3721
SDDGAN	7.0352	45.8426	3.1325	1.3724	0.3987	6.4354
DAGANFuse	7.1253	41.8938	4.0462	2.6519	0.5367	8.5184

. The best values are indicated in red and bold font, and the sub-optimal values are indicated by underlining. In Table 2, compared with other state-of-the-art fusion methods, our proposed method (DAGANFuse) achieves four best values (EN, AG, VIF, and SF) and two sub-optimal values (SD and MI). This means that DAGANFuse can retain more complementary information from the pixel level and feature level. In conclusion, our proposed DAGANFuse has a good fusion effect in both quantitative index evaluation and qualitative visual description.

4.4. Fusion Results on Roadscene Dataset

To further test the effectiveness of our DAGANFuse, we selected 40 pairs of images from the Roadscene dataset for experimental analysis.

Firstly, we selected two pairs of typical images, FLIR_07210 and FLIR_06570, for qualitative analysis. The fusion results are shown in Figure 7 and Figure 8. Compared with other methods, DAGANFuse has obvious advantages. Firstly, for the prominent infrared targets, such as the streetlight marked with the red box in Figure 7, DAGANFuse can maintain their original shapes while retaining the original brightness as much as possible. Moreover, for the visible texture information, like the “STOP SIGN” and house that are locally magnified with the blue boxes, DAGANFuse preserves clearer information compared to other methods.

Meanwhile, we conducted a quantitative comparison between DAGANFuse and seven other methods on the Roadscene dataset. The index values are shown in

Table 3. The average metrics values obtained by the existing fusion methods and the proposed network on Roadscene.

	EN	SD	AG	MI	VIF	SF
FusionGAN	7.0087	37.1567	2.6824	1.8554	0.2914	5.8114
DDcGAN	7.5950	56.6306	3.5391	1.4670	0.1763	7.0693
MDLatLRR	7.3976	54.5890	5.3283	1.2881	0.3524	10.2970
DenseFuse	7.2343	42.0498	3.3448	1.7809	0.3531	6.8627
MFEIF	7.0695	38.8528	2.9874	2.1377	0.3787	6.3558
Res2Fusion	7.3308	46.6483	3.3744	2.3449	0.4073	7.1569
SDDGAN	7.5453	56.1106	3.2748	2.0076	0.3529	6.7498
DAGANFuse	7.8632	47.3390	6.4322	2.4970	0.4505	12.3760

. Among the six evaluation indicators, DAGANFuse has significant advantages in five of them, namely entropy (EN), average gradient (AG), mutual information (MI), visual information fidelity (VIF), and spatial frequency (SF). As for the standard deviation (SD) indicator, it ranks fourth, behind DDcGAN, SDDGAN, and MDLatLRR. DAGANFuse achieves the optimal values in three indicators, AG, VIF, and SF, which indicates that the fusion results generated by the proposed model DAGANFuse not only contain richer gradient and texture information but also have better visual effects. The optimal value obtained in the MI indicator shows that DAGANFuse can retain more modality features in the fused image. It also means that the cross-modal differential features attention module proposed in DAGANFuse can effectively distinguish the intrinsic content of different modal images and integrate them effectively into the fusion results. Although DAGANFuse does not achieve the highest value in the SD indicator, from the overall evaluation of the six indicators and the results of qualitative comparison, DAGANFuse has the best fusion effect.

5. Conclusions

In this paper, we proposed an infrared and visible image adversarial network based on cross-scale attention mechanism, termed DAGANFuse. Different from the existing end-to-end fusion methods, we designed a cross-scale differential features attention module to let fused images contain more complementary information of infrared and visible images on the basis of combining the intensity and texture information of the source images. A parallel path of differential features and fusion features is adopted to calculate the attention weights, and parallel spatial and channel attention weight calculations are carried out on the two paths. Moreover, a cross-scale decoder framework is constructed to interact with feature information of different scales and continuously optimize the activity level in an iterative manner. In terms of the discriminator, a dual discriminator is used to establish a more balanced adversarial game with the generator. In addition, the effectiveness of the key parts of DAGANFuse is verified through ablation experiments, and comparisons with seven state-of-the-art methods are made on the public datasets TNO and Roadscene. Qualitative and quantitative analyses verify that DAGANFuse has significant advantages in fusion performance.