BCMFIFuse: A Bilateral Cross-Modal Feature Interaction-Based Network for Infrared and Visible Image Fusion

Abstract

The main purpose of infrared and visible image fusion is to produce a fusion image that incorporates less redundant information while incorporating more complementary information, thereby facilitating subsequent high-level visual tasks. However, obtaining complementary information from different modalities of images is a challenge. Existing fusion methods often consider only relevance and neglect the complementarity of different modalities’ features, leading to the loss of some cross-modal complementary information. To enhance complementary information, it is believed that more comprehensive cross-modal interactions should be provided. Therefore, a fusion network for infrared and visible fusion is proposed, which is based on bilateral cross-feature interaction, termed BCMFIFuse. To obtain features in images of different modalities, we devise a two-stream network. During the feature extraction, a cross-modal feature correction block (CMFC) is introduced, which calibrates the current modality features by leveraging feature correlations from different modalities in both spatial and channel dimensions. Then, a feature fusion block (FFB) is employed to effectively integrate cross-modal information. The FFB aims to explore and integrate the most discriminative features from the infrared and visible image, enabling long-range contextual interactions to enhance global cross-modal features. In addition, to extract more comprehensive multi-scale features, we develop a hybrid pyramid dilated convolution block (HPDCB). Comprehensive experiments on different datasets reveal that our method performs excellently in qualitative, quantitative, and object detection evaluations.

1. Introduction

Image fusion technology can generate high-quality fusion images containing rich detail information by integrating two or more source images [1,2]. This technology covers various image types, including medical image fusion (MIF), multi-exposure image fusion (MEIF), and infrared–visible image fusion (IVIF). The main goal is to provide a clear and comprehensive scene representation to enhance scene understanding. Among them, IVIF is the most widespread and challenging since it requires the effective extraction and combination of cross-modal features from different sensors. The core goal of IVIF is to retain the abundant texture details from the visible image and the salient target from the infrared image [3]. By fusing these two, it can avoid problems caused by the low resolution and noise of infrared images, and simultaneously overcome the limitations of visible images in harsh working environments (such as rain, snow, fog, low illumination, etc.). High-quality fusion images are very helpful for downstream high-level visual tasks, including remote sensing [4], object detection [5], image segmentation [6], and autonomous driving [7].

Figure 1 demonstrates the fusion and detection results in a typical smoke scenario. In such conditions, humans and other important targets are often obscured by smoke. Nonetheless, owing to the unique imaging principles inherent to infrared images, they can penetrate smoke and distinctly reveal these targets. As illustrated in Figure 1, the fusion image significantly enhances the visibility of targets in smoky conditions, thereby improving detection accuracy. For target detection, visible images fail to detect critical targets obscured by smoke. The FGAN [8] misses critical target detections, while the DSF [9] results in false detections. SuperF [10] and the proposed method both successfully detected these targets, with our method having higher detection accuracy. This indicates that the proposed method efficiently integrates useful information from infrared and visible images, enhancing subsequent target detection accuracy. IVIF has tremendous potential for applications in remote sensing, especially in monitoring complex environments. For example, in forest fire monitoring, IVIF can help identify fire sources and hotspot areas, clearly showing the location and range of the fire, which is very helpful for decision-makers in formulating more effective rescue plans.

Existing IVIF methods are primarily divided into traditional and deep learning-based methods. Traditional fusion methods encompass methods based on saliency [11], subspace [12], sparse representation (SR) [13], multi-scale transform (MST) [14], and hybrid [15]. Although the above methods obtain satisfactory results, they still have some obvious limitations: (1) These methods depend on manually designed fusion rules that may not comprehensively retain the effective features in the source images, especially as source image complexity increases, necessitating more intricate fusion rules. (2) These methods usually apply uniform feature extraction for images of different modalities, neglecting the differences between modalities, which could cause the loss of some unique modality features in the source images. However, methods based on deep learning exhibit powerful feature extraction capabilities, overcoming inherent deficiencies in traditional methods and garnering attention from numerous scholars. Researchers have developed various models based on deep learning for IVIF, which are generally classified into methods based on the generative adversarial network (GAN) [5,8,16,17], autoencoder (AE) [18,19,20,21], convolutional neural network (CNN) [22,23,24,25], and transformer [26,27].

While the methods mentioned above have achieved satisfactory fusion performance, several issues remain unresolved. Firstly, most of these methods extract features using a single network or two parallel networks, without considering the differences in source image features, thus ignoring the interaction of features from different modalities and losing some important cross-modal information. It possibly affects the fusion performance to a certain extent. For example, DenseFuse [18] and FusionDN [28] use a single network for feature extraction without accounting for the differences between different modalities’ images, which may lead to the loss of some modality-specific information. Secondly, balancing the discrepancies between the fusion image and the source images is difficult because IVIF lacks ground truth. Some researchers have introduced GANs to address this issue, such as FusionGAN [8], DDcGAN [16], and GANMcC [29]. Although GAN-based methods have achieved acceptable fusion results, it is challenging for GAN to effectively utilize the unique information in multi-modal images, and GAN-based methods are difficult to achieve training balance. Finally, they overlook the extraction of diverse features in the source images and may extract some redundant information or miss some important information, thus affecting the fusion performance. For instance, SDNet [30] and PMGI [31] extract features using a single convolution kernel, which has limited receptive fields and may ignore some important features. Considering the extraction of multi-scale features, NestFuse [21] and RFNNest [32] introduce a nested network, which fuses features from different layers using a fusion strategy, but different layers still use a single convolution kernel. However, these methods do not sufficiently account for the inherent characteristics of different modalities, making it challenging to learn diverse feature representations. How to learn diverse feature representation and effectively use the beneficial characteristics of bilateral modality to enhance fusion performance is still a challenge.

To tackle the aforementioned problem, we introduce BCMFIFuse, a network that utilizes bilateral cross-modal feature interaction for IVIF. Firstly, to adequately extract features from infrared and visible images, we construct a two-stream feature extraction network. Given that infrared and visible images pertain to different modalities, we design CMFC to better extract complementary features from these two modalities. CMFC calibrates the current modality features by combining features from different modalities in the channel dimension and spatial dimension. Then, we use FFB to effectively integrate the calibrated features. The FFB is constructed using a cross-attention mechanism to facilitate long-range context information interaction, thereby enhancing global bilateral modality features. Finally, to ensure feature continuity and reduce information loss during transmission, skip connections are utilized between the encoder and decoder. Moreover, to prevent feature loss from using a single convolution kernel, multi-scale feature extraction is essential. Dilated convolution is highly effective in capturing multi-scale features and exploring contextual information. However, regular dilated convolutions detect input features within a square window, which limits their flexibility in capturing diverse features. Simply using a large square dilated convolution window is not an effective solution as it tends to extract redundant features. To tackle this issue, we introduce an HPDCB that integrates rectangular dilated convolutions to collect more diverse and specific contextual information. To capture long-range relationships of isolated areas, we employ a combination of long and narrow kernel shapes in the design of HPDCB. We first employ a long kernel shape with a variable dilation rate along one spatial dimension; then, a narrow kernel shape is employed in the other spatial dimension. The key contributions of the proposed method are outlined below:

(a)

A bilateral cross-modal feature interaction-based method for IVIF is suggested. The goal of this method is to provide comprehensive cross-modal interactions and fully leverage the complementary potential of cross-modal features. In addition, we use a hybrid pyramid dilated convolution block (HPDCB) to extract multi-scale features, effectively collect various contextual information, and learn diverse feature representations.

(b)

A cross-modal feature correction block (CMFC) is introduced. The module combines the features from different modalities in spatial and channel dimensions to calibrate the current modality features. This enables the two feature extraction branches to better focus on complementary information from both modalities, thereby mitigating uncertainties and noise effects from different modalities and achieving better feature extraction and interaction.

(c)

A feature fusion block (FFB) is developed. This module effectively integrates cross-modal features and merges the calibrated features from the CMFC into a single feature for subsequent image reconstruction. This module considers interaction fusion at both the channel and spatial dimensions, which is crucial for the generalization of cross-modal feature combinations.

The remainder of this paper is organized as follows. Section 2 reviews the related work in the field of image fusion. Section 3 describes the proposed method’s framework in detail. Section 4 presents the relevant experiments and results. Finally, Section 5 concludes the main content of this paper and outlines directions for future improvements.

2. Related Works

This section outlines a thorough overview of current IVIF methods, encompassing methods based on traditional and deep learning.

2.1. Traditional Image Fusion Methods

Traditional methods vary based on feature extraction, fusion strategy, and feature reconstruction methods, and are classified as methods based on saliency, subspace, SR, MST, and hybrid. The methods based on SR and MST are the most commonly employed.

Methods based on MST (e.g., discrete wavelet transform (DWT) [33] and Laplace pyramid transform (LAP) [34]) primarily decompose the source images into sub-images of varying scales and orientations. These sub-images are then merged following specific fusion rules, and an inverse transformation is employed to generate fusion images. These methods can preserve the source images with their multi-scale features but may lose some detail information, leading to distortion or edge blur of the fusion image.

Methods based on sparse representation mainly rely on the learning of overcomplete dictionaries and the decomposition algorithm of sparse coefficients. Initially, an overcomplete dictionary is obtained through learning, followed by sparse coding of the input images. The obtained sparse coefficients are then fused using various fusion rules, and the image is finally reconstructed with the dictionary and fusion coefficients. These methods are able to preserve the details and structure of source images, whereas they have high computational complexity. Furthermore, the choice of fusion rules and dictionaries is of utmost importance, as they will affect the fusion results.

2.2. Deep Learning-Based Image Fusion Methods

Generally speaking, methods based on deep learning are mainly divided into four types: fusion methods based on AE, GAN, CNN, and transformer.

2.2.1. AE-Based Fusion Methods

These methods leverage encoder and decoder networks for feature extraction and image reconstruction; then, they apply artificially designed fusion rules for fusing features. Densefuse [18] incorporates dense blocks during the encoding process to effectively extract and utilize features, and reconstruct the fusion image with a decoder. Subsequently, Li et al. [32] introduced a residual architecture-based residual fusion network (RFN) to enhance the performance of image fusion. AUIF [35] is based on the principle of algorithm expansion and decomposes the source images into high- and low-frequency information. In addition, to further enhance the fusion performance, NestFuse [21] integrates an attention mechanism into the model. SEDRFuse [36] developed a symmetric network framework with residual blocks and introduced a feature fusion rule based on attention. Res2Fusion [37] incorporates dense Res2Net into the encoder and develops a dual non-local attention-based fusion strategy. Despite the significant fusion performance achieved by these methods, the necessity for manual formulation of fusion rules greatly limits the fusion performance improvement.

2.2.2. CNN-Based Fusion Methods

To tackle the issues of AE-based fusion methods, a number of CNN-based end-to-end fusion techniques were introduced, yielding impressive fusion results. For instance, the PIAFusion [25] network leverages illumination perception for fusing infrared and visible images. STDFusionNet [24] extracts the background regions from visible images and the target regions from infrared images using semantic segmentation, while network optimization is guided by a new loss function. Notably, the key innovation in these approaches is in the design of the loss function. Long et al. [38] introduced RXDNFuse, which has a relatively innovative network structure and uses an aggregated residual dense network to effectively extract and fuse features. To address various image fusion tasks within a single framework, PMGI [31] is proposed, which emphasizes gradient and intensity ratio preservation, enabling multiple image fusion tasks. Additionally, Zhang et al. [30] introduced SDNet, treating the fusion problem as a task of extracting and reconstructing gradient and intensity information. Xu et al. [28,39] contributed to the field with their general image fusion frameworks, U2Fusion and FusionDN. However, despite the noteworthy results these fusion methods have made, they commonly neglect the modality differences inherent between visible and infrared images. They utilize the same network structure for extracting features from different modalities, which limits their ability to distinguish inherent feature differences between modalities, subsequently restricting their fusion performance.

2.2.3. GAN-Based Fusion Methods

IVIF lacks ground truth; to address this challenge, researchers have introduced GAN. FusionGAN [8] is a new beginning in the field of fusion, marking the official application of GAN in this field. Subsequently, numerous GAN-based fusion methods have emerged. For example, ResNetFusion [40] tackles the problems of target edge blurring and texture detail loss in FusionGAN. This method incorporates target edge enhancement loss and detail loss functions to sharpen target edges and enhance detail information. Furthermore, proposed by Ma et al. [16], DDcGAN contains a generator and dual discriminators, which can train the generator more comprehensively and prevent information loss that could occur with a single discriminator. GANMcC [29] introduces a multi-class constraint, transforming the fusion problem into a simultaneous estimation of multiple distributions and achieving excellent fusion results. TarDAL [5] is a dual adversarial learning network using target perception and applies to image fusion and downstream object detection tasks. GAN-FM, proposed by Zhang et al. [41], involves a full-scale skip-connected generator to leverage multi-scale information during the fusion process. Additionally, many researchers introduced attention mechanisms in their frameworks. For example, AttentionFGAN [42] incorporates multi-scale attention into the generator and the discriminator, significantly enhancing fusion performance. TC-GAN [43] applies squeeze and excitation modules in the generator to better retain important texture details in the fusion images. However, achieving training balance in GAN-based methods still poses a challenge.

2.2.4. Transformer-Based Fusion Methods

Models of this kind employ the multi-head self-attention mechanism, exhibiting remarkable performance in capturing global information and overcoming the limited receptive field issue of CNNs. Thanks to its exceptional ability to explore the global context, the transformer model has excelled in computer vision tasks, giving rise to various fusion methods based on transformer [26,27,44]. For instance, SwinFusion [26] introduces a fusion framework that combines the convolutional neural network and the transformer, enabling the utilization of both local and global information. YDTR [44] captures local important information and crucial contextual detail information by utilizing a dynamic transformer, preserving texture details and salient targets in the source images. However, transformer-based methods, while delivering outstanding performance, also entail significant computational costs.

3. Method

In this section, we describe the framework of the BCMFIFuse, which is a method for IVIF based on bilateral cross-modal feature interaction. First, an outline of the overall architecture of BCMFIFuse is presented. Next, we introduce network structures of hybrid pyramid dilated convolution block (HPDCB), cross-modal feature correction block (CMFC), and feature fusion block (FFB). Finally, we delve into the multiple constraint loss functions employed in the BCMFIFuse network model.

3.1. Overall Architecture

IVIF aims to maintain essential target information in infrared images while simultaneously retaining the abundant texture details in visible images. Extracting and leveraging complementary features from the visible and infrared images remains challenging on account of differences in sensor acquisition and imaging mechanisms. Existing fusion methods suffer from limitations in feature representation, potentially leading to information loss and negatively impacting the fusion results. Therefore, achieving comprehensive cross-modal interaction is crucial for fully leveraging the complementary features from the infrared and visible images. In this study, we propose BCMFIFuse, an IVIF network based on bilateral cross-modal feature interaction. Figure 2 demonstrates the overall framework of BCMFIFuse, which is constructed with a two-stream design for feature extraction from infrared and visible images. The architecture of BCMFIFuse primarily comprises four key components: HPDCB, CMFC, FFB, and FR. We use the HPDCB module to extract comprehensive multi-scale feature information, then employ the CMFC to calibrate the extracted features, and subsequently use the FFB to fuse the calibrated features. Additionally, to ensure feature continuity and reduce information loss during transmission, we implement residual connections between the encoder and decoder. The FR is primarily employed to ensure that the reconstructed visible and infrared images contain complete information.

3.2. Hybrid Pyramid Dilated Convolution Block (HPDCB)

The size and shape of the convolution kernel can affect the features extracted. As the scale changes of the target objects in the dataset may not always be regular, to extract a more comprehensive and richer image feature representation, we need to capture features at different scales. Conventional dilated convolutions mainly detect input features of square windows, which may limit their flexibility in capturing features. Relying solely on large square dilated convolution windows is insufficient to fully tackle this issue and might even lead to the extraction of redundant features. To tackle this issue, we design a module named HPDCB, which aims to obtain more specific and diverse contextual information, as shown in Figure 3. HPDCB is capable of capturing the long-range relationships between isolated areas. During the design phase of HPDCB, rectangular dilated convolution is incorporated into the dilated convolution framework and adopts a method combining long kernel and narrow kernel shapes. First, a long kernel shape with a variable dilation rate along one spatial dimension is deployed; then, a narrow kernel shape is deployed in the other spatial dimension. The process mentioned above can be formulated as follows:

$$\begin{array}{l}{h}_1=relu\left(con{v}_{3\times 1}^{d=1}\left(relu\left(con{v}_{1\times 3}^{d=1}\left(F_a\right)\right)\right)\right)\\ h_2=relu\left(con{v}_{3\times 1}^{d=2}\left(relu\left(con{v}_{1\times 3}^{d=2}\left(F_a\right)\right)\right)\right)\\ h_3=relu\left(con{v}_{3\times 1}^{d=3}\left(relu\left(con{v}_{1\times 3}^{d=3}\left(F_a\right)\right)\right)\right)\\ h_4=relu\left(con{v}_{3\times 3}\right(F_a\left)\right)\\ h=h_1+h_2+h_3+h_4\end{array}$$

(1)

where $F_a$ denotes the input feature, d denotes dilation rate, $relu$ stands for ReLU activation function, $conv$ stands for convolution, h is the output feature, and $3\times 1$ and $3\times 1$ are the kernel sizes.

3.3. Cross-Modal Feature Correction Block (CMFC)

In our work, we construct two parallel branches for extracting features from the visible and infrared images, respectively. As is widely recognized, multimodal images often contain a large amount of noise from different modalities. Nevertheless, infrared and visible images offer complementary information, with the potential for their features to calibrate the noise information pertaining to one another. Therefore, we design CMFC to correct features from different modalities. Figure 4 depicts the structure of CMFC. The calibrated features are then fed into the subsequent phases for further enhancement and refinement of feature extraction. Assuming that the inputs of CMFC are $f_i^{ir}$ and $f_i^{vi}$, the workflow can be expressed as follows:

$$\begin{array}{c}{M}_{ir1}^i=sig\left(conv\left(GMP\left(f_i^{ir}\right)\right)\right)\hfill \\ M_{ir2}^i=sig\left(conv\left(GAP\left(f_i^{ir}\right)\right)\right)\hfill \\ M_{ir3}^i=sig\left(conv\left(AP\left(f_i^{ir}\right)\right)\right)\hfill \\ M_{ir4}^i=sig\left(conv\left(MP\left(f_i^{ir}\right)\right)\right)\hfill \end{array},$$

(2)

$$\begin{array}{c}{M}_{vi1}^i=sig\left(conv\left(GMP\left(f_i^{vi}\right)\right)\right)\hfill \\ M_{vi2}^i=sig\left(conv\left(GAP\left(f_i^{vi}\right)\right)\right)\hfill \\ M_{vi3}^i=sig\left(conv\left(AP\left(f_i^{vi}\right)\right)\right)\hfill \\ M_{vi4}^i=sig\left(conv\left(MP\left(f_i^{vi}\right)\right)\right)\hfill \end{array},$$

(3)

$$\begin{array}{c}{M}_{a1}^i=softmax(M_{ir1}^i+M_{vi1}^i)\otimes f_i^a\hfill \\ M_{a2}^i=softmax(M_{ir2}^i+M_{vi2}^i)\otimes f_i^a\hfill \\ M_{a3}^i=softmax(M_{ir3}^i+M_{vi3}^i)\otimes f_i^a\hfill \\ M_{a4}^i=softmax(M_{ir4}^i+M_{vi4}^i)\otimes f_i^a\hfill \end{array},$$

(4)

$${\widehat{f}}_i^a=M_{a1}^i+M_{a2}^i+M_{a3}^i+M_{a4}^i,$$

(5)

where $conv$ stands for convolution, $sig$ denotes the sigmoid function, and ⊗ represents element-wise multiplication. $M_{aj}^i$ ($a=ir,vi;i,j=1,2,3,4$) signifies the result of the j-th branch at the i-th layer. ${\widehat{f}}_i^s$ represents the calibrated features.

3.4. Feature Fusion Block (FFB)

To extract and integrate the most discriminative features in the infrared and visible images, facilitating long-range contextual interaction and enhancing global cross-modal features, we design a feature fusion block (FFB). This module integrates two features calibrated by CMFC into a feature map, which is then added to the output of different levels of the decoder to transform into the ultimate output feature. The framework of FFB is depicted in Figure 5.

Assuming the i-th HPDCB of the visible and infrared branches generate $f_i^{vi}$ and $f_i^{ir}$, respectively, we first adopt GAP and GMP to obtain global features. Subsequently, after a series of processing, the channel attention weights $cat{t}_i^x,(x=ir,vi)$ generated by the infrared and visible branches are obtained. These attention weights are then multiplied with the corresponding input features $f_i^x,(i=1,2,3)$, aiding the model in suppressing unimportant scene features and emphasizing crucial ones. Following this, the attention weights of these two branches are added; then, the softmax function is employed to acquire the cross-channel attention weight $W_i^c$. The multiplication of the input features and weight $W_i^c$ of the visible and infrared images yields the cross-channel attention output for both modalities. Additionally, $f_i^{ir}$ and $f_i^{vi}$ undergo a spatial attention module (composed of MP, AP, convolution, and sigmoid function), producing the spatial attention output $sat{t}_i^x$ for the visible and infrared images. Subsequent operations are similar to the above process (see Figure 5). The final cross-modal fusion feature $f_i$ is acquired by adding the acquired $f_i^c$ and $f_i^s$. The process mentioned above can be formulated as follows:

$$\begin{array}{c}ga{p}_i^x=relu\left(conv1\times 1\left(GAP\left(f_i^x\right)\right)\right)\hfill \\ gm{p}_i^x=relu\left(conv1\times 1\left(GMP\left(f_i^x\right)\right)\right)\hfill \\ cat{t}_i^x=sig(ga{p}_i^x+gm{p}_i^x)\hfill \\ {\widehat{F}}_{ci}^x=cat{t}_i^x\otimes f_i^x\hfill \\ W_i^c=softmax(cat{t}_i^{ir}+cat{t}_i^{vi})\hfill \\ F_{ci}^x=W_i^c\otimes f_i^x+\widehat{F_{ci}^x}\hfill \\ f_i^c=conv1\times 1\left(concat(F_{ci}^{ir},F_{ci}^{vi})\right)\hfill \end{array},$$

(6)

$$\begin{array}{c}sat{t}_i^x=sig\left(conv7\times 7\left(concat(AP\left(f_i^x\right),MP\left(f_i^x\right))\right)\right)\hfill \\ {\widehat{F}}_{si}^x=sat{t}_i^x\otimes f_i^x\hfill \\ W_i^s=softmax(sat{t}_i^{ir}+sat{t}_i^{vi})\hfill \\ F_{si}^x=W_i^s\otimes f_i^x+\widehat{F_{si}^x}\hfill \\ f_i^s=conv1\times 1\left(concat(F_{si}^{ir},F_{si}^{vi})\right)\hfill \end{array},$$

(7)

$$f_i=(f_i^c+f_i^s)\times 0.5,$$

(8)

where $relu$ is the ReLU activation function and $softmax$ denotes the softmax activation function.

3.5. Feature Reconstruction Module (FR)

To ensure that the reconstructed visible and infrared images incorporate complete information, we designed the FR module. This module aims to constrain the final reconstruction features to contain more comprehensive information by reconstructing the visible and infrared images. Figure 6 depicts the architecture of the FR. Assuming X is the input feature, the workflow of FR can be expressed as follows:

$$\begin{array}{l}{R}_1=B\left(conv\left(relu\left(B\left(conv\left(relu\left(B\left(conv\left(X\right)\right)\right)\right)\right)\right)\right)\right)\\ R_2=B\left(conv\left(relu\left(B\left(conv\left(GAP\left(X\right)\right)\right)\right)\right)\right)\\ R_3=\mathit{sig}(R_1\otimes R_2)\otimes X\\ R=\mathit{tanh}\left(\mathit{conv}\right(B\left(R_3\right)\left)\right)\end{array},$$

(9)

where B denotes batch normalization, $conv$ stands for convolution with kernel size $3\times 3$, $relu$ stands for ReLU activation function, $GAP$ represent global average pooling, and $tanh$ is Tanh activation function.

3.6. Loss Function

Designing the loss function is pivotal for model training, as it not only guides the optimization direction of the model but also influences the proportion of various information from the source images preserved in the fusion images. The loss function of the proposed method is formed by five terms: intensity loss ($L_{\mathrm{int}}$), detail loss ($L_{\mathrm{detail}}$), structural similarity loss ($L_{ssim}$), triplet loss ($L_{tri}$), and reconstruction loss ($L_{re}$). We express the total loss $L_{total}$ as follows:

$$L_{total}=\alpha L_{\mathrm{int}}+\beta L_{\mathrm{detail}}+L_{ssim}+L_{\mathrm{tri}}+\gamma L_{re},$$

(10)

where the coefficients $\alpha$, $\beta$, and $\gamma$ are employed to harmonize various loss functions.

To retain important targets in source images, we introduce a saliency-related intensity loss, defined as follows:

$$L_{\mathrm{int}}=MSE(x_f,(W_{ir}\otimes x_{ir}+W_{vi}\otimes x_{vi})),$$

(11)

where MSE is the mean squared error; $x_f$, $x_{vi}$, and $x_{ir}$ stand for the fusion, visible, and infrared image; $W_{iv}$ and $W_{vi}$ denote weighted maps, $W_{vi}=S_{vi}/(S_{vi}-S_{ir})$ and $W_{ir}=1-W_{vi}$; and S denotes the saliency matrix, which is computed by [45].

To maintain abundant texture details in the fusion images, we incorporate a detail loss, defined as follows:

$$L_{\mathrm{detail}}={∥\left|\nabla x_f\right|-max\left(\left|\nabla x_{ir}\right|,\left|\nabla x_{vi}\right|\right)∥}_1,$$

(12)

where ${\parallel ·\parallel }_1$ and ∇ denote the $l_1$-norm and Sobel gradient operator; $max(·)$ and $|·|$ stand for the element-wise maximum values operation and the absolute operation symbol.

In order to achieve an ideal image that possesses rich texture information, prominent target information, and also retains the overall structure of source images, a modified structural similarity loss [46] $L_{ssim}$ is introduced into $L_{total}$, defined as follows:

$$L_{\mathrm{ssim}}=\left\{\begin{array}{c}1-SSIM\left(x_{vi},x_f\right)\hfill \\ \mathrm{if}{\sigma }^2\left(x_{vi}\right)>{\sigma }^2\left(x_{ir}\right)\hfill \\ 1-SSIM\left(x_{ir},x_f\right)\hfill \\ \mathrm{if}{\sigma }^2\left(x_{ir}\right)>={\sigma }^2\left(x_{vi}\right)\hfill \end{array}\right.,$$

(13)

where ${\sigma }^2$ denotes variance. The use of SSIM provides a metric for quantifying the similarity between two images, where a higher value corresponds to a stronger similarity between the images.

Furthermore, to assist the network in learning more discriminative feature representations and enhance the fusion image quality, we introduce the triplet loss $L_{\mathrm{tri}}$.

$$L_{\mathrm{tri}}={\displaystyle \frac{1}{n}}\underset{i=1}{\sum ^3}max(d(f_i,f_i^{pos})-d(f_i,f_i^{neg})+b,0),$$

(14)

where the value of n is 3, representing the count of convolutional blocks involved; $d\left(·\right)$ denotes the Euclidean distance; ∑ stands for summation operator; $pos$ and $neg$ are positive and negative samples; and b represents a parameter and is set to 1.0.

We design an FR module to ensure that the reconstructed source images contain more comprehensive information, thereby optimizing the final fusion result. Accordingly, a reconstruction loss function is introduced, which is defined as follows:

$$\begin{array}{c}\hfill L_{re}=\parallel |\nabla x_{ir}|-|\nabla {\widehat{x}}_{v_i}{|\parallel }_1+\parallel |\nabla x_{vi}|-|\nabla {\widehat{x}}_{vi}{|\parallel }_1\\ \hfill +MSE(x_{ir},{\widehat{x}}_{ir})+MSE(x_{vi}-{\widehat{x}}_{vi}),\end{array}$$

(15)

where ${\widehat{x}}_{ir}$ stands for reconstructed infrared image and ${\widehat{x}}_{vi}$ denotes reconstructed visible image.

4. Experiments and Results

We evaluate the fusion performance of our method by conducting various comparative experiments in this section. Firstly, comprehensive details are presented regarding the experimental datasets, training details, comparison methods, and metrics for evaluation. Then, the exceptional performance of the proposed method is demonstrated through conducting quantitative and qualitative comparisons on different public datasets. Subsequently, we execute ablation studies on each key module to validate their essentiality. Finally, we conduct fusion efficiency experiments. Furthermore, we extend the experiments to object detection, showing that our method enhances the performance of downstream high-level visual tasks.

4.1. Datasets and Training Details

(1) Datasets. $M^{3}FD$: The $M^{3}FD$ dataset [5] contains high-resolution visible and infrared image pairs of various object types and scenes. The image pairs cover four typical types in different seasons, daytime, overcast, and night. In our study, we choose 2720 image pairs from this dataset to form our training set. The diversity of $M^{3}FD$ provides convenience for exploring image fusion algorithms.

TNO: The TNO dataset https://figshare.com/articles/TN_Image_Fusion_Dataset/1008029 (accessed on 30 January 2024), which contains multi-band nighttime images depicting military scenarios, is generally applied in the fusion of visible and infrared. We randomly choose 39 image pairs from it to serve as the test set.

RoadScene: The Roadscene dataset https://github.com/hanna-xu/RoadScene (accessed on 30 January 2024) is composed of 221 aligned pairs of visible and infrared images. These image pairs showcase representative traffic scenes, including pedestrians, traffic signs, vehicles, and roads. The dataset has been formed through careful preprocessing and image registration of some of the most representative scenes from the FILR dataset.

LLVIP: Most of the images in the LLVIP dataset [47] are taken in very dark scenes, making it applicable for low-light vision. This dataset can verify the effectiveness of fusion algorithms under low-light conditions.

(2) Training details. In this work, we utilize 2720 image pairs from the $M^{3}FD$ dataset for model training. We perform extensive quantitative and qualitative evaluations of our method and comparison methods on the $M^{3}FD$, RoadScene, TNO, and LLVIP datasets. To expand the training data, increase data diversity, and mitigate the risk of overfitting [48], we employ random horizontal flipping during the training phase and resize the images to 352 × 352. During the training process, the parameter update employs the Adam optimizer, with a batch size of 16. The learning rate and training epochs are set to ${10}^{-4}$ and 100, respectively. The coefficients $\alpha =50$, $\beta =50$, and $\gamma =5$ are determined based on extensive experiments and experience. We employ the PyTorch 1.7 framework to implement our approach and train it on an NVIDIA A100 GPU (NVIDIA, Santa Clara, CA, USA).

4.2. Comparison Methods and Evaluation Metrics

We will assess the fusion images from subjective and objective perspectives to validate the fusion performance of our method. For subjective evaluation, the primary factor is human visual perception. The fusion image’s quality is evaluated by the detail richness and target salience, which inherently entails a certain level of subjectivity. On the other hand, objective evaluation methods rely on quantitative metrics to objectively assess image quality.

Comparison methods. We assess the fusion performance of our method by comparing it with twelve state-of-the-art (SOTA) fusion methods (GTF [49], FGAN [8], STDF [24], SwinF [26], TarDAL [5], SeAF [22], SuperF [10], DIDF [19], BDLF [50], SHIP [51], DSF [9], and TCMoA [52]). To ensure fairness, we adhere to the parameter settings from the original publications by the authors in our comparative experiments, without modifying any other configurations.

Evaluation metrics. Given the deficiency of ground truth in IVIF, it is insufficient to rely solely on subjective visual assessments to evaluate the quality of fusion images, particularly when the two fusion images are nearly indistinguishable visually. To make the results more convincing, we select twelve quantitative metrics to comprehensively and objectively evaluate our method, including the sum of the correlation differences (SCD) [53], correlation coefficient (CC) [54], average gradient (AG) [55], visual information fidelity (VIF) [56], mean square error (MSE) [57], peak signal-to-noise ratio (PSNR) [58], edge information ($Q_{abf}$) [59], multi-scale structural similarity (MSSSIM) [60], structural similarity (SSIM) [61], mutual information (based on wavelet feature ($FM{I}_w$), discrete cosine feature ($FM{I}_d$), and pixel feature ($FM{I}_p$)) [62]. Except for MSE, higher values indicate superior fusion performance.

4.3. Comparison Experiment

We proceed with a comprehensive experiment on the $M^{3}FD$ dataset to validate the superiority of our method in fusion performance. The following will present both qualitative and quantitative analyses of the results obtained from our method as well as the comparison methods.

Qualitative analysis. Figure 7 shows two groups of qualitative experimental results from the $M^{3}FD$ dataset. In the first group of images, the visible image fails to provide effective scene details due to contamination. For instance, distinguishing the target person and the car logo inside the red box is challenging in the visible image but clear in the infrared image. The fusion images effectively integrate complementary features from visible and infrared images, but present visual differences across different methods. The results of GTF and FGAN tend towards the infrared image, while STDF, SwinF, TarDAL, SeAF, SuperF, DIDF, and BDLF preserve target information from the source image but lack texture detail. SHIP, DSF, and TCMoA retain significant texture detail but overlook contrast information. Remarkably, the proposed method not only performs well in preserving texture details and salient targets but also has superior visual effects. According to the analysis of the results, our method can effectively utilize infrared image information to complement severely contaminated visible images when they fail to provide scene details. This ability is attributed to CFMR and FFB, which calibrate and supplement the complementary features in source images. Furthermore, our method still demonstrates commendable performance under conditions of low illumination and when targets and backgrounds are similar, as illustrated in the second group in Figure 7. Overall, our method provides more thorough scene data and clearer visuals than other comparison methods.

Quantitative analysis. To enhance the credibility of the evaluation results, we conduct a quantitative analysis using twelve evaluation metrics, as illustrated in Figure 8. The testing set comprises 76 image pairs randomly selected from the $M^{3}FD$ dataset. To make comparison easier, we rank the mean values of the twelve evaluation metrics for different methods to obtain an average ranking, as indicated in

Table 1. The ranking results of different metrics on the $M^{3}FD$ dataset for different methods. Red serves to highlight the best result, blue for the second-best, and green for the third-best.

	PSNR ↑	MSE ↓	VIF ↑	AG ↑	CC ↑	SCD ↑	${\mathit{Q}}_{\mathit{abf}}$↑	SSIM ↑	MSSSIM ↑	${\mathit{FMI}}_{\mathit{p}}$↑	${\mathit{FMI}}_{\mathit{d}}$↑	${\mathit{FMI}}_{\mathit{w}}$↑
GTF	10	10	12	9	13	10	8	8	9	8	3	4
FGAN	4	4	13	13	11	12	13	13	13	11	5	10
STDF	6	6	8	7	9	11	7	10	12	9	4	3
SwinF	12	12	4	6	6	3	4	3	4	4	2	2
TarDAL	7	8	9	8	8	8	12	12	11	13	13	12
SeAF	13	13	6	5	7	4	5	2	3	6	12	11
SuperF	5	5	10	11	5	7	10	9	6	10	9	5
DIDF	8	7	7	4	3	1	9	7	5	12	8	7
BDLF	2	2	11	12	2	6	11	11	8	7	7	9
SHIP	11	11	2	1	10	9	3	6	7	2	11	8
DSF	9	9	5	2	12	13	2	4	10	3	6	6
TCMoA	3	3	3	10	4	5	6	5	2	5	10	13
Ours	1	1	1	3	1	2	1	1	1	1	1	1

. According to the ranking results, our method excels over others in metrics such as PSNR, MSE, VIF, CC, $Q_{abf}$, SSIM, MSSSIM, $FM{I}_p$, $FM{I}_d$, and $FM{I}_w$. Regarding SCD, our results outperform all methods except DIDF, and we achieve the third-best in AG. Quantitative analysis demonstrates that our method can ensure the maximum preservation of texture details in visible images while retaining the prominent target in infrared images.

4.4. Generalization Experiment

Generalization assessment plays a vital role in evaluating fusion algorithms. We perform comprehensive experiments on the TNO, RoadScene, and LLVIP datasets to additionally validate the generalization and scalability of our method. The following will present a detailed analysis of the qualitative and quantitative experimental results of different methods on these three datasets.

4.4.1. Results on TNO Dataset

Qualitative analysis. We compare our method against twelve SOTA fusion methods to assess its generalization performance and robustness. Figure 9 presents the fusion results of two representative scenes: one is a smoke scene and the other is a daytime scene with low illumination. To facilitate the observation of the fusion results, we use red and green rectangles to mark some significant targets and texture detail information, respectively, in Figure 9. As observed in Figure 9, all methods can complete the basic fusion task. However, our method exhibits remarkable performance in subjective visual quality and retaining the crucial features in the source image. The fusion images generated by GTF and FGAN tend to infrared images, preserving the significant targets from the infrared images; nevertheless, they fail to integrate the texture detail from the visible images. STDF, SwinF, TarDAL, SeAF, SuperF, DIDF, BDLF, SHIP, and DSF can retain plentiful texture details from the visible images but insufficiently maintain significant targets from the infrared images. TCMoA can maintain the prominent targets from the infrared images and the texture details from the visible images but suffers from low contrast. In comparison, our method has satisfactory visual quality while retaining abundant texture detail from visible images and significant targets from infrared images.

Quantitative analysis. Figure 10 and

Table 2. The ranking results of different metrics on the TNO dataset for different methods. Red serves to highlight the best result, blue for the second-best, and green for the third-best.

	PSNR ↑	MSE ↓	VIF ↑	AG ↑	CC ↑	SCD ↑	${\mathit{Q}}_{\mathit{abf}}$↑	SSIM ↑	MSSSIM ↑	${\mathit{FMI}}_{\mathit{p}}$↑	${\mathit{FMI}}_{\mathit{d}}$↑	${\mathit{FMI}}_{\mathit{w}}$↑
GTF	10	11	13	12	13	13	12	12	11	9	13	13
FGAN	13	9	12	13	10	12	13	13	13	12	4	6
STDF	8	8	2	4	11	11	7	11	12	10	3	2
SwinF	12	13	5	3	8	5	4	6	7	8	2	3
TarDAL	6	5	6	7	6	6	10	10	8	11	12	10
SeAF	9	10	1	1	4	3	6	7	4	5	10	12
SuperF	4	7	11	11	7	8	8	3	6	7	5	4
DIDF	7	4	8	8	2	2	11	8	3	13	7	5
BDLF	2	2	9	10	3	4	9	9	5	6	6	8
SHIP	11	12	3	2	9	9	3	5	9	1	11	9
DSF	3	3	10	6	12	10	2	2	10	4	8	7
TCMoA	5	6	4	9	5	7	5	4	2	3	9	11
Ours	1	1	7	5	1	1	1	1	1	2	1	1

demonstrate the quantitative results of different methods. It is apparent from the results that our method excels over others in terms of PSNR, MSE, CC, SCD, $Q_{abf}$, SSIM, MSSSIM, $FM{I}_d$, and $FM{I}_w$, and ranks second only to MPCF on $FM{I}_p$. The analysis of these metrics indicates that our fusion results are extremely similar to the source images—with low noise, rich details, and high contrast—and demonstrate excellent visual effects.

4.4.2. Results on RoadScene Dataset

Qualitative analysis. IVIF is instrumental in various traffic applications. Therefore, we choose to test twelve SOTA methods and our method on the RoadScene dataset. Figure 11 displays two sets of the most representative image pairs. The first set is a daytime scene and the second is a night scene with strong light illumination. The details of the utility poles are clearly depicted in our fusion results for the first group of images, while in the second group, even the details of a person concealed in darkness are clearly visible from our result. In comparison, the fusion results from GTF and FGAN are relatively blurry and lose important salient target information, such as the person within the green rectangle of the second set. DIDF, BDLF, and DSF suffer from low contrast, while STDF, SwinF, TarDAL, SeAF, and SuperF lose some texture detail information, marked in red rectangles in Figure 11. While SHIP, TCMoA, and our method retain salient target information and sufficient texture details, SHIP and TCMoA still have some artifacts. Generally, our method maintains essential features from the source images while mitigating interference from useless information, conforming to human visual perception.

Quantitative analysis. Quantitative analysis was conducted on 110 image pairs randomly selected from the RoadScene dataset. Figure 12 and

Table 3. The ranking results of different metrics on the RoadScene dataset for different methods. Red serves to highlight the best result, blue for the second-best, and green for the third-best.

	PSNR ↑	MSE ↓	VIF ↑	AG ↑	CC ↑	SCD ↑	${\mathit{Q}}_{\mathit{abf}}$↑	SSIM ↑	MSSSIM ↑	${\mathit{FMI}}_{\mathit{p}}$↑	${\mathit{FMI}}_{\mathit{d}}$↑	${\mathit{FMI}}_{\mathit{w}}$↑
GTF	9	10	11	11	13	13	12	11	11	6	4	10
FGAN	13	13	13	13	11	11	13	13	12	11	6	13
STDF	12	11	1	2	7	6	10	12	13	7	7	4
SwinF	11	12	5	10	5	4	8	9	7	9	8	5
TarDAL	4	5	9	8	9	9	9	10	9	12	13	9
SeAF	8	8	3	1	6	5	5	6	4	10	9	6
SuperF	6	6	7	9	8	8	6	4	5	5	1	3
DIDF	5	4	8	6	2	1	7	2	3	13	3	2
BDLF	1	2	10	12	3	7	11	8	6	8	5	8
SHIP	10	9	2	3	10	10	3	7	10	3	12	11
DSF	7	7	12	5	12	12	2	5	8	1	11	12
TCMoA	3	3	4	7	4	3	4	3	2	4	10	7
Ours	2	1	6	4	1	2	1	1	1	2	2	1

present our quantitative results, which reveal that our method excels in multiple evaluation metrics. Specifically, our method excels in MSE, CC, $Q_{abf}$, SSIM, MSSSIM, and $FM{I}_w$, achieving the highest scores, while ranking second in PSNR, SCD, $FM{I}_p$, and $FM{I}_d$. In general, the proposed method exhibits excellent generalization performance and can still achieve satisfactory fusion results even in environments with small differences between the background and the target and insufficient lighting.

4.4.3. Results on LLVIP Dataset

Qualitative analysis. Compared to the fusion of daytime scenes, the low illumination image fusion presents more challenges. When visible images are affected by strong illumination sources and fail to provide sufficient scene information, high-performance fusion methods can effectively utilize infrared images for supplementation. To additionally confirm the superiority of our method, a fusion test on the LLVIP dataset is conducted, with qualitative results depicted in Figure 13. The first group depicts scenes with insufficient illumination at night, while the second group shows nighttime scenes with strong illumination interference. From the displayed results, our method preserves significant targets from the infrared image and the abundant texture details from the visible image while also effectively resisting strong illumination interference to generate high-quality fusion images.

Quantitative analysis. We conduct a quantitative analysis using 96 image pairs randomly selected from the LLVIP dataset. As depicted in Figure 14 and

Table 4. The ranking results of different metrics on the LLVIP dataset for different methods. Red serves to highlight the best result, blue for the second-best, and green for the third-best.

	PSNR ↑	MSE ↓	VIF ↑	AG ↑	CC ↑	SCD ↑	${\mathit{Q}}_{\mathit{abf}}$↑	SSIM ↑	MSSSIM ↑	${\mathit{FMI}}_{\mathit{p}}$↑	${\mathit{FMI}}_{\mathit{d}}$↑	${\mathit{FMI}}_{\mathit{w}}$↑
GTF	10	10	8	6	13	11	8	7	9	10	3	1
FGAN	5	5	13	13	10	13	13	13	13	13	11	13
STDF	4	4	11	9	12	12	7	9	12	8	4	7
SwinF	7	7	3	4	3	2	3	2	3	3	2	3
TarDAL	13	13	10	8	9	8	11	11	10	12	13	11
SeAF	11	12	5	5	2	3	5	4	5	5	9	2
SuperF	6	6	9	11	5	7	9	8	7	6	8	10
DIDF	12	11	12	10	6	5	12	12	11	11	10	12
BDLF	2	2	6	12	8	9	10	10	8	9	5	9
SHIP	9	9	4	1	7	6	2	3	4	2	7	8
DSF	3	3	7	3	11	10	4	5	6	4	6	5
TCMoA	8	8	2	7	4	4	6	6	2	7	12	6
Ours	1	1	1	2	1	1	1	1	1	1	1	4

, the quantitative results demonstrate that our method outperforms others on multiple evaluation metrics. Specifically, it excels in PSNR, MSE, VIF, CC, SCD, $Q_{abf}$, SSIM, MSSSIM, $FM{I}_p$, and $FM{I}_d$. On the AG metric, our method is ranked second. On the $FM{I}_w$ metric, our method ranks fourth. Overall, the proposed method exhibits good generalization ability and robustness, efficiently accomplishing image fusion tasks in various complex environments.

4.5. Ablation Experiment

We will evaluate the effectiveness of each module in this section. The proposed method comprises three crucial modules, namely, CMFC, FFB, and HPDCB. In our ablation experiments, we verify different combinations of these modules. Additionally, we perform ablation experiments related to FR and $l_{tri}$ loss.

Quantitative analysis. As detailed in

Table 5. The quantitative analysis of various modules on the $M^{3}FD$ dataset. Red serves to highlight the best result, blue for the second-best, and green for the third-best.

Model	M1	M2	M3	M4	M5	M6	M7 (Ours)
CMFC	×	×	✓	✓	✓	✓	✓
FFB	×	✓	×	✓	✓	✓	✓
HPDCB	×	✓	✓	×	✓	✓	✓
FR	×	✓	✓	✓	×	✓	✓
$L_{tri}$	×	✓	✓	✓	✓	×	✓
PSNR ↑	63.6507	63.4210	63.5573	63.4941	63.3877	63.4001	63.3531
MSE ↓	0.0329	0.0341	0.0334	0.0336	0.0342	0.0342	0.0346
VIF ↑	0.7298	0.8480	0.7603	0.7797	0.8410	0.8491	0.8977
AG ↑	5.9649	6.0204	5.9805	5.9546	5.9800	5.9264	6.0230
CC ↑	0.5946	0.5807	0.5930	0.5984	0.5883	0.5880	0.5995
SCD ↑	1.6584	1.6623	1.6962	1.7565	1.7076	1.7179	1.7835
$Q_{abf}$↑	0.6147	0.6857	0.6160	0.6248	0.6766	0.6697	0.6803
SSIM ↑	0.9804	0.9812	0.9800	0.9800	0.9809	0.9801	0.9797
MSSSIM ↑	0.9643	0.9642	0.9674	0.9678	0.9640	0.9638	0.9638
$FM{I}_p$↑	0.8886	0.8988	0.8877	0.8883	0.8973	0.8970	0.8977
$FM{I}_d$↑	0.3684	0.4158	0.3871	0.3804	0.4236	0.4337	0.4317
$FM{I}_w$↑	0.3832	0.4246	0.3920	0.3991	0.4264	0.4301	0.4341

, seven groups of ablation experiments were executed to assess the necessity of our proposed modules. According to the quantitative indicators from Table 5, our method exhibits excellent performance in VIF, AG, CC, SCD, and $FM{I}_w$, and ranks second in $Q_{abf}$, $FM{I}_p$, and $FM{I}_d$. Overall, every module we proposed contributes to improving the quality of the fusion result.

Qualitative analysis. Figure 15 presents the qualitative results of two groups of ablation experiments. It is evident from the displayed results that the fusion results of M1, M2, M3, and M4 exhibit some blurring, while the contrast of M5 and M6 is also lower compared to our method. Generally speaking, our method excels in maintaining high contrast and complete information while also demonstrating excellent clarity compared to other incomplete combination modules.

4.6. Efficiency Comparison

The average runtime of various fusion methods on the TNO dataset is presented in

Table 6. The average runtime of various fusion methods on the TNO dataset.

Methods	GTF	FGAN	STDF	SwinF	TarDAL	SeAF	SuperF
time (s)	2.8868	0.4853	0.3270	2.4847	0.7605	0.0989	0.2218
Methods	DIDF	BDLF	SHIP	DSF	TCMoA	Ours	-
time (s)	1.4027	0.2110	0.4293	0.1614	1.0688	0.2025	-

. We conduct all experiments on the same device to ensure fairness. The result from Table 6 reveals that the proposed method surpasses GTF, FGAN, STDF, SwinF, BDLF, TarDAL, SuperF, DIDF, SHIP, and TCMoA in runtime. Overall, the proposed method demonstrates high fusion efficiency while ensuring fusion performance.

4.7. Fusion for Object Detection

IVIF lacks ground truth, often necessitating qualitative and quantitative analyses to assess algorithm performance. However, it remains uncertain as to whether fusion images enhance performance in downstream visual tasks (e.g., object detection). To explore this question, experiments were conducted on the $M^{3}FD$ dataset. To ensure fairness, we employ the YOLOv5 framework to perform object detection on visible, infrared, and fusion images.

Qualitative analysis. To facilitate the observation of differences in object detection performance between various fusion images and source images, two groups of visualized object detection examples are presented (see Figure 16). In the first scenario, YOLOv5 can accurately detect people in the images. However, the results suggest that our fusion results significantly enhance the image clarity and the prominence of the person, leading to higher detection accuracy compared to other fusion images and the source image. In the second scenario, YOLOv5 loses some important target information, such as people, when detecting the source images and the fusion images from other SOTA methods. Our fusion images detect more comprehensive targets, demonstrating that our method effectively integrates useful information from source images, thereby improving the precision of target detection. Our method achieves higher detection precision relative to the twelve SOTA fusion methods.

Quantitative analysis. As indicated in

Table 7. The detection precision of different images. Red serves to highlight the best result, blue for the second-best, and green for the third-best. VI and IR represent the visible and infrared images.

	Precision	[email protected]	mAP@[0.5:0.95]
VI	0.6737	0.5979	0.3916
IR	0.6027	0.5305	0.3003
GTF	0.5351	0.5492	0.3411
FGAN	0.5338	0.5117	0.3148
STDF	0.5760	0.5632	0.3512
SwinF	0.5984	0.5783	0.3716
TarDAL	0.6236	0.5997	0.3820
SeAF	0.5835	0.5772	0.3659
SuperF	0.5699	0.5365	0.3367
DIDF	0.6256	0.5941	0.3774
BDLF	0.6124	0.5751	0.3597
SHIP	0.5619	0.5316	0.3347
DSF	0.6596	0.6421	0.4279
TCMoA	0.6220	0.5890	0.3752
Ours	0.6995	0.6533	0.4341

, we calculate the average precision for each fusion method to further assess their detection performance. The results demonstrate that our method obtains the best detection accuracy. This further validates that the proposed method positively impacts practical object detection tasks.

5. Conclusions

In this study, we propose BCMFIFuse—a network based on bilateral cross-modal feature interaction for IVIF. Firstly, to effectively extract features from the source image, we construct a dual-stream feature extraction network. Next, a CMFC is introduced to calibrate the features of the current modality to better extract complementary features from different modalities. Subsequently, we employ an FFB to effectively integrate the calibrated features. The FFB is built using a cross-attention mechanism, which can realize long-range contextual interaction, thereby enhancing global bilateral modal features. Finally, to ensure the continuity of features and minimize feature loss during transmission, we use shortcut connections between the encoder and decoder. Additionally, for gathering specific and diversified context information and capturing long-range dependencies in isolated areas, we design an HPDCB. Comparison and generalization experiments on multiple datasets indicate that our method has certain advantages in quantitative and qualitative aspects. Furthermore, the evaluation of object detection performance can also reflect the superiority of our method. In upcoming research, we plan to continue optimizing our algorithm to improve fusion efficiency. Moreover, we consider integrating fusion tasks with other high-level visual tasks or modal information (e.g., text information).