A Three-Branch Pansharpening Network Based on Spatial and Frequency Domain Interaction

Abstract

Pansharpening technology plays a crucial role in remote sensing image processing by integrating low-resolution multispectral (LRMS) images and high-resolution panchromatic (PAN) images to generate high-resolution multispectral (HRMS) images. This process addresses the limitations of satellite sensors, which cannot directly capture HRMS images. Despite significant developments achieved by deep learning-based pansharpening methods over traditional approaches, most existing techniques either fail to account for the modal differences between LRMS and PAN images, relying on direct concatenation, or use similar network structures to extract spectral and spatial information. Additionally, many methods neglect the extraction of common features between LRMS and PAN images and lack network architectures specifically designed to extract spectral features. To address these limitations, this study proposed a novel three-branch pansharpening network that leverages both spatial and frequency domain interactions, resulting in improved spectral and spatial fidelity in the fusion outputs. The proposed method was validated on three datasets, including IKONOS, WorldView-3 (WV3), and WorldView-4 (WV4). The results demonstrate that the proposed method surpasses several leading techniques, achieving superior performance in both visual quality and quantitative metrics.

1. Introduction

High-resolution multispectral (HRMS) imagery has numerous applications in the domain of remote sensing image processing, such as target detection [1], military defense [2], climate monitoring [3], area classification [4], and so on. However, due to constraints associated with hardware and physical imaging capabilities, various satellite sensors are unable to directly capture HRMS data. Instead, they can only acquire paired low-resolution multispectral (LRMS) images, which contain extensive spectral information, along with panchromatic (PAN) images that offer high spatial resolution. Therefore, pansharpening technology has been developed to merge the information from these two types of images, thereby producing HRMS imagery that encompasses both rich spectral and spatial information.

Generally speaking, pansharpening methods can be divided into traditional methods and deep learning-based methods. Among them, traditional methods can be divided into three categories: the component substitution (CS) method [5], the multiresolution analysis (MRA) method [6], and the variational optimization (VO) method [7].

The CS method transforms multispectral images into a certain coordinate space, separates the spectral and spatial information, replaces the spatial information with PAN images that have been histogram-matched, and finally generates HRMS through inverse transformation. Classic algorithms include GS [8], IHS [9], and PCA [10]. This type of method will destroy the original spectral structure and produce fusion results with serious spectral distortion.

The MRA method extracts spatial detail structures by designing high-pass filters and then injects these spatial detail structures into multispectral images to obtain HRMS. Representative algorithms include GLP [11], HPF [12], and SFIM [13]. The spectral information preservation ability of this type of method is good, but during the process of injecting spatial details, spatial details are lost and introduced as shadows, resulting in poor spatial quality of the fusion result.

The VO method constructs an energy function by analyzing the relationship between LRMS, PAN, and HRMS and optimizes the model by optimizing the energy function. Typical methods include observation model-based methods [14,15] and sparse representation-based methods [16,17]. This type of method relies heavily on prior information, which often has unpredictability, resulting in a certain degree of distortion in the spectral and spatial information of the final fusion result.

The pansharpening method utilizing deep learning techniques, particularly convolutional neural networks (CNNs), has garnered significant attention in recent years. The robust non-linear feature extraction capabilities of CNNs have facilitated their extensive application in the domain of pansharpening. Drawing inspiration from super-resolution convolutional neural networks (SRCNNs) [18], Masi et al. [19] were among the first to incorporate CNNs into pansharpening networks, developing a three-layer convolutional network architecture to achieve pansharpening results. Building upon this foundation, Wei et al. [20] designed convolutional kernels of varying sizes to extract information at different scales from the source images. Recognizing that LRMS and PAN images contain different modalities of information, Liu et al. [21] proposed a dual-stream fusion network that employs independent branches to extract features from both LRMS and PAN source images. However, these independent branches, despite having identical structures, still struggle to effectively characterize spectral and spatial features. In response to this limitation, Yong et al. [22] developed separate channel and spatial attention (SA) mechanisms [23] to extract spectral and spatial features, respectively. They subsequently multiplied the channel weights by the spatial weights to derive overall weights, thereby guiding LRMS and PAN branches to enhance their feature extraction capabilities. To mitigate redundant information across different subnetworks, Wang et al. [24] introduced an adaptive feature fusion module (AFFM) that infers weight maps for various branches from the feature map, thereby increasing the network’s flexibility. Other researchers reconstruct HRMS by extracting and combining common unique features. Some researchers reconstruct HRMS by extracting and combining common unique features. Xu et al. [25] used Convolutional Sparse Coding (CSC) to extract the lateral information of panchromatic images. At the same time, they split low-resolution multispectral images into panchromatic image-related feature maps and panchromatic image-independent feature maps. The former was regularized by the lateral information of the panchromatic image, and the proposed model was extended to a deep neural network using the principle of algorithm expansion techniques. Cao et al. [26] designed a CSC with two sets of filters (a common filter and a unique filter) to model PAN and MS images separately, and this model is called PanCSC. At the same time, they derived an effective optimization algorithm to optimize the PanCSC model. Inspired by the learnable iterative soft thresholding algorithm, Yin [27] proposed a coupled convolutional sparse coding-based panchromatic sharpening (PSCSC) model. The solution of PSCSC follows traditional algorithms, and PSCSC uses a deep unfolding strategy to develop interpretable end-to-end depth panchromatic sharpening networks. Zhi et al. [28] proposed a module based on cross attention to extract common and unique features from MS and PAN images. Lin et al. [29] encodes LRMS and PAN into unique and common features, then averages the common features and combines them with the unique features in the source image to reconstruct the fused image. Deng et al. [30] developed a cross-convolutional attention network that dynamically adjusts the parameters of the convolutional kernels, thereby enhancing the interaction between the two branches to obtain complementary information. Zhou et al. [31] proposed a PAN-guided multispectral feature enhancement network that employs multi-scale convolutional blocks to aggregate features across multiple scales. Jia et al. [32] introduced an attention-based progressive network that incrementally enhances detail information to generate an enhanced version of LRMS that corresponds to the size of PAN. Concurrently, they utilized concatenated spectral data and multi-scale SA blocks to progressively extract spatial and spectral features for the reconstruction of HRMS images. Despite the promising spectral and spatial quality of the fusion results produced by existing methods, several challenges remain to be addressed as follows:

• Many dual-branch architecture methods overlook the common features shared between LRMS and PAN images, such as those captured by [33], leading to architectural inefficiencies;
• A lack of adaptive convolution in most current approaches weakens the network’s flexibility and adaptability;
• The majority of methods operate solely in the spatial domain, without incorporating network structures designed to process frequency domain information.

To address the identified issues, this study developed a three-branch pansharpening network that leverages the interaction between spatial and frequency domains. Each branch is responsible for extracting different types of information: one branch extracts spectral information from LRMS images, another captures spatial information from PAN images, and the third branch integrates common information from both sources. The research implemented a spectral feature extraction module designed to identify nonlinear relationships among the channels in LRMS, thereby facilitating the acquisition of spectral information. Additionally, a spatial feature extraction module was created to capture spatial information from PAN images. This module employs multi-scale blocks to extract spatial features at various scales, followed by the application of SA mechanisms, as well as Sobel and Laplacian operators, to derive general spatial features and to enhance edge and texture features, respectively. Following the interaction and fusion of these two feature sets, a directional perception module is introduced to embed positional information, thereby augmenting the expressive capacity of the spatial features. For the common features derived from both LRMS and PAN, this study utilized a concatenation method that combines the bands from each image type separately and designed a dynamic convolution mechanism based on the extracted spectral and spatial features. Notably, the content of the convolution kernel is adapted according to the spectral and spatial characteristics of the input image. Subsequently, the features obtained from both the spectral and spatial feature extraction modules are processed to derive their corresponding amplitude and phase using discrete Fourier transform (DFT). These amplitude and phase features are then integrated and subjected to inverse discrete Fourier transform (IDFT) to yield frequency domain features. The branch responsible for extracting common features from LRMS and PAN utilizes these features as spatial domain inputs, which are then interacted with and fused with the frequency domain features, ultimately leading to the reconstruction of HRMS images. In summary, the contributions can be delineated as follows:

• A three-branch pansharpening network based on spatial and frequency domain interaction was proposed, considering the distinct modal characteristics of LRMS and PAN images, as well as their common features;
• A dynamic convolution based on spectral–spatial feature extraction was designed to improve network flexibility and adaptability;
• Spatial–frequency domain feature fusion module (SFFIM) was developed to achieve interactive fusion of spatial and frequency domain information, leading to more comprehensive and enriched fused image features.

2. Proposed Method

Figure 1 illustrates the overall proposed network architecture. The design of the network structure is approached from two perspectives: the upsampled version of the original LRMS images, along with their unique features, and the common features shared between them, where H represents the height of the image, W represents the width of the image, and B represents the number of bands present in LRMS. Acknowledging the differences between LRMS images and PAN images, this study developed a spectral feature extraction module to extract spectral information from LRMS images and a spatial feature extraction module to derive spatial information from PAN images. Furthermore, recognizing that LRMS images and PAN images share certain common features, the research implemented a common feature extraction module. Due to the specificity of each band in LRMS [34], the study focused on each band of LRMS images with PAN images individually as input for this branch. This approach facilitates substantial interaction between PAN image and each band of LRMS. Additionally, a feature fusion module was designed to adaptively integrate information from the various branches, ultimately performing feature reconstruction to yield HRMS images.

2.1. Spectral Feature Extraction Module

LRMS images encompass extensive spectral information, which is derived from the interrelationships among various spectral bands [35]. To facilitate the extraction of this spectral information, a spectral feature extraction module was developed (Figure 2). Firstly, a 3 × 3 convolution followed by a ReLU activation function is applied sequentially to preprocess LRMS image, resulting in the output $\tilde{\mathit{L}}\in {\mathit{R}}^{H\times W\times C}$. Subsequently, global average pooling (GAP) is performed on the output $\tilde{\mathit{L}}$, which is then processed through two parallel multi-layer perceptrons (MLPs) to generate two vectors (${\mathit{L}}_1, {\mathit{L}}_2\in R^{C\times 1}$) that represent the channel features. This entire procedure is expressed through Equation (1):

$${\mathit{L}}_i=MLP(GAP(\tilde{\mathit{L}}))i=1, 2$$

(1)

Subsequently, a matrix multiplication operation is conducted on matrices ${\mathit{L}}_1$ and ${\mathit{L}}_2^T$, where ${\mathit{L}}_2^T$ represents the matrix transposition of ${\mathit{L}}_2$. This is followed by the application of the Sigmoid activation function $\sigma \left(·\right)$ in the subsequent equation, to derive the relationship graph ${\mathit{M}}_{\mathit{C}}\in R^{C\times C}$ among the various channels. Concurrently, the shape of matrix $\tilde{\mathit{L}}$ is transformed to yield matrix $\overline{\mathit{L}}\in R^{(H\times W)\times C}$. Following this, matrix multiplication is performed on matrices $\overline{\mathit{L}}$ and ${\mathit{M}}_{\mathit{C}}$, after which their shapes are adjusted to produce matrix ${\mathit{L}}_3\in R^{H\times W\times C}$. This entire process can be represented in Equation (2):

$${\mathit{L}}_3=Re((\sigma ({\mathit{L}}_1{\mathit{L}}_2^T))\overline{\mathit{L}})$$

(2)

where Re (·) represents the undergoing shape changes. Finally, a skip connection operation is executed to retain the information from the source image, thereby yielding the output of the module ${\mathit{L}}_{\mathit{out}}$.

Figure 2. The architecture of the spectral feature extraction module.

Figure 2. The architecture of the spectral feature extraction module.

2.2. Spatial Feature Extraction Module

To acquire comprehensive spatial information in PAN imagery, this study developed a spatial feature extraction module comprising three parallel branches (Figure 3a). The subsequent sections will detail the functionality of these three branches in a top-to-bottom sequence (Figure 3).

Firstly, the PAN image was preprocessed using a 3 × 3 convolution and ReLU activation function, followed by a 3 × 3 convolution and ReLU activation function and a sequence of multi-scale blocks to preprocess the PAN image, thereby facilitating the preliminary extraction of multi-scale features. Subsequently, convolutional layers and ReLU activation functions were employed for channel dimensionality reduction, resulting in the preprocessed image $\tilde{\mathit{P}}$. In the first branch, this study utilized convolutional layers in conjunction with SA mechanisms to comprehensively extract spatial features from the PAN image while also implementing skip connection operations, which yielded the output of the branch ${\mathit{P}}_1$.

However, addressing this issue in isolation neglects the extraction of finer textures from PAN images, which are predominantly found in details such as object edges. This observation has also been noted by several scholars. Drawing inspiration from [36,37], this study incorporated the Sobel operator and Laplacian operator in the second branch to extract fine texture details. The Sobel operator is a first-order differential operator that emphasizes pixels in proximity to the central pixel and has a high weighting on the nearest pixel to the central pixel, thereby aiding in noise reduction. It can be used to preserve strong texture features. Some researchers have also demonstrated the superiority of Sobel in extracting detailed information [38]. However, the Sobel operator is only a first-order operator, and compared with first-order operators, second-order differential operators have better edge localization ability and better pansharpening effect. The Laplace operator in second-order operators is widely used for fine feature extraction in the field of image processing, such as [37,39], which can extract weak textures of features that complement previous strong textures. Therefore, we chose the Laplace operator. Specifically, the research focused on the features extracted by these two operators, subsequently applying a convolutional layer and executing a skip connection operation to derive the output of the branch ${\mathit{P}}_2$.

After extracting the features from the two aforementioned branches, a methodical approach was employed to integrate the information from these branches, ensuring that they mutually inform one another and that the details are examined. Specifically, for each branch, the Sigmoid activation function is utilized to derive the weight maps ${\mathit{M}}_1$ and ${\mathit{M}}_2$ corresponding to the feature maps of each branch. Subsequently, ${\mathit{M}}_1$ and ${\mathit{M}}_2$ are cross-multiplied with ${\mathit{P}}_1$ and ${\mathit{P}}_2$. Ultimately, the information from the two branches, following their interaction, is combined to yield ${\mathit{P}}_{12}$ (Equation (3)):

$${\mathit{P}}_{12}={\mathit{P}}_1·\sigma \left({\mathit{P}}_2\right)+{\mathit{P}}_2·\sigma \left({\mathit{P}}_1\right)$$

(3)

In addition to providing rich spatial information, including details and contours, PAN images also encompass positional information. Specifically, the rows and columns within PAN images exhibit interrelatedness. The first two branches do not distinguish between rows and columns of the image but use a unified operation, which means a lack of perception of positional information. To effectively capture this location information, this study developed a spatial perception module within the third branch (Figure 3c). Following the pooling, $\tilde{\mathit{P}}$, along the width and height dimensions of the feature map, feature representation in the height dimension ${\mathit{P}}_{3\mathit{h}}\in R^{1\times W\times C}$ and feature representation of the width dimension ${\mathit{P}}_{3\mathit{w}}\in R^{H\times 1\times C}$ were derived. Subsequently, a multiplication of ${\mathit{P}}_{3\mathit{h}}$ and ${\mathit{P}}_{3\mathit{w}}$ was performed along the channel dimension, applying the Sigmoid activation function to generate the direction-aware map ${\mathit{M}}_{\mathit{D}}$. Ultimately, the multiplication of variable ${\mathit{M}}_{\mathit{D}}$ with ${\mathit{P}}_{12}$, conducted channel by channel and element by element, makes each element in ${\mathit{P}}_{12}$ obtain a weight map with directional information, serves to embed the positional information into ${\mathit{P}}_{12}$, thereby yielding the output of the spatial feature extraction module ${\mathit{P}}_{\mathit{out}}$. This process is articulated in Equation (4) as follows:

$${\mathit{P}}_{\mathit{out}}=\sigma \left({\mathit{P}}_{3\mathit{w}}\ast {\mathit{P}}_{3\mathit{h}}\right)\odot {\mathit{P}}_{12}$$

(4)

where ∗ represents the matrix multiplication along the channel direction and ⊙ represents the multiplication channel by channel and element by element.

2.3. Common Feature Extraction Module

The integration of spatial information from LRMS images and spectral information from PAN images reveals a shared informational basis between the two modalities. To effectively harness this common information, this study developed a dedicated feature extraction module, the architecture of which is depicted in Figure 4. Given the unique characteristics of each spectral band in LRMS images, the research implemented a dynamic convolution kernel, termed spectral spatial feature extraction (SSDConv), tailored for the corresponding branch of each band, rather than employing conventional convolution with uniform parameters. This approach not only takes into account the different properties of each band but also enhances the model’s adaptability. The detailed structure of SSDConv is presented in Figure 5a.

The fundamental aspect of generating dynamic convolution kernels involves acquiring the weight matrix of local blocks $\mathit{W}\in R^{k\times k\times C_{in}}$, where the local blocks present the patch segments created by concatenating the PAN image with each band of LRMS data, followed by the segmentation of the feature map through preprocessing in the convolutional layer; k and ${\mathit{C}}_{\mathit{in}}$ represent the dimensions and the number of channels of the input local blocks, respectively. Given that the methodology for generating dynamic convolution kernels for each branch corresponding to a specific band is consistent, a particular band was used as a representative example and select a local block ${\mathit{F}}_{\mathit{P}}\in R^{k\times k\times {\mathit{C}}_{\mathit{in}}}$ to elucidate the process of generating dynamic convolution kernels.

${\mathit{F}}_{\mathit{P}}$ comprises information derived from a specific band of LRMS and PAN imagery. Consequently, numerous studies examined the channel and spatial characteristics of the feature map, leading to the development of three-branch feature extraction architectures that address both channel (one-dimensional) and spatial (two-dimensional) dimensions. However, the network structures of these three branches are identical and do not adequately account for the inherent differences between spatial and spectral features. To address this limitation, this study proposed a dynamic convolution approach focused on spectral–spatial feature extraction. For channel features, the research employed the “squeeze excitation” operation [40] (SE(·) in Equation (5)) to derive the weight for the channel dimension ${\mathit{W}}_{\mathit{C}}\in R^{1\times 1\times C}$. In terms of spatial features, this study utilized the SA mechanism $S{A}_W\left(·\right)$ and $S{A}_H\left(·\right)$ in Equation (5), to generate the corresponding weight maps ${\mathit{W}}_{\mathit{w}}\in R^{H\times 1\times C}$ and ${\mathit{W}}_{\mathit{h}}\in R^{1\times W\times C}$ for the width and height dimensions of the feature map, respectively. Ultimately, these three weights were combined to obtain W (Equation (5)):

$$\mathit{W}=SE\left({\mathit{F}}_{\mathit{P}}\right)\odot \left(S{A}_W\left({\mathit{F}}_{\mathit{P}}\right)\otimes S{A}_H\left({\mathit{F}}_{\mathit{P}}\right)\right)$$

(5)

where ⊙ represents the channel-wise multiplication and ⊗ represents the matrix multiplication along the channel direction. Following this, an inner product operation is conducted between W and a set of candidate kernels, $\mathit{K}\in R^{C_{out}\times k\times k\times C_{in}}$, to derive a dynamic convolution kernel. Subsequently, this dynamic convolution kernel is employed to execute convolution operations on the input local block, resulting in the corresponding features of the local block ${{\mathit{F}}^{\prime }}_{\mathit{P}}$ (Equation (6)):

$${{\mathit{F}}^{\prime }}_P=\left(W\odot K\right)\ast {\mathit{F}}_P$$

(6)

where ⊙ represents the inner product operation and ∗ represents the convolution operation.

At the conclusion of the common feature extraction module, this study focused on features from various bands and subsequently processed them through convolutional layers followed by ReLU activation functions to derive the output of the module ${\mathit{C}}_{\mathit{out}}$.

2.4. Space–Frequency Domain Feature Interaction Fusion Module

Most existing methodologies primarily focus on designing fusion strategies within the spatial domain, employing techniques such as direct addition, concatenation, or the application of adaptive weights. While these approaches have yielded satisfactory fusion results, there has been a lack of corresponding network designs that address frequency domain information. Furthermore, although certain methods, such as FAFNet [41], extract features in the frequency domain, they tend to overlook the information present in the spatial domain. To enhance the accuracy of the fusion results, this study developed SFFIM (Figure 6). The research used DFT to ${\mathit{L}}_{\mathit{out}}$ and ${\mathit{P}}_{\mathit{out}}$ individually to derive their respective amplitudes (${\mathit{A}}_{\mathit{m}}$, ${\mathit{A}}_{\mathit{p}}$) and phases (${\mathit{P}}_{\mathit{m}}$, ${\mathit{P}}_{\mathit{p}}$). Subsequently, this study focused on the amplitudes and phases from the different branches separately, thereby integrating the amplitude information $\mathit{A}$ and phase information P. This is performed to ensure that the amplitude and phase information in the frequency domain of ${\mathit{L}}_{\mathit{out}}$ and ${\mathit{P}}_{\mathit{out}}$ are fully communicated. If it is spatial domain interaction between ${\mathit{L}}_{\mathit{out}}$ and ${\mathit{P}}_{\mathit{out}}$, then only conventional strategies such as addition or concatenation can be used. Therefore, we consider the interaction between ${\mathit{L}}_{\mathit{out}}$ and ${\mathit{P}}_{\mathit{out}}$ in the frequency domain as an effective way to integrate the information from these two branches. Finally, IDFT was performed on the combined A and $\mathit{P}$ to obtain the frequency domain feature ${\mathit{F}}_{\mathit{mp}}$ (Equation (7)):

$$\begin{array}{cc}\hfill & {\mathit{A}}_{\mathit{m}}, {\mathit{P}}_{\mathit{m}}=DFT\left({\mathit{L}}_{\mathit{out}}\right)\hfill \\ \hfill & {\mathit{A}}_{\mathit{p}}, {\mathit{P}}_{\mathit{p}}=DFT\left({\mathit{P}}_{\mathit{out}}\right)\hfill \\ \hfill & \mathit{A}=Con\left(Concat[{\mathit{A}}_{\mathit{m}}, {\mathit{A}}_{\mathit{p}}]\right)\hfill \\ \hfill & \mathit{P}=Con\left(Concat[{\mathit{P}}_{\mathit{m}}, {\mathit{P}}_{\mathit{p}}]\right)\hfill \\ \hfill & {\mathit{F}}_{\mathit{mp}}=IDFT\left(\mathit{A}, \mathit{P}\right)\hfill \end{array}$$

(7)

where Con (·) represents the convolutional layer of two consecutive layers in Figure 6.

For ${\mathit{C}}_{\mathit{out}}$, two successive convolutional layers are employed to extract the spatial domain feature ${\mathit{S}}_{\mathit{mp}}$. If ${\mathit{C}}_{\mathit{out}}$ is processed in the frequency domain by simply performing DFT and IDFT transformations, there is no information exchange involved. Following this, ${\mathit{S}}_{\mathit{mp}}$ and ${\mathit{F}}_{\mathit{mp}}$ are fused by separately applying Sigmoid activation functions to each of them to derive their respective feature map weights, ${\mathit{M}}_{\mathit{S}}$ and ${\mathit{M}}_{\mathit{F}}$. Subsequently, the weights ${\mathit{M}}_{\mathit{S}}$ and ${\mathit{M}}_{\mathit{F}}$ are cross-multiplied with ${\mathit{S}}_{\mathit{mp}}$ and ${\mathit{F}}_{\mathit{mp}}$, respectively, and the results are concatenated to produce the output feature ${\mathit{F}}_{\mathit{out}}$. The detailed process is expressed in Equation (8):

$${\mathit{F}}_{\mathit{out}}=Concat[{\mathit{S}}_{\mathit{mp}}·\sigma \left({\mathit{F}}_{\mathit{mp}}\right), {\mathit{F}}_{\mathit{mp}}·\sigma \left({\mathit{S}}_{\mathit{mp}}\right)]$$

(8)

2.5. Loss Function Design

Most pansharpening networks primarily utilize ℓ₁ loss function during the training process to evaluate the discrepancy between the fusion output and the reference image. This methodology neglects the supervision of the outputs from the intermediate layers of the network [42]. The unavoidable presence of redundant information across various branches can adversely impact the quality of the fusion results. To mitigate the redundancy inherent in different modalities, this study proposed the incorporation of mutual information constraints, which aim to minimize the mutual information between different branches ($MI$(·, ·) in Equation (9) [28]. This approach is intended to reduce redundancy and enhance the feature representation capabilities of the various branches. The mutual information loss function is presented in Equation (9).

$${\mathit{L}}_{\mathit{MI}}=MI\left({\mathit{L}}_{\mathit{out}}, {\mathit{P}}_{\mathit{out}}\right)+MI\left({\mathit{L}}_{\mathit{out}}, {\mathit{C}}_{\mathit{out}}\right)+MI\left({\mathit{C}}_{\mathit{out}}, {\mathit{P}}_{\mathit{out}}\right)$$

(9)

Additionally, it was considered that the frequency domain characteristics of the output should closely match those of the ground truth (GT). To address this, a frequency domain loss function was designed to impose constraints on the frequency domain features (Equation (10)):

$${\mathit{L}}_{\mathit{F}}={∥A\left(\mathit{HRMS}\right)-A\left(\mathit{GT}\right)∥}_1+{∥P\left(\mathit{HRMS}\right)-P\left(\mathit{GT}\right)∥}_1$$

(10)

where $A\left(\mathit{HRMS}\right)$ and $P\left(\mathit{HRMS}\right)$ represent the amplitude and phase of the fusion result, respectively, while $A\left(\mathit{GT}\right)$ and $P\left(\mathit{GT}\right)$ represent the amplitude and phase of the reference image, respectively.

In summary, the developed loss function is expressed as follows:

$$\mathit{L}={∥\mathit{HRMS}-\mathit{GT}∥}_1+\alpha {\mathit{L}}_{\mathit{F}}+\beta {\mathit{L}}_{\mathit{MI}}$$

(11)

where $\alpha =\beta =0.1$, as indicated by [43,44].

3. Experimental Results

3.1. Datasets

To assess the efficacy of the suggested methodology, the research carried out validations using simulated as well as real datasets acquired from the IKONOS, WV3, and WV4 satellites. The LRMS images from IKONOS and WV4 contain four spectral bands, while the LRMS images from WV3 include eight spectral bands. The PAN images from all three satellite datasets are composed of single-band imagery. Specifically, the spatial resolutions for the LRMS and PAN images in the IKONOS dataset are 4 m and 1 m, respectively. In the case of the WV4 and WV3 datasets, the spatial resolutions for both LRMS and PAN images are 1.24 m and 0.31 m, respectively. Additional details and acquisition approaches for these three datasets are described in [45]. The assessment encompasses experiments with both simulated as well as actual data to objectively validate the approach’s effectiveness. While there exists no directly relevant simulation dataset for the simulated data assessment, training pansharpening networks relying on CNN is carried out on simulated datasets. As a result, we apply the Wald protocol, which involves applying MTF [46] filtering to the original LRMS and PAN images, followed by 4× down-sampling. By following this process, we obtain 1200, 1500, and 960 pairs of simulated images from the IKONOS, WV4, and WV3 datasets, respectively. The original LRMS image serves as the reference image. For every simulated dataset, 80% is used for training, 10% for validation, and 10% for testing. The dimensions of the employed LRMS and PAN images are 64 × 64 × 4 (64 × 64 × 8) and 256 × 256, respectively. For the real data assessment, the original LRMS as well as PAN images (with dimensions of 256 × 256 × 4 (256 × 256 × 8) and 1024 × 1024, respectively) are directly used as network inputs for testing (the original image pairs are only employed as the testing dataset), with every satellite’s testing dataset having the same amount of image pairs as the simulated data assessment.

3.2. Evaluation Indicators and Comparison Methods

On the simulated dataset, we use ERGAS (erreur relative globale adimensionnelle de synthese) [47], UIQI (universal image quality index) [48], SAM (spectral angle mapping) [49], SCC (structural correlation coefficient) [50], and Q4 (Q8) [47]. as evaluation metrics. It is worth noting that the higher the values of UIQI, SCC, and Q4 (Q8), the higher the quality, while the lower the values of ERGAS, RMSE, RASE, and SAM, the better the performance. On the real dataset, we use the quality without reference (QNR), spectral distortion index ($D_{\lambda }$), and spatial distortion index ($D_s$) to evaluate the full-resolution performance. The larger the QNR, the better the quality, while the smaller the $D_{\lambda }$ and $D_s$ [51], the smaller the distortion. This article compares our proposed method with some classic methods (GS [8], HPF [12], PanNet [52], MSDCNN [20], TFNet [21], GPPNN [53], SSAF [22], SSE [24], CIKA [30], Band aware [31], and PAPS [32]) to verify the effectiveness of our proposed method. Among them, GS and HPF are traditional methods, while the rest are DL methods.

3.3. Experimental Details

All traditional methods were obtained by running MATLAB R2023a on a computer with Intel (R) Core (TM) i5-13500H processors and 16GB of memory. All DL-based network training and testing are implemented using the PyTorch 1.13.1 framework and accelerated using a GPU with a GeForce RTX 4090 graphics card. For the proposed method, the Adam optimizer is used to train the model with 1000 training rounds and a learning rate set to $1\times {10}^{-4}$. The model decays to the original 0.1 for every 200 training rounds, Set the batch size to 4.

3.4. Experimental Results on Simulated Datasets

Figure 7 and Figure 8 depict the fusion outcomes of different approaches on the IKONOS simulated dataset as well as their residual plots with the reference image, respectively. Overall, the fusion results of the HPF approach seem quite blurry, and there is a certain degree of loss of spectral information in the fusion results of the GPPNN method. Upon analyzing the magnified region, the color of the road in the GS approach’s fusion outcome exhibits some deviation from the surrounding area, and the spectral data in the SSE approach’s fusion outcome demonstrate certain discrepancies compared with the reference image. The fusion result of the PanNet method shows that the circular texture along the highway edge is very blurry. The fusion outcomes of the SSAF and MSDCNN approaches show some fuzzy characteristics at the highway edges. Among the remaining approaches, the fusion outcomes of Band aware, CIKA, and our proposed approach bear relatively close resemblance to the reference image, and the errors of these three approaches are also comparable in Figure 8. However, the quantitative analysis in

Table 1. Quantitative evaluations using different methods on the IKONOS dataset.

	ERGAS	UIQI	SAM	SCC	Q4	QNR	${\mathit{D}}_{\mathit{\lambda }}$	${\mathit{D}}_{\mathit{s}}$
GS	2.7475	0.9328	3.8596	0.8661	0.6477	0.7926	0.0794	0.1411
HPF	2.9985	0.8793	2.9933	0.7100	0.5092	0.7942	0.0979	0.1252
PanNet	1.5637	0.9743	2.1368	0.9341	0.7941	0.8925	0.0461	0.0657
MSDCNN	2.0111	0.9565	2.6961	0.9002	0.7182	0.8997	0.0600	0.0438
TFNet	1.5739	0.9752	2.0298	0.9424	0.7829	0.8774	0.0727	0.0568
GPPNN	1.5834	0.9742	2.0745	0.9423	0.7948	0.8456	0.0841	0.0838
SSAF	2.1664	0.9393	2.7219	0.9125	0.7064	0.8141	0.0658	0.1331
SSE	1.8113	0.9656	2.4252	0.9252	0.7507	0.8645	0.0702	0.0746
CIKA	1.3815	0.9799	1.7580	0.9501	0.8315	0.9094	0.0403	0.0542
Band aware	1.3523	0.9805	1.7095	0.9521	0.8361	0.9072	0.0332	0.0650
PAPS	1.4452	0.9780	1.8512	0.9475	0.8152	0.8728	0.0700	0.0642
Ours	1.2795	0.9825	1.6102	0.9567	0.8492	0.9181	0.0377	0.0478

reveals that all indicators of our proposed approach surpassed those of other approaches. The best and suboptimal values are highlighted in bold and underlined, respectively(as noted in subsequent tables).

Figure 9 and Figure 10 illustrate the fusion results for different techniques employed on the simulated WV4 dataset, in addition to their residual plots in comparison to the reference image. The HPF approach’s fusion outcomes are severely impacted by spatial data loss, resulting in a blurry appearance of all buildings. The GS and MSDCNN approaches’ fusion outcomes depict slightly brighter roof colors on the building in the upper-right corner, which ought to have been a dark gray shade. A small portion of the building’s roof has been magnified as the area of interest and placed in the bottom-right corner for closer inspection. The enlarged area of the fusion result of the PanNet method is relatively blurry. The SSE and GPPNN approach’s fusion outcome retains only a faint orange color on the roof, while the TFNet approach’s outcome also fails to capture the corresponding roof color data accurately. The PAPS approach’s fusion outcome exhibits significant color discrepancies in all roofs when compared with the reference image. The CIKA approach’s fusion outcome features smoothed roof edges, while the other approaches’ fusion outcomes bear a close resemblance to the reference image. Figure 10 clearly demonstrates that our proposed approach incurs the least amount of data loss in comparison to other techniques. Moreover, our approach achieved the highest indicator values across different simulated dataset indicators, as shown in

Table 2. Quantitative evaluations using different methods on the WV4 dataset.

	ERGAS	UIQI	SAM	SCC	Q4	QNR	${\mathit{D}}_{\mathit{\lambda }}$	${\mathit{D}}_{\mathit{s}}$
GS	2.7630	0.9667	2.8919	0.8823	0.6508	0.8581	0.0539	0.0946
HPF	3.1100	0.9594	2.3687	0.7802	0.5569	0.8872	0.0557	0.0626
PanNet	1.9192	0.9832	1.9704	0.9052	0.7448	0.9152	0.0357	0.0454
MSDCNN	2.3100	0.9801	2.5487	0.8904	0.6761	0.8906	0.0621	0.0511
TFNet	1.8147	0.9871	1.8626	0.9250	0.7381	0.8914	0.0610	0.0514
GPPNN	2.1848	0.9816	2.3348	0.9042	0.7250	0.8819	0.0606	0.0630
SSAF	1.5764	0.9879	1.6118	0.9364	0.8025	0.9279	0.0341	0.0396
SSE	2.3583	0.9780	2.7840	0.8831	0.6932	0.8426	0.0911	0.0753
CIKA	1.9091	0.9848	1.9392	0.9067	0.7566	0.9205	0.0379	0.0434
Band aware	1.5609	0.9898	1.6113	0.9374	0.7983	0.8953	0.0434	0.0646
PAPS	2.0560	0.9836	2.1098	0.9124	0.7388	0.9002	0.0616	0.0415
Ours	1.3773	0.9924	1.4797	0.9522	0.8246	0.9336	0.0310	0.0375

.

Figure 11 and Figure 12 present the fusion results of different approaches applied to the simulated WV3 dataset, in addition to their residual plots compared with the reference image. Figure 11 shows the HPF approach’s fusion outcome appears significantly blurry overall. The industrial park’s roof color, which should have been blue, is notably distorted in the fusion outcomes of the MSDCNN, GPPNN, TFNet, and SSE approaches, deviating considerably from the reference image. The GS approach’s fusion outcomes also exhibit some degree of spectral distortion in the roof color. Upon analyzing the magnified area, it is evident that the PanNet and SSAF’s fusion outcome have a slightly lighter color, while the other approaches’ fusion outcomes closely match the reference image within the enlarged region. Figure 12 clearly demonstrates that our proposed approach produces the smallest residual compared with other techniques. The quantitative analysis shown in

Table 3. Quantitative evaluations using different methods on the WV3 dataset.

	ERGAS	UIQI	SAM	SCC	Q8	QNR	${\mathit{D}}_{\mathit{\lambda }}$	${\mathit{D}}_{\mathit{s}}$
GS	6.0708	0.9088	6.9071	0.8641	0.6323	0.8277	0.0722	0.1231
HPF	8.0068	0.8368	6.8444	0.5927	0.4292	0.8134	0.0958	0.1176
PanNet	5.0879	0.9512	6.2120	0.8696	0.7055	0.8499	0.0668	0.1023
MSDCNN	4.7723	0.9560	5.5764	0.8929	0.7297	0.8276	0.0978	0.0968
TFNet	4.7073	0.9558	5.4749	0.8998	0.7089	0.8275	0.0884	0.1030
GPPNN	4.8313	0.9538	5.7267	0.8909	0.6999	0.7908	0.1092	0.1346
SSAF	3.0722	0.9814	4.1068	0.9598	0.7949	0.8383	0.0862	0.0988
SSE	4.9640	0.9522	5.9466	0.8893	0.7124	0.8127	0.0993	0.1135
CIKA	3.4862	0.9759	4.6022	0.9423	0.7862	0.8591	0.0744	0.0831
Band aware	3.0315	0.9826	4.1554	0.9612	0.8000	0.7761	0.1005	0.1430
PAPS	2.9046	0.9835	3.9959	0.9652	0.8028	0.8731	0.0588	0.0834
Ours	2.6121	0.9865	3.6363	0.9730	0.8130	0.8886	0.0539	0.0657

further confirms the outstanding performance of our introduced approach.

3.5. Experimental Results on Real Datasets

Figure 13 showcases the outcomes of different fusion techniques employed on the real IKONOS dataset. Due to the absence of reference images, spatial data are sourced from PAN images, while spectral data are derived from MS images. To facilitate the examination of small objects in every sub-figure, a heart-shaped building complex has been magnified as the region of interest and positioned in the top–left corner. The fusion outcomes generated by the GPPNN, SSAF, SSE, and MSDCNN approaches display a yellowish tint on the building rooftops, whereas the TFNet approach’s outcome exhibits an orange hue, markedly diverging from the red color observed in the MS images. The PanNet and PAPS approaches yield lighter color in their fusion outcome, and the HPF approach’s outcomes show some blurriness. The other approaches have largely succeeded in preserving both spectral and spatial data. Our proposed approach demonstrates competitiveness, as evidenced by the quantitative assessment shown in Table 1.

Figure 14 shows the fusion results of various methods on the real dataset of WV4. The vegetation color in the lower right corner of the river in the fusion result of the MSDCNN and GPPNN methods is slightly black, and the color of the corresponding area in the fusion result of the SSE method is also significantly different from that in the MS image. We choose a section of river and its surrounding banks as the area of interest, enlarge it, and place it in the upper left corner. The river color in the fusion result of the GS method is significantly lighter, while the river color in the fusion result of the SSAF method lacks brightness. The fusion result of the TFNet method shows that the soil color on the riverbank deviates from the soil color at the corresponding position in the MS image. The fusion results of Band aware and CIKA methods show a fuzzy phenomenon within the river. The fusion results of other methods almost retain the original spectral information and spatial phenomena, but the river lines in the fusion results of our proposed method are more natural. Table 2 also demonstrates that the proposed method outperforms other methods.

Figure 15 presents the results of different fusion approaches applied to the real WV3 dataset. Akin to Figure 13 and Figure 14, the real dataset does not include reference images. The PAN images capture spatial details, while the MS images contain spectral data. Upon closer inspection, it becomes apparent that the SSAF fusion output exhibits an overall green tint in the ocean’s hue, deviating from the true color. Zooming in on specific regions reveals further disparities. The bevery color in the MSDCNN and SSE fusion outcomes fails to align with the spectral data accordingly within the MS image. Strikingly, the GPPNN and SSE approach’s output completely omits the color data of the red vehicles, while the TFNet approach’s fusion outcome preserves only a faint trace. Similarly, the GS fusion output displays inconsistencies in the bevery color’s spectral data when compared with the MS image. Although other approaches successfully retain both spectral and spatial details to varying degrees, our proposed approach stands out by generating a more vivid and visually appealing fused image. The superiority of the introduced approach is further corroborated by the overall advantages presented in Table 3.

3.6. Analysis of Ablation Experiment Results

In the ablation experiment, we will conduct network module ablation experiments and loss function partial ablation experiments separately and analyze the impact of each part on the overall performance of the model through experimental results.

We conducted ablation experiments on the IKONOS simulation dataset to analyze the contributions of each module in the proposed method. In this section, comparative experiments were conducted on removing different modules and their different combinations, with ‘w/o’ indicating that this module is not used. Figure 16 shows the fusion results after removing some modules, as well as the error maps between each fusion result and the reference image. There is not much difference between the several images, both overall and in the enlarged area. However,

Table 4. Quantitative evaluations of module ablation experiments on the simulated IKONOS dataset.

	ERGAS	UIQI	SAM	SCC	Q4
w/o Spectral feature extraction module	1.3125	0.9816	1.6760	0.9541	0.8430
w/o Spatial feature extraction module	1.3238	0.9813	1.6808	0.9535	0.8419
w/o Common feature extraction module	1.3076	0.9817	1.6599	0.9546	0.8445
w/o SFFIM	1.3293	0.9812	1.6944	0.9532	0.8397
ours	1.2795	0.9825	1.6102	0.9567	0.8492

shows the various indicators after removing each module. It can be seen that the network after removing the spatial frequency interaction module has the worst performance in all indicators, indicating that this module has the greatest impact on our proposed network model.

In addition to verifying the impact of network modules on the overall network, we also verified the influence of loss functions in each part on the fusion results. Similarly, this part was experimented on the IKONOS simulation dataset. ‘W/o’ means not using this loss function part. Figure 17 shows the mutual information loss function, spectral loss function, and fusion results after removing both from formula 10, as well as the error plots between each fusion result and the reference image. The quantitative evaluation of the fusion results is shown in

Table 5. Quantitative evaluations of oss Abalation experiments on the simulated IKONOS dataset.

	ERGAS	UIQI	SAM	SCC	Q4
w/o $L_{MI}$	1.3744	0.9800	1.7581	0.9505	0.8331
w/o $L_F$	1.3614	0.9801	1.7309	0.9511	0.8350
only ℓ1	1.4064	0.9794	1.8261	0.9492	0.8282
ours	1.2795	0.9825	1.6102	0.9567	0.8492

. In Figure 17, overall, the fusion results of each version are not significantly different. From the enlarged area, it can be seen that there is a certain deviation between the green color of the fusion result of removing two loss functions and the reference image, and the soil part is almost not restored. The soil spectral information of the fusion result after removing $L_{MI}$ also has a certain degree of loss, and the soil edge of the fusion result after removing $L_F$ is not as clear as using the complete loss function. The various indicators in Table 5 further demonstrate the effectiveness of our designed loss function.

3.7. Analysis of Experimental Results on Network Structure

In the network architecture experiment, we compared the performance differences between different network architectures and the fusion results of our proposed network architecture, which were tested on the IKONOS simulation dataset. Among them, single branch refers to the branch that retains the input of LRMS and PAN for band-by-band splicing, as well as the corresponding part of this branch in the fusion module after retention. Dual branch refers to the branch that retains the input of LRMS and the input of PAN, as well as the corresponding part of this branch in the fusion module after retention. Direct splicing refers to not performing band-by-band splicing, and the rest is consistent with our proposed network model. Figure 18 shows the fusion results using different network structures and their errors with the reference image, while

Table 6. Quantitative evaluations of loss ablation experiments on the simulated IKONOS dataset.

	ERGAS	UIQI	SAM	SCC	Q4
Single branch	1.3618	0.9804	1.7364	0.9513	0.8346
Dual branch	1.3360	0.9810	1.7038	0.9527	0.8389
Direct Concatenation	1.2918	0.9822	1.6268	0.9558	0.8465
ours	1.2795	0.9825	1.6102	0.9567	0.8492

presents the corresponding indicators. From the enlarged area, it can be seen that the colors of the two pools in the fusion result using a single-branch structure are significantly different from the reference image, and there are artifacts at the edge of the left pool. This is due to the differences in spatial and spectral information of the source image, which were not taken into account in the single-branch structure. The edge of the water pool in the fusion result using a dual branch structure is slightly smoothed, and the network structure using direct stitching is similar to our proposed model (possibly due to fully preserving the source image information), but slightly inferior in image clarity. The various indicators in Table 6 further validate the superiority of our designed network structure.

3.8. Complexity Analyses

To further compare the network performance, we also compared the number of parameters, running time, and floating point of operations (FLOPs) of our proposed method with other deep learning-based methods on the test set (all compared on the WV3 dataset), as shown in

Table 7. Model sizes and running times of DL-based pansharpening methods.

	Param (M)	Time (s)	Flops (G)
PanNet	0.08	1.0579	5.4211
MSDCNN	0.18	3.0727	14.9595
TFNet	2.36	3.5929	30.9760
GPPNN	0.24	3.5016	11.1065
SSAF	1.08	5.2443	70.7027
SSE	0.47	4.0478	30.7840
CIKA	0.07	3.7104	4.7322
Band aware	0.59	4.5207	54.4818
PAPS	0.48	6.0989	32.9523
Ours	0.29	3.9182	36.4856

. Although the number of parameters in our proposed model is not the minimum, it remains within 0.5 M. In terms of runtime, the MSDCNN model has the fastest running speed, but the performance of our proposed model is far superior to MSDCNN. The slow running speed and large FLOPs value of our proposed model may be due to the presence of many matrix multiplication operations in the network.

4. Conclusions

This study introduced a novel three-branch pansharpening network that leverages interactions between the spatial and frequency domains. Each branch is designed to extract specific types of information: spectral features from LRMS images, spatial features from PAN images, and common features shared by both. These extracted features are exchanged across spatial and frequency domains before undergoing feature reconstruction. The proposed design includes a spectral feature extraction module capable of capturing nonlinear relationships between the spectral bands in LRMS images and a spatial feature extraction module that effectively captures texture details and embeds positional information, thereby enhancing the completeness of the spatial features. Additionally, the research proposed a dynamic convolution mechanism that adapts to spectral and spatial features, improving network flexibility. The proposed SSFM facilitates robust interaction between spatial and frequency domain information. Furthermore, this study developed a loss function aimed at reducing redundant information between branches while ensuring the frequency domain features closely align with the reference image. Comprehensive experiments, including comparative, ablation, and network structure evaluations across three datasets (IKONOS, WV3, and WV4), consistently demonstrate the superior performance of the proposed method.

It is important to acknowledge that, like many CNN-based approaches, the proposed method was trained on simulated datasets and evaluated on both simulated and real datasets. However, there is still room for improvement in terms of spatial details in our proposed method, and we will attempt to use more cutting-edge edge operator regions to extract detailed textures from the source image.In addition, its performance on real datasets with scale variations remains modest. Future work will focus on jointly training with both simulated and real datasets to improve performance and achieve a balanced effectiveness across diverse datasets.