ConvMambaSR: Leveraging State-Space Models and CNNs in a Dual-Branch Architecture for Remote Sensing Imagery Super-Resolution

Abstract

Deep learning-based super-resolution (SR) techniques play a crucial role in enhancing the spatial resolution of images. However, remote sensing images present substantial challenges due to their diverse features, complex structures, and significant size variations in ground objects. Moreover, recovering lost details from low-resolution remote sensing images with complex and unknown degradations, such as downsampling, noise, and compression, remains a critical issue. To address these challenges, we propose ConvMambaSR, a novel super-resolution framework that integrates state-space models (SSMs) and Convolutional Neural Networks (CNNs). This framework is specifically designed to handle heterogeneous and complex ground features, as well as unknown degradations in remote sensing imagery. ConvMambaSR leverages SSMs to model global dependencies, activating more pixels in the super-resolution task. Concurrently, it employs CNNs to extract local detail features, enhancing the model’s ability to capture image textures and edges. Furthermore, we have developed a global–detail reconstruction module (GDRM) to integrate diverse levels of global and local information efficiently. We rigorously validated the proposed method on two distinct datasets, RSSCN7 and RSSRD-KQ, and benchmarked its performance against state-of-the-art SR models. Experiments show that our method achieves SOTA PSNR values of 26.06 and 24.29 on these datasets, respectively, and is visually superior, effectively addressing a variety of scenarios and significantly outperforming existing methods.

1. Introduction

High-resolution (HR) remote sensing imagery is of great importance in urban planning and management [1], agricultural resource optimization [2], biodiversity conservation [3], climate change research [4], environmental monitoring [5], and other fields. For instance, in environmental monitoring, HR imagery can identify changes in land cover [6], vegetation health [7], and water resources [8], which can assist in evaluating ecological trends and assessing the effectiveness of environmental protection efforts.

However, the acquisition of high-resolution remote sensing images is often constrained by technical and financial limitations. The image super-resolution technique employs an advanced algorithm to reconstruct the relatively low-resolution (LR) image from the HR image, thereby offering a cost-effective and efficient means of acquiring HR images.

In recent years, single-image super-resolution (SISR) techniques have emerged as a significant area of research [9,10]. The core objective of this technique is to recover an HR image from a single LR image. Offering advantages such as cost-effectiveness and efficiency, SISR techniques have attracted extensive academic attention [11], especially deep learning-based methods [12], which have made significant progress in reconstructing images with detail and clarity. Deep learning-based SISR methods rely on training a large number of LR/HR image pairs to build mapping models. In addition, the performance of super-resolution networks can be significantly improved by introducing techniques such as residual connections [13], dense connections [14], generative adversarial networks [15], and attention mechanisms [16]. Despite the success of SISR techniques in many fields, their application in remote sensing images still faces many challenges. Ground objects in remote sensing images exhibit complex structural characteristics, significant size differences, and a substantial amount of noise, in addition to highly localized features [17]. These characteristics present significant challenges for the application of SISR to remote sensing.

Despite the advancements in SISR methods, several fundamental challenges remain unresolved, particularly in capturing long-range dependencies and handling complex and unknown degradations in remote sensing images. These challenges highlight the need for a more sophisticated approach that can overcome the limitations of both CNN-based and Transformer-based models.

Typical CNN-based SR methods include SRCNN [18], RRDBNet [14], and EDSR [19]. SRCNN is the first CNN-based image SR method that learns an end-to-end nonlinear mapping from LR to HR images through a three-layer convolutional network. RRDBNet combines the advantages of residue-in-residue and dense block residue-in-residue structures and is widely used in high-magnification super-resolution and generative adversarial training. EDSR improves super-resolution performance by removing the batch normalization layer and increasing the depth and width of the network. Although these CNN-based SR methods have significantly improved in performance, they are constrained by the receptive field due to the limitations of convolutional operations, which impede their ability to capture global contextual information [20]. Furthermore, while the superior efficiency of convolutional parallel operations makes them well-suited for deployment on resource-constrained devices, these methods face significant challenges in processing remotely sensed images with complex structures and diverse features. These factors present significant challenges to the super-resolution reconstruction of remote sensing images.

Transformer-based deep learning models have achieved state-of-the-art performance in a variety of computer vision applications [21,22,23,24], which have demonstrated an efficient ability to capture global background information by utilizing the self-attention mechanism [25], Transformer-based SR methods have also evolved significantly. Despite its global receptive field, Transformer [26] exhibits quadratic complexity in processing input sequences, which presents a challenge when dealing with common large-size image restoration tasks. SwinIR [20] represents a state-of-the-art approach to SISR tasks based on Swin Transformer [27]. In comparison with purely convolutional structures and ViT-based architectures, SwinIR is more efficient in public datasets such as DIV2K. Nevertheless, certain studies have demonstrated that the performance of a single Transformer may not be superior to that of a CNN due to the compression of image blocks into a 1D sequence, which may result in the loss of structural information [28]. Conversely, the incorporation of efficient attention techniques, such as the window-shift attention mechanism [27], often entails the compromise of a globally effective receptive field, indicating that there is a trade-off between the global receptive field and efficient computation.

The challenge in modeling long-range dependencies stems from the inherent limitations of convolutional operations, which are restricted by their local receptive fields. Transformer models, while capable of capturing global context, struggle with computational complexity and may lose structural information during the process of sequence transformation. These issues necessitate a re-evaluation of existing methodologies and drive the development of novel approaches.

Recently, state-space models derived from control systems have attracted attention for their linear complexity in dealing with input sequences. In particular, the enhanced version of Mamba has become an efficient and effective backbone for developing complex networks [29,30,31,32,33]. The discrete state-space equations in Mamba can be formalized into recursive form and, when equipped with a specifically designed structured parameterization [34], very long dependencies can be modeled. However, the standard Mamba [30] designed for 1D sequential data in NLP processes 1D image sequences recursively, which may result in spatially close pixels being very far away in the spread sequence. This can lead to localized pixel forgetting problems that are not suitable for image super-resolution scenarios.

To address the aforementioned challenges, this paper proposes a novel remote sensing image super-resolution framework (ConvMambaSR) for hybrid models, offering a new perspective on efficiently modeling long-range dependencies while maintaining the integrity of local details in remote sensing images. The framework is comprised of three principal steps: shallow feature extraction, deep feature extraction, and upsampling. In the deep feature extraction stage, ConvMambaSR employs an elaborate residual state-space group (RSSG) and residual convolution group (RCG) to capture high-dimensional features at different levels. Subsequently, GDRM is employed to facilitate the efficient integration of local details and global contextual information.

The principal contributions of this paper are summarized as follows:

• ConvMambaSR is proposed as a hybrid model combining CNN and Mamba. It employs a dual-branch architecture: the CNN branch extracts local features and processes spatial information, while the Mamba branch captures global features and long-range dependencies.
• A global–detail reconstruction module is introduced within ConvMambaSR, designed to integrate local details from the CNN with global contextual information from the Mamba. This module enhances the synergy between the branches by merging local features with global information, thereby improving model performance across various tasks.

2. Related Works

2.1. Advances in SISR and Applications to Remote Sensing

Recent advancements in SISR have been driven by both Convolutional Neural Networks and Vision Transformers.

CNN-based methods have achieved significant milestones, starting with the SRCNN model proposed by Dong et al. [18], which pioneered deep learning in SR. Shi et al. [35] introduced subpixel convolution, while Ledig et al. [36] incorporated ResNet and GANs into SISR, resulting in models like SRGAN [36] and ESRGAN [37]. Attention mechanisms were introduced by Zhang et al. [16], leading to advanced models such as RCAN [16] and HAN [38]. Despite their success, these CNN-based models often struggle with modeling long-range dependencies [20,39].

Vision Transformers, introduced by Dosovitskiy et al. [40], reshaped visual tasks by treating images as sequences of patches. The Swin Transformer [27] reduced computational complexity through window-based self-attention and became the backbone for many visual tasks, including SISR. Liang et al. [20] developed SwinIR, and subsequent improvements like HAT [41] and NGswin [42] achieved competitive reconstruction performance in capturing long-range dependencies and cross-window connections.

In the realm of remote sensing, SISR has gained traction due to the challenges of acquiring multiple images for Multi-Image SR (MISR) techniques [43]. Building on natural image SR techniques, models like SRCNN were adapted to remote sensing by Ducournau and Fablet [44], while other advancements include RDBPN [45], PMSRN [46], and hybrid models like SWCGAN [47], HSTNet [48], and ConvFormerSR [49], enhancing spatial resolution and spectral consistency in remote sensing images.

2.2. State-Space Models in Deep Learning

State-space models have recently gained prominence in deep learning, particularly for their ability to address long-range dependency challenges by drawing on continuous state-space modeling from control systems [32,50,51]. A notable example is the Structured State-Space Sequence model (S4) [50], which uses parameter normalization with diagonal structures as an alternative to CNNs and Transformers for modeling long-distance dependencies. Building on this, the S5 model [32] introduces multiple-input multiple-output (MIMO) SSM and efficient parallel scanning, while the gated state-space layer [31] enhances the expressive power of S4 by incorporating gating mechanisms.

Recently, Gu et al. [30] introduced a data-dependent SSM layer and the Mamba language model backbone, which outperforms Transformers on large-scale real-world data with linear scalability for sequence lengths. Mamba’s computational efficiency in image processing highlights the potential of SSMs in image restoration, offering novel insights and advantages over traditional deep learning models.

3. Methodology

In this section, we first introduce the overall structure of ConvMambaSR and then describe its three important modules, namely RSSG, RCG, and GDRM.

3.1. Overall Structure of ConvMambaSR

As illustrated in Figure 1, ConvMambaSR is comprised of three stages: shallow feature extraction, deep feature extraction, and high-quality reconstruction. In the shallow feature extraction stage, ConvMambaSR first extracts shallowfeatures ${\mathbf{F}}_S\in {\mathbb{R}}^{H\times W\times C}$ from the low-resolution input image ${\mathbf{I}}_{LQ}\in {\mathbb{R}}^{H\times W\times 3}$ through a 3 × 3 convolutional layer, where H and W denote the height and width of the input image, respectively, and C denotes the number of channels.

In the deep feature extraction stage, the deep features ${\mathbf{F}}_{D\_m}$ and ${\mathbf{F}}_{D\_c}$ are obtained by parallel Mamba branching and CNN branching, respectively. These features are then fused by the GDRM module.

In the high-quality reconstruction stage, global residual concatenation is employed to integrate the low-level features with the deep-level features, thereby generating the input ${\mathbf{F}}_R$ for the high-quality reconstruction stage. Finally, the pixel rearrangement method [35] is utilized for upsampling, resulting in the SR result ${\mathbf{I}}^{SR}$.

The SR process of ConvMambaSR can be expressed mathematically as

$${\mathbf{F}}_R=G({\mathbf{F}}_{D\_m},{\mathbf{F}}_{D\_c})+{\mathbf{F}}_S$$

(1)

$${\mathbf{I}}^{\mathrm{SR}}=\mathrm{Up}\left({\mathbf{F}}_R\right)$$

(2)

where $\mathrm{Up}(·)$ denotes the upsampling function and $\mathrm{G}(·)$ denotes the global–detail reconstruction operation. ${\mathbf{I}}^{SR}$ is the reconstructed super-resolution image.

3.2. Residual State-Space Group

3.2.1. Vision State-Space Module

Transformer-based super-resolution networks typically partition the input into small patches [52] or employ shift-window attention [20] to ensure efficiency. However, this approach hinders interaction at the whole image level. In response, we introduce the Visual State-Space Module (VSSM) [53] to the image super-resolution task, thereby enabling the model to benefit from Mamba’s success in long-range modeling of linear complexity.

VSSM is capable of capturing long-range dependencies through the use of state-space equations. As illustrated in Figure 2, which follows [53], the input feature ${\mathbf{F}}_{in}\in {\mathbb{R}}^{H\times W\times C}$ will undergo processing through two parallel branches. In the initial branch, the feature channel is expanded to $\beta C$ through a linear layer, where $\beta$ is a predefined channel extension factor. This is followed by a depth-wise convolution, a SiLU [54] activation function, and a 2D Selective Scanning Module (2D-SSM) layer [55] and LayerNorm (LN). In the subsequent branch, the feature channel is also extended to $\beta C$ through a linear layer, followed by the SiLU activation function. Subsequently, the features from both branches are aggregated using the Hadamard product. Finally, the number of channels is projected back to C, generating an output of the same shape as the input $X_{out}$:

$${\mathbf{F}}_1=\mathrm{LN}\left(2\mathrm{DSSM}\left(\sigma \left(\mathrm{DWConv}\left(\mathrm{Linear}\left({\mathbf{F}}_{in}\right)\right)\right)\right)\right)$$

(3)

$${\mathbf{F}}_2=\sigma \left(\mathrm{Linear}\left({\mathbf{F}}_{in}\right)\right)$$

(4)

$${\mathbf{F}}_{out}=\mathrm{Linear}({\mathbf{F}}_1\odot {\mathbf{F}}_2)$$

(5)

where DWConv denotes the depth-wise convolution, $\sigma$ denotes the SiLU activation function, and ⊙ denotes the Hadamard product.

3.2.2. Two-Dimensional Selective Scan Module

Standard Mamba [30] captures information in scanned data during causal processing, making it well-suited for sequential NLP tasks but less effective for noncausal data-like images. We introduce the 2D Selective Scanning Module, as shown in Figure 3. Two-dimensional image features are converted into 1D sequences and scanned in four directions, and long-range dependencies are captured using the discrete state-space equation. Finally, the sequences are merged and reshaped to restore the 2D structure.

3.2.3. Residual State-Space Block

Since SSM deals with flattened feature maps as one-dimensional token sequences, it has been demonstrated that the number of neighboring pixels in the sequence can be significantly influenced by the flattening strategy [55]. When employing the four-direction unfolding strategy proposed by [53], only four nearest neighbors are accessible for anchor pixels. This discrepancy between spatial proximity in a two-dimensional feature map and temporal proximity in a one-dimensional token sequence can result in local pixel forgetting. Furthermore, SSMs typically incorporate a substantial number of hidden states to accommodate long-range dependencies, and there is a notable degree of channel redundancy in the activation results of different channels [55]. To address these issues, we propose the incorporation of a Channel Attention Block (CAB) within the residual state-space block (RSSB) framework. In this manner, SSMs can assist in alleviating the local pixel forgetting issue by extracting local features through convolution. Furthermore, it can be configured to prioritize the learning of diverse channel representations while avoiding channel redundancy by selecting key channels through subsequent Channel Attention.

As illustrated in Figure 4, given a feature map ${\mathbf{F}}_A\in {\mathbb{R}}^{C\times H\times W}$, CAB is first compressed by a compression factor ${\gamma }_1$ to obtain features with shape ${\mathbb{R}}^{{\displaystyle \frac{c}{{\gamma }_1}}\times H\times W}$. Thereafter, a channel expansion is performed to recover the original shape. By first compressing and then expanding the channel dimensions, the model can efficiently learn the different channels’ nonlinear interactions while constraining the model complexity. Subsequently, Channel Attention (CA) [56] is introduced. CAB is calculated as follows:

$${\mathbf{F}}_B=W_1\sigma \left(W_0\left({\mathbf{F}}_A\right)\right)$$

(6)

$${\mathbf{F}}_C={\mathbf{F}}_B\otimes \mathcal{S}\left(W_3\sigma \left(W_2\left({\mathbf{F}}_{\mathrm{avg}}^B\right)\right)\right)$$

(7)

where $\sigma$ denotes the StarReLU [57] activation function, expressed as

$$\mathrm{StarReLU}\left(x\right)=s·{\left(\mathrm{ReLU}\left(x\right)\right)}^2+b$$

(8)

where $s\in \mathbb{R}$ and $b\in \mathbb{R}$ are scalars of scale and bias, respectively. $W_0\in {\mathbb{R}}^{{\displaystyle \frac{c}{\gamma 1}}\times C}$, $W_1\in {\mathbb{R}}^{C\times {\displaystyle \frac{C}{\gamma 1}}}$, $W_2\in {\mathbb{R}}^{{\displaystyle \frac{c}{\gamma 2}}\times C}$, $W_3\in {\mathbb{R}}^{C\times {\displaystyle \frac{C}{\gamma 2}}}$, ${\mathbf{F}}_{\mathrm{avg}}^B$ denotes global average pooling over the features ${\mathbf{F}}_B$, $\mathcal{S}(·)$ denotes the sigmoid function, ⊗ denotes matrix multiplication, and $\gamma 1$, $\gamma 2$ denote two different compression factors.

As illustrated in Figure 5, given the input deep features ${\mathbf{F}}_{D\_m}^{l-1}\in {\mathbb{R}}^{H\times W\times C}$, we first use LayerNorm, followed by the Visual State-Space Module [53] to capture spatial long-range dependencies, using a learnable scaling factor $s\in {\mathbb{R}}^{\mathcal{C}}$ to control the jumps from the information obtained from the connections. At this point, the RSSB can be expressed as

$${\mathbf{F}}_{D\_m}^l=\mathrm{VSSM}\left(\mathrm{LN}\left({\mathbf{F}}_{D\_m}^{l-1}\right)\right)+s·{\mathbf{F}}_{D\_m}^{l-1}$$

(9)

$${\mathbf{F}}_{D\_m}^l=\mathrm{CAB}\left(\mathrm{LN}\left({\mathbf{F}}_{D\_m}^l\right)\right)+s^{\prime }·{\mathbf{F}}_{D\_m}^l$$

(10)

where s and $s^{\prime }$ denote different learnable scaling factors.

Ultimately, the RSSG can be expressed as

$${\mathrm{RSSG}}^k={\mathrm{RSSG}}^{k-1}+W{\mathbf{F}}_{D\_m}^l\left({\mathbf{F}}_{D\_m}^{l-1}(\dots {\mathbf{F}}_{D\_m}^1\left({\mathrm{RSSG}}^{k-1}\right)\dots )\right)$$

(11)

where ${\mathrm{RSSG}}^k$ and ${\mathrm{RSSG}}^{k-1}$ denote the kth RSSG and $k-1$th RSSG, respectively. ${\mathbf{F}}_{D\_m}^l$ is the lth RSSB. W is a convolutional layer that serves to enhance the translational equivariance of the Mamba layer.

3.3. Residual Convolution Group

It has been demonstrated that deep convolutional networks can enhance super-resolution performance [16,19]. Accordingly, as illustrated in Figure 6, we constructed RCGs with successive residual units to capture the details of LR images. A given RCB can be expressed as

$${\mathbf{F}}_{D\_c}^{l+1}={\mathbf{F}}_{D\_c}^l+W_1\sigma \left(W_0\left({\mathbf{F}}_{D\_c}^l\right)\right)$$

(12)

where $\sigma$ denotes the leaky ReLU activation function.

The N RCBs are connected to form an RCG module expressed as

$${\mathrm{RCG}}^k={\mathbf{F}}_{D\_c}^{N-1}+W_1\sigma \left(W_0\left({\mathbf{F}}_{D\_c}^{N-1}\right)\right),N\ge 1$$

(13)

$${\mathbf{F}}_{D\_c}^0={\mathrm{RCG}}^{k-1}$$

(14)

where ${\mathrm{RCG}}^k$ is the kth RCG feature map.

3.4. Global–Detail Reconstruction Module

CNN is capable of fully exploiting spatial and channel information in the receptive field, yet it is deficient in explicitly modeling inter-channel relationships [58]. In contrast, Mamba is capable of fully utilizing long-range and global information, although it is challenging to capture local details due to the absence of spatial generalization bias. Consequently, we devised GDRM intending to fuse the deep features of varying dimensions extracted by CNN and Mamba. The structure of GDRM and MSSA is depicted in Figure 7.

GDRM can be expressed as

$${\mathbf{F}}_{\mathrm{GDRM}}=W_1\left(\mathrm{SimAM}\left({\mathbf{F}}_{D\_m}\right)\right)+W_2\left(\mathrm{MSSA}\left({\mathbf{F}}_{D\_c}\right)\right)$$

(15)

where $W_1$ and $W_2$ denote the convolutional layers used to automatically learn the weights of different features extracted by the CNN and Mamba.

MSSA and SimAM denote multiscale spatial attention and 3D attention based on energy functions, derived from SegNeXt [59] and SimAM [60].

We employed depth-wise strip convolutions to approximate standard depth-wise convolutions with large kernels. This approach is advantageous because strip convolution is lightweight. To simulate a standard 2D convolution with a 7 × 7 kernel size, we only require a pair of 7 × 1 and 1 × 7 convolutions. In contrast, remote sensing scenes frequently exhibit strip-like features, such as rivers and farmlands. Consequently, strip convolution can be employed as a supplement to grid convolutions [61,62,63], with the MSSA facilitating the extraction of strip-like features by the CNN branch.

The MSSA formula is expressed as follows:

$${\mathbf{F}}_{D\_MSSA}={\mathbf{F}}_{D\_c}\otimes {\mathrm{Conv}}_{1\times 1}(\sum _{i=0}^3{\mathrm{Scale}}_i\left(\mathrm{DWConv}\left({\mathbf{F}}_{D\_c}\right)\right))$$

(16)

where ${\mathbf{F}}_{D\_c}$ and ${\mathbf{F}}_{D\_c}$ denote the CNN branch input features and Mamba branch input features received by the GDRM in the deep feature extraction stage. ${\mathbf{F}}_{D\_MSSA}$ denotes the output of the MSSA. DWConv denotes depth-wise convolution, and ${\mathrm{Scale}}_i$, denotes the ith branch.

The SimAM formula is expressed as follows:

$$\mu ={\displaystyle \frac{1}{M}}\sum _{i=1}^M{\mathbf{F}}_i$$

(17)

$$\sigma ={\displaystyle \frac{1}{M}}\sum _{i=1}^M{({\mathbf{F}}_i-\mu )}^2$$

(18)

$$E_{\mathrm{inv}}={\displaystyle \frac{{(\mathbf{F}-\mu )}^2}{4(\sigma +\lambda )}}+0.5$$

(19)

$${\mathbf{F}}_e=\mathcal{S}\left(E_{\mathrm{inv}}\right)·\mathbf{F}$$

(20)

where $\mu$ denotes the mean value of all neurons in the channel, $x_i$ denotes the value of the ith neuron, M denotes the total number of neurons in the channel (i.e., H × W, which denotes the spatial dimensions of the channel), $\sigma$ denotes the variance of all the neurons in the channel, $\mathcal{S}(·)$ denotes the sigmoid function, $\mathbf{F}$ denotes the input feature map, ${\mathbf{F}}_e$ denotes the output feature map, and $\lambda$ is a set of hyperparameters.

3.5. Loss Function

We optimized ConvMambaSR for image super-resolution using $\mathrm{L}1$ loss, which is the most common loss function in super-resolution tasks. Given a training set ${\left\{{\mathbf{I}}_{LR}^i,{\mathbf{I}}_{HR}^i\right\}}_{i=1}^N$, the loss can be expressed as

$${\displaystyle \frac{1}{N}}\sum _{i=1}^N{∥ConvMambaSR\left({\mathbf{I}}_{LR}^i\right)-{\mathbf{I}}_{HR}^i∥}_1$$

(21)

4. Experiments

4.1. Datasets

In order to validate the effectiveness of our model, we conducted experiments on two datasets, the RSSCN7 public remote sensing dataset [64] and our own RSSRD-KQ remote sensing dataset.

The RSSCN7 dataset contains 2800 images of remotely sensed scenes from seven typical categories, namely grasslands, farmlands, industrial areas, rivers and lakes, forests, residential areas, and parking lots. Each category has 400 images from Google Earth, sampled at four scales of 100 images each. Each image is 400 × 400 pixels in size. The variety of scene images taken in different seasons and weather conditions and sampled at different scales made this dataset quite challenging. The LR images of RSSCN7 were obtained by bicubic interpolation. This dataset is divided into two equal parts, one is used as a training set with 1400 images, and the other is used as a test set, where 20% of the training set is used as the validation set.

The RSSRD-KQ dataset is situated within the Sanjiangyuan region of Qinghai Province, China. The HR images were captured by a DJI Phantom 4 RTK drone, equipped with a visible-light sensor, which served as the data collection platform for conducting aerial photography over two days, from 21 to 22 April 2024. To minimize the impact of solar radiation and atmospheric conditions on the imagery, the flights were conducted during the midday hours of 12:00 to 14:00 under clear sky conditions. The drone was flown at an altitude of 30 m, with a spatial resolution of 0.82 cm and a flight speed of 2.3 m per s. The total area of the test site that was imaged was 100 square meters. Its geographical coordinates are 33°44′36″N to 33°44′42″N and from 98°59′30″E to 98°59′38″E. The RSSRD-KQ LR images were obtained by unknown degradation of the blind super-resolution task [65,66], which performs a series of randomly sequenced degradation operations, including motion blurring, Gaussian blurring, random downsampling (nearest-neighbor interpolation, bilinear interpolation, bicubic interpolation), simple scaling of the image, addition of color, grayscale, or mixed Gaussian noise, addition of probabilistic JPEG compression noise, and finally, addition of final JPEG compression noise processing before random cropping. In this dataset, 90% of the 3162 images were randomly selected as the training set images, 10% as the validation set images, and another 672 images outside the region were used as the test set, making a total of 3834 images, each of which has a size of 480 × 480 pixels. We used such HR/LR paired images for further analysis and model training. Figure 8 shows sample examples of these two datasets.

4.2. Experiment Settings

In super-resolution tasks, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) [67] are commonly evaluated metrics. Assuming that the high-resolution image is x and the reconstructed super-resolution image is y, they are calculated as follows:

$$\mathrm{MSE}(x,y)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(x_i-y_i)}^2$$

(22)

$$\mathrm{PSNR}(x,y)=10{\log }_{10}{\displaystyle \frac{1}{\mathrm{MSE}(x,y)}}$$

(23)

$$\mathrm{SSIM}(x,y)={\displaystyle \frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}}$$

(24)

where ${\mu }_x$ and ${\sigma }_x$ denote the mean and variance of x, respectively, ${\sigma }_{xy}$ denotes the covariance between x and y, and $c_1$ and $c_2$ are constants.

In addition, root-mean-square error (RMSE), spectral angle mapper (SAM) [68], and relative dimensionless global error in synthesis (ERGAS) [69] are also used as the metrics that are widely used to evaluate the quality of reconstructed remote sensing images. They are calculated as follows:

$$\mathrm{SAM}={cos}^{-1}{\displaystyle \frac{\sum xy}{\sqrt{\sum {\left(x\right)}^2\sum {\left(y\right)}^2}}}$$

(25)

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{N}}\sum _{t=1}^N{(x_i-y_i)}^2}$$

(26)

$$\mathrm{ERGAS}=100{\displaystyle \frac{h}{l}}\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left({\displaystyle \frac{{\mathrm{RMSE}}_i}{{\mu }_i}}\right)}^2}$$

(27)

where h and l denote the spatial resolution of the super-resolution image and the original image, respectively, and N is the number of bands (channels).

We evaluate real-world images using the perception index (PI) [70], which combines Ma et al.’s reference-free image quality metric [71] and NIQE [72] and can be expressed as

$$\mathrm{PI}={\displaystyle \frac{1}{2}}((10-\mathrm{Ma})+\mathrm{NIQE})$$

(28)

Higher PSNR and SSIM values indicate a better SR quality, while lower RMSE, SAM, ERGAS, and PI values indicate a better reconstruction quality.

We implemented our SR model using PyTorch and performed all experiments on an HPC platform equipped with an Nvidia A100 80G GPU. The model in the RSSCN7 dataset takes as input a randomly cropped LR image with a size of 48 × 48 pixels, while the corresponding HR size is 192 × 192 pixels. The model in the RSSRD-KQ dataset takes as input a randomly cropped LR image with a size of 64 × 64 pixels, while the corresponding HR size is 192 × 192 pixels. During training, the batch size was set to 16 and the HR/LR images were randomly rotated for data enhancement. The initial learning rate was set to $1\times {10}^{-4}$ and reduced by a factor of 0.1 after 80 epochs. A total of 200 epochs were trained using the Adam optimizer with ${\beta }_1$ = 0.9 and ${\beta }_2$ = 0.99.

Regarding the model parameter settings, we configured the number of RSSGs, RCGs, and GDRMs to be 4. The parameter settings of the RSSBs in the RSSGs were consistent with those of the MambaIR, including six VSSB modules and an SSM state expansion factor of 16. The value of $\lambda$ in SimAM in GDRM was set to $1\times {10}^{-4}$, and the convolutional kernel sizes of MSSA were 3, 7, 13, and 17, with corresponding padding of 1, 3, 6, and 8.

4.3. Results

We performed comparative analyses with current state-of-the-art methods, including Bicubic, SRCNN [18], SRGAN [36], EDSR [19], RRDBNet [14], RCAN [16], SwinIR [20], HAT [41], SwinIR-NG [42], and MambaIR [55]. Visual comparison results on the RSSCN7 and RSSRD-KQ datasets are shown in Figure 9 and Figure 10, respectively. Figure 9 shows the full results, while Figure 10 focuses on the local zoomed-in details of the results from the different models, where the subfigures within the red and green rectangles represent the zoomed-in view of the yellow rectangle.

In the RSSCN7 dataset, the agricultural scene contains a variety of complex geographic and artificial features, making detail and edge processing one of the key factors in evaluating model performance. As shown in Figure 9, the image displays a farm field, with the upper half showing plowed land and the lower half showing untreated land. ConvMambaSR shows significant advantages, especially in detail reconstruction and edge sharpening. The strip-like features commonly found in agricultural scenes require a model with high resolution and detail retention. In contrast, the results of other models are often blurry, failing to capture the boundaries and details of these features accurately, and they are particularly poor at reconstructing strip-like features.

ConvMambaSR also shows significant advantages on the RSSRD-KQ dataset when processing images that have undergone complex quality reduction. As shown in Figure 10, the image displays four adult yaks and two young yaks. The LR image shows that after complex texture reduction processing, the image contains very little local information. Other models generally exhibit blurring phenomena when processing complex mass reduction images and lack sufficient detail extraction and reconstruction capabilities, resulting in blurred details in the reconstructed images, which cannot effectively recover the overall contours and shapes of the yak. In contrast, ConvMambaSR acquires the global receptive field by fusing Mamba, which enables the extraction of more globally dependent features, successfully reconstructs the overall contour and shape of the yak, and improves the recognizability of the image. It is worth noting that MambaIR also achieves good results.

In addition, a comprehensive quantitative evaluation of the model’s performance on both datasets is shown in

Table 1. Quantitative comparison results for the RSSCN7 dataset and RSSRD-KQ dataset. Bold data indicate the best method.

Dataset	Method	PSNR (dB)↑	SSIM↑	RMSE↓	SAM↓	ERGAS↓
RSSCN7	Bicubic	25.50	0.6457	15.0848	0.1570	4.2513
	SRCNN [18]	25.70	0.6551	14.7016	0.1534	4.1536
	SRGAN [36]	25.96	0.6686	14.2594	0.1492	4.0402
	EDSR [19]	26.00	0.6706	14.2090	0.1487	4.0256
	RRDBNet [14]	25.98	0.6696	14.2405	0.1489	4.0346
	RCAN [16]	26.00	0.6694	14.2109	0.1487	4.0257
	SwinIR [20]	26.02	0.6727	14.1720	0.1483	4.0153
	HAT [41]	26.04	0.6734	14.1480	0.1481	4.0092
	SwinIR-NG [42]	26.04	0.6734	14.1453	0.1480	4.0084
	MambaIR [55]	25.98	0.6697	14.2493	0.1490	4.0363
	ConvMambaSR(ours)	26.06	0.6751	14.1029	0.1477	3.9985
RSSRD-KQ	Bicubic	23.74	0.3471	16.9950	0.1495	3.8872
	SRCNN [18]	24.06	0.3541	16.2812	0.1435	3.7173
	SRGAN [36]	24.06	0.3582	16.2846	0.1435	3.7155
	EDSR [19]	24.18	0.3648	16.0725	0.1415	3.6685
	RRDBNet [14]	24.19	0.3666	16.0470	0.1414	3.6614
	RCAN [16]	24.27	0.3740	15.9059	0.1404	3.6290
	SwinIR [20]	24.20	0.3672	16.0310	0.1412	3.6586
	HAT [41]	24.21	0.3684	16.0068	0.1411	3.6530
	SwinIR-NG [42]	24.20	0.3676	16.0305	0.1412	3.6587
	MambaIR [55]	24.23	0.3663	15.9726	0.1407	3.6450
	ConvMambaSR(ours)	24.29	0.3752	15.8632	0.1398	3.6205

. The proposed ConvMambaSR model outperforms other models in terms of PSNR, SSIM, RMSE, SAM, and ERGAS. Transformer-based models outperformed CNN-based models in overall performance across both datasets. This superiority is especially evident in agricultural scenes with complex geographic and artificial features as shown in Figure 9. These results are consistent with previous research, such as the work on SwinIR [20], which demonstrated that Transformer architectures generally excel in capturing fine details and maintaining high-quality reconstructions compared with CNN-based approaches. The introduction of the attention mechanism [16,41,42] plays a key role in RSISR, and models based on the attention mechanism such as RCAN, HAT, SwinIR-NG, and ConvMambaSR show excellent performance. It is worth noting that MambaIR shows excellent performance under the RSSRD-KQ dataset, and the global receptive field brought by Mamba can better help the model to improve its performance in complex degradation scenarios compared with SwinIR, HAT, and SwinIR-NG. Furthermore, RCAN’s Channel Attention mechanism allows it to adaptively adjust the features by taking into account the interdependencies between the feature channels, allowing it to achieve excellent results on the RSSRD-KQ dataset as well. Combining the excellent performance of MambaIR and RCAN and the achievements of other models, in the RSSRD-KQ dataset, after the complex degradation of the image, there is a lot of information between the global dependence and the channels, and the application of the global receptive field and the Channel Attention can greatly improve the performance of the model in complex degradation scenarios.

Figure 11 shows the performance of each model for the public dataset RSSCN7 in seven categories: grassland, farmland, industrial areas, rivers and lakes, forests, residential areas, and parking lots. The experiments show that grassland, rivers, and lakes have the highest SR reconstruction accuracy, while residential areas, forests, and parking lots have the lowest SR accuracy. The grassland and farmland scenes are mainly composed of low-frequency components, and the relatively smooth texture leads to higher PNSR accuracy, but in this case, the PSNR may not adequately reflect the real image quality improvement, and the SSIM metrics are more in line with human visual perception. On the other hand, residential areas, forests, and parking lots are full of complex details and contain a large amount of high-frequency information. The recovery and reconstruction of this high-frequency information is more challenging, resulting in a relatively low SR accuracy.

Overall, the SR performance of all deep learning models, except bicubic interpolation, is consistent across all land cover categories. ConvMambaSR and SwinIR-NG perform well across all categories.

4.4. Effects of GDRM

To assess the efficacy of GDRM, we conducted ablation experiments utilizing CNN and Mamba branches, respectively. The experimental outcomes are presented in

Table 2. Comparison of the proposed model with different branch models on the RSSCN7 dataset and RSSRD-KQ dataset. Bold data indicate the best method.

Dataset	Method	PSNR(dB)↑	SSIM↑	RMSE↓	SAM↓	ERGAS↓
RSSCN7	CNN Branch	26.05	0.6740	14.1201	0.1478	4.0032
	Mamba Branch	26.00	0.6704	14.2115	0.1487	4.0269
	ConvMambaSR	26.06	0.6751	14.1029	0.1477	3.9985
RSSRD-KQ	CNN Branch	24.23	0.3698	15.9706	0.1408	3.6443
	Mamba Branch	24.25	0.3706	15.9296	0.1405	3.6353
	ConvMambaSR	24.29	0.3752	15.8632	0.1398	3.6205

. The comparison demonstrates that GDRM exhibits superior performance compared with the single-branch model. ConvMambaSR exhibits higher PSNR and SSIM values, as well as lower RMSE, SAM, and ERGAS values. This indicates that ConvMambaSR effectively fuses the local details of CNN and the global context information of Mamba. Furthermore, we observe that the CNN branch outperforms the Mamba branch on the RSSCN7 dataset but is inferior to the Mamba branch on the RSSRD-KQ dataset. This suggests that deep residual-based CNNs produce favorable results for the pixel-intensive SR reconstruction task, despite the limited ability of CNNs to model long-range dependencies. However, for complex degraded scenes, the substantial reduction in local details in the image leads to a deteriorated CNN performance, whereas the global receptive field of Mamba enables the extraction of more information. This indicates that the fusion of local details and global features enables the model to achieve excellent performance in different SR reconstruction tasks.

Figure 12 presents a visual comparison of the RSSCN7 and RSSRD-KQ datasets across different branches; the image below displays three adult yaks. It can be observed that the performance is significantly enhanced following the fusion of the two branches using GDRM. ConvMambaSR is able to perfectly reconstruct the strip-like features in the farmland scene of the RSSCN7 dataset. Furthermore, ConvMambaSR is adept at effectively mitigating the adverse effects of motion blur degradation observed in the RSSRD-KQ dataset. Additionally, ConvMambaSR demonstrates satisfactory performance in terms of SSIM metrics. The high SSIM values validate the fused model’s advantages in detailed and global feature reconstruction and demonstrate that it outperforms the CNN or Mamba branch alone in overall visual perception.

4.5. Ablation Study on RCG Count

To further explore the impact of the number of RCGs on model performance, we conducted an ablation study by varying the RCG count while monitoring the number of parameters, FLOPs, PSNR, and SSIM.

The results in Table 2 and

Table 3. Performance metrics with varying RCG counts on the RSSCN7 dataset.

RCG Count	#Para ms (M)	FLOPs (G)	PSNR (dB)↑	SSIM↑
1	6.72	175	26.05	0.6742
4	8.71	226	26.05	0.6742
8	11.37	294	26.06	0.6751
12	14.02	362	26.06	0.6751
16	16.68	430	26.07	0.6756
20	19.34	498	26.07	0.6759

indicate that using only the Mamba branch, without any RCGs, the model achieves a PSNR of 26.00 dB and an SSIM of 0.6704. Introducing just 1 RCG significantly improves the performance, raising the PSNR to 26.05 dB and the SSIM to 0.6742. This indicates that incorporating RCGs into the architecture enhances the model’s ability to capture essential features, leading to better reconstruction quality.

As the number of RCGs increases, further improvements in both PSNR and SSIM are observed. For instance, increasing the RCG count from 1 to 20 results in a PSNR increase from 26.05 dB to 26.07 dB and an SSIM increase from 0.6742 to 0.6759. However, these performance gains diminish with higher RCG counts, suggesting that while more RCGs contribute positively to the model’s output, the marginal benefits decrease after a certain threshold.

Moreover, the increase in the number of RCGs leads to a corresponding rise in the model’s computational complexity, as evidenced by the growing number of parameters and FLOPs. Therefore, it is crucial to balance the trade-off between the performance improvements and the associated computational costs when selecting the optimal RCG count, particularly for practical applications where efficiency is a critical consideration.

4.6. LAM Analysis and Feature Visualization

Local Attribution Map (LAM) [73] is a method based on Integrated Gradients [74] designed to analyze and visualize the contribution of individual input pixels to the output of deep SR networks. Additionally, LAM introduces a Diffusion Index (DI) to quantitatively measure the extent of pixel involvement in the reconstruction process.

With LAM, we can identify which input pixels contribute to the selected region. As shown in Figure 13, the points marked in red are the pixels that contribute to the reconstruction. It is evident that Transformer-based SwinIR [20] has a considerably larger receptive field and activates a greater number of pixels with a wider distribution than CNN-based EDSR [19]. However, SSM-based MambaIR [55] is capable of utilizing pixels across a variety of regions within the entire image, thereby facilitating reconstruction and activating the greatest number of pixels. The dual-branching structure of Mamba and CNN, along with the design of GDRM, enables ConvMambaSR to leverage the strengths of both SSMs and CNNs. Despite a reduction in the number of activated pixels, ConvMambaSR is capable of utilizing the correct pixels globally for reconstruction, as evidenced by its superior performance.

4.7. Complexity and Efficiency Evaluation

In order to quantitatively evaluate the complexity and computational efficiency of our proposed models, we calculated the number of training parameters (#Params), the number of floating-point operations (FLOPs), and the number of frames processed per second (FPS) for the different models. The FLOPs are measured with an input size of 160 × 160 × 3. The FPS is measured on an Nvidia A100 80G GPU. The scale factor was set to 4.

As demonstrated in

Table 4. Comparison of model complexity and efficiency.

Model	#Params (M)	FLOPs (G)	FPS
Bicubic	-	-	22557.1
SRCNN [18]	0.02	8	620.7
EDSR [19]	1.51	50	268.0
RRDBNet [14]	16.69	459	27.5
RCAN [16]	15.59	408	20.6
MambaIR [55]	4.65	123	10.3
HAT [41]	26.03	703	4.4
SwinIR [20]	16.57	456	0.5
SwinIR-NG [42]	19.35	460	0.4
ConvMambaSR(ours)	14.02	362	8.9

, the CNN-based model exhibits the fastest inference speed among all the deep learning models, a result that can be attributed to the superior efficiency of the convolutional parallel operation. The inference speed of Mamba is considerably faster than that of Transformer due to its linear complexity and efficient inference. A comparison of Table 1 and Table 4 reveals that the complexity and inference speed of our proposed model are comparable to those of most other models, while the model performance is markedly superior. This provides compelling evidence of the model’s potential for practical applications.

4.8. Real-World Image Testing

The performance of each model was evaluated using real-world images in RSSRD-KQ, comprising a total of 48 images, each with a size of 480 × 480 pixels. The scale factor was set to 3.

As illustrated in

Table 5. Quantitative comparison results for real-world images. Bold data indicate the best method.

Method	PI↓
SRCNN [18]	8.6022
EDSR [19]	8.2448
RRDBNet [14]	8.1700
RCAN [16]	8.2068
SwinIR [20]	8.2149
HAT [41]	8.1619
SwinIR-NG [42]	8.2149
MambaIR [55]	8.1346
ConvMambaSR(ours)	8.0496

, ConvMambaSR exhibits the lowest PI relative to other deep learning models. It is noteworthy that, similar to the outcomes of the preceding experiments, MambaIR continues to achieve highly favorable outcomes. The global receptive field introduced by Mamba is of significant benefit in addressing complex degradation cases, offering a substantial advantage over Transformer and CNN. This indicates that the presence of more information in the image is dependent on distance, and the enlargement of the model’s receptive field will result in enhanced performance. The Mamba architecture is capable of significantly improving the blind SISR of remote sensing images.

Furthermore, the experiments demonstrate that the degradation method employed for RSSRD-KQ enables the model to learn certain degradation processes in the real world. As shown in Figure 14, the image displays a high-altitude wetland scene, with some snow covering the area, the phenomena such as motion blur and color fringes that are evident in the original image captured by the UAV are partially resolved.

5. Conclusions

This paper presents ConvMambaSR, a novel SISR method tailored for remote sensing applications. ConvMambaSR leverages the complementary strengths of CNNs and SSMs, effectively combining their abilities. Our experimental results reveal that a single model architecture is often inadequate for addressing the diverse challenges posed by remote sensing imagery, highlighting the necessity of integrating multiple models to achieve superior results. To comprehensively capture both local and global information from different branches, we introduce the global–detail reconstruction module, which fuses the locality-capturing capability of CNNs with the global dependency modeling power of SSMs. The experimental outcomes confirm the effectiveness of ConvMambaSR, demonstrating its significant improvements over existing methods. ConvMambaSR demonstrates state-of-the-art performance on the RSSCN7 and RSSRD-KQ datasets, outperforming existing methods in visual quality. Specifically, the model achieves PSNR and SSIM scores of 26.06 and 0.6751 on the RSSCN7 dataset and 24.29 and 0.3752 on the RSSRD-KQ dataset. Notably, across the seven representative categories within the RSSCN7 dataset, ConvMambaSR consistently exhibits superior performance. Furthermore, the model significantly exceeds the inference speed of Transformer-based approaches, achieving 8.9 FPS. This combination of efficiency and high performance underscores ConvMambaSR’s potential for a wide range of super-resolution tasks, positioning it as a strong candidate for broader real-world applications.

Despite the promising results achieved with ConvMambaSR, there are several limitations in the current study. One notable limitation observed through LAM analysis and feature visualization is that, although the model inherits the global modeling capabilities of SSMs and activates more correct pixels for reconstruction, there is still an issue with the reduction in the number of activated pixels. Moreover, although the current approach is effective, it is computationally intensive, leading to significant resource consumption. In future research, we will explore further performance enhancements of hybrid models for blind remote sensing SISR and optimize our approach to improve model efficiency.