Multiscale Spatial–Spectral Dense Residual Attention Fusion Network for Spectral Reconstruction from Multispectral Images

Abstract

Spectral reconstruction (SR) from multispectral images (MSIs) is a crucial task in remote sensing image processing, aiming to enhance the spectral resolution of MSIs to produce hyperspectral images (HSIs). However, most existing deep learning-based SR methods primarily focus on deeper network architectures, often overlooking the importance of extracting multiscale spatial and spectral features in the MSIs. To bridge this gap, this paper proposes a multiscale spatial–spectral dense residual attention fusion network (MS2Net) for SR. Specifically, considering the multiscale nature of the land-cover types in the MSIs, a three-dimensional multiscale hierarchical residual module is designed and embedded in the head of the proposed MS2Net to extract spatial and spectral multiscale features. Subsequently, we employ a two-pathway architecture to extract deep spatial and spectral features. Both pathways are constructed with a single-shot dense residual module for efficient feature learning and a residual composite soft attention module to enhance salient spatial and spectral features. Finally, the spatial and spectral features extracted from the different pathways are integrated using an adaptive weighted feature fusion module to reconstruct HSIs. Extensive experiments on both simulated and real-world datasets demonstrate that the proposed MS2Net achieves superior performance compared to state-of-the-art SR methods. Moreover, classification experiments on the reconstructed HSIs show that the proposed MS2Net-reconstructed HSIs achieve classification accuracy that is comparable to that of real HSIs.

1. Introduction

Hyperspectral remote sensing has attracted much attention due to its ability to capture continuous narrow-band spectral signatures, generally from 400 to 2500 nm, covering the visible to the short-wave infrared [1,2]. The captured hyperspectral image (HSI) products, which contain rich spectral information, play a crucial role in a variety of applications, such as agricultural analysis [3], environmental monitoring [4], and mineral exploration [5]. Despite their immense potential, acquiring HSIs remains costly and technically challenging due to hardware constraints, which limits their widespread application. A practical alternative is to use multispectral images (MSIs), which are more accessible and easier to obtain. However, MSIs suffer from lower spectral resolution, which limits their ability to capture fine spectral details and to distinguish subtle variations in land-cover characteristics effectively. To address this limitation, this study focuses on enhancing the spectral resolution of MSIs to generate HSIs using the spectral reconstruction (SR) technique.

Generally, SR is an ill-posed transition problem since the reconstructed HSIs are not unique [6]. Numerous SR methods have been proposed over the past decade to mitigate this issue. In earlier studies, researchers preferred to apply the sparse representation-based method to reconstruct HSIs from MSIs. These methods first extract the HSIs dictionary and then calculate the sparse coefficients of the MSIs. Subsequently, the learned HSI dictionary is combined with the sparse coefficients to reconstruct the HSIs [7,8,9,10,11,12,13]. However, the HSI dictionary does not accurately represent the nonlinear spectral signatures of complex objects. Additionally, these sparse representation-based methods are sensitive to parameter settings and lack adaptive parameter selection capability, affecting their generalization ability and robustness. Recently, with the significant growth of machine learning, learning-based SR methods have emerged, aiming to build mappings from MSIs to the corresponding HSIs using learned models. Among these models, deep learning (DL)-based SR methods are highly preferred by researchers. Since the HSCNN that was proposed in 2017 [14], a growing number of convolutional neural network (CNN)-based SR methods have been proposed [15,16,17,18,19,20,21,22,23,24,25] due to their robust nonlinear fitting and learning capabilities. Compared with the sparse representation-based schemes, these CNN-based methods significantly improve the reconstruction performance through deeper or wider architectures. Nevertheless, as the network depth or width increases, a large number of redundant features tend to be extracted, which not only hinders the learning process, but also reduces the efficiency of exploiting the global modeling capabilities, thus limiting the reconstruction quality.

Fortunately, attention mechanisms, which can guide the network to adaptively focus on key features and enhance global dependency modeling capabilities, have been widely studied and adopted in SR tasks [26]. Most attention-based SR methods primarily leverage channel attention, spatial attention, or self-attention mechanisms to design effective attention modules. Specifically, channel attention focuses on adaptive re-weighting different feature maps to highlight the most informative ones, and thus improves the model capacity to prioritize key features in the SR process [27,28,29,30,31]. The spatial attention could make the model focus on specific spatial regions, and thus enhance the spatial details of the reconstructed HSIs [32]. To capture both channel-wise dependencies and spatial features simultaneously, some SR schemes based on dual attention have been proposed [33,34,35,36,37]. Moreover, leveraging on the advantages of Vision Transformer (ViT) in capturing long-range dependencies and global context, self-attention has recently been widely adopted in SR tasks, which significantly improves its detail recovery and global perception [38,39,40,41].

Although the aforementioned studies have achieved promising reconstruction performance, most of them struggle to effectively capture the multiscale spatial and spectral features inherent in MSIs [42,43], primarily due to the significant variations in object sizes and the divergence of spectral signatures across different geographical regions, as illustrated in Figure 1. Fortunately, increasing the model depth or incorporating attention mechanisms have been widely used methods to enhance feature extraction. However, these approaches inevitably increase computational complexity and may cause gradient vanishing or exploding, thus limiting their practical applicability. Consequently, it is challenging to effectively capture the rich multiscale spatial and spectral features of MSIs while simultaneously maintaining a balance between computational efficiency and model stability.

To solve the above challenges, this paper proposes a multiscale spatial–spectral dense residual attention fusion network (MS2Net) for SR. Technically, we first apply a three-dimensional (3-D) multiscale hierarchical residual (3-D MHR) module to extract the multiscale spatial and spectral features from the MSIs. Then, the extracted multiscale features are passed through two branches to further extract deep spatial and spectral features. In particular, we design the single-shot dense residual (SSDR) module by replacing dense connections with single-shot connections in each branch to reduce the computational cost and maintain the gradient flow. Furthermore, the residual composite soft attention (RCSA) module is designed to dynamically guide the model to focus on valuable features and empowers the model with global modeling capabilities. Finally, these high-level spectral and spatial features, extracted from the two branches, are fused by the adaptive weighted feature fusion (AWFF) module to reconstruct the HSIs. The main contributions and findings of this paper can be summarized as follows:

• First, the 3-D MHR module is proposed to extract multiscale spatial and spectral features from MSIs. Specifically, the proposed 3-D MHR module not only expands the receptive field through the hierarchical residual structure but also enables fine-grained multiscale feature extraction by dividing the input feature into subsets, where each subset is processed sequentially to capture specific spatial and spectral details. In this way, the feature diversity and representation are enhanced.
• Second, a novel two-branch network architecture is designed, comprising a spectral and a spatial feature extraction branch, each equipped with an SSDR and an RCSA module. The SSDR module in the spectral branch employs a 1 × 1 × 3 convolution to capture spectral dependencies, whereas, in the spatial branch, it uses a 3 × 3 × 1 convolution to extract local spatial patterns. Moreover, instead of using dense connections, the SSDR module uses single-shot connections to facilitate feature reuse, which reduces complexity while ensuring effective feature propagation. The RCSA module is connected behind the SSDR module, which can adaptively recalibrate feature responses by emphasizing important spectral and spatial features while enhancing the global representation capability.
• Third, the AWFF module is introduced to integrate spectral and spatial features extracted from different branches to improve the reconstruction quality. Extensive experiments on four MSI datasets, including two simulated and two real-world datasets, using eight state-of-the-art methods, consistently demonstrates that the proposed MS2Net achieves superior reconstruction performance.

The remainder of this paper is organized as follows. Section 2 reviews several SR methods, such as sparse representation-based methods, CNN-based methods, and attention-based methods. Section 3 presents the details of the proposed method. After that, the experiment results and discussion are provided in Section 4 and Section 5, respectively. At last, Section 6 presents the conclusions and future research directions.

2. Related Works

2.1. Sparse Representation-Based Methods for SR

In the early stages of this research, sparse representation-based methods dominated SR tasks. Specifically, Yi et al. [8] combined the sparse representation and unmixing models to reconstruct HSIs. Gao et al. [9] proposed a simple but effective method, named J-SLoL, to improve MSIs spectral resolution using the extracted low-rank HSI dictionary and sparse coefficients. Similarly, Fotiadou et al. [10] designed a post-acquisition method based on sparse representations and dictionary learning to improve the spectral resolution of MSIs. Han et al. [11] proposed a spectral library-based dictionary learning method to recover missing spectral signatures from MSIs. Furthermore, some sparse representation methods are dedicated to reconstructing HSIs from visible-light images (e.g., RGB images), which can be regarded as a specific case of MSI SR. For instance, Arad and BenShahar [12] proposed a low-cost and fast approach to recover HSIs directly from RGB images. Their method first learns a sparse dictionary from the HSIs prior via K-means decomposition and projects it to the corresponding RGB images. Then, the orthogonal match pursuit algorithm is used to recover HSIs. Moreover, Aeschbacher et al. [13] used shallow learning models to compute a sparse dictionary from a specific spectral prior to recovering lost spectral resolution, which further improved the reconstruction performance of [12].

2.2. CNN-Based Methods for SR

In the past few years, CNN-based SR methods have received much attention due to their ability to learn deep feature representations automatically from MSIs. Specifically, inspired by residual networks [16] and dense networks [17], Shi et al. [15] proposed HSCNN-R and HSCNN-D networks for SR, respectively. Similarly, Can and Timofte [18] proposed a nine-layer residual network for SR, in which the adaptively parametric rectifier linear unit is used to increase the nonlinearity of the features. Furthermore, learning from the semantic segmentation task, Stiebel et al. [19] proposed SR networks based on U-net, in which the pooling and batch normalization operations are abandoned. To make the most of the contextual features, Zhao et al. [20] employed the pixel unshuffle operation to generate four-level different resolution inputs and thus construct a hierarchical regression network for SR. In addition, Koundinya et al. [21] proposed two-dimensional (2-D) and three-dimensional (3-D) CNNs for SR, in which the former network yielded low computational complexity while the latter network achieved the best reconstruction results. Fu et al. [22] presented an HSI reconstruction network based on spectral nonlinear mapping units and spatial similarity units and applied camera spectral response selection to improve the network efficiency. Wu et al. [23] introduced a lightweight SR method in which a polymorphic residual context restructuring module is used instead of the traditional residual module to encode spatial and spectral context features. In addition, based on the intrinsic characteristics of HSIs, such as spectral correlation and projection properties, Hang et al. [24] proposed a three-stage SR model. In the first stage, the band correlation coefficients are calculated to obtain the correlation matrix, and the subsequent two stages use CNN-based residual blocks to extract deep features. Similarly, Dian et al. [25] proposed an HSI prior learning module based on CNNs and used it to construct two sub-networks for extracting spatial and spectral prior features from HSIs.

2.3. Attention-Based Methods for SR

Inspired by human visual processing, the attention mechanism is introduced to help networks focus on important features while ignoring irrelevant ones; it is widely used in SR tasks. For example, Li et al. [27] proposed an adaptive weighted attention network for SR, in which the squeeze-and-excitation attention module [28] is introduced to recalibrate each channel contribution of feature maps. Similarly, in [29], a hybrid 2-D and 3-D depth residual attention network is proposed, in which the 2-D and 3-D attention mechanisms can adaptively recalibrate the channels and bands of the feature maps, respectively. Inspired by [30], Zheng et al. [31] proposed a spatial–spectral residual attention network for SR from MSI to maintain the association between adjacent bands. In addition, many studies based on dual attention mechanisms have been presented for SR which enable the network to simultaneously enhance and model global features in both channel and spatial dimensions. Specifically, Wang et al. [33] proposed a dense residual network with dual attention. The channel attention module in their network follows the design of [28]. For the spatial attention module, a downsampling strategy is first employed to enhance spatial information, followed by a non-local operation [32] to compute spatial attention weights. Li et al. [34] constructed a double-branch hybrid attention network for SR in which the dual attention network (DANet) [35] is inserted separately into two branches. Sun et al. [36] proposed a hybrid spectral and texture attention pyramid network, in which convolutional block attention modules [37] are used to form spatial and spectral constraint units. Recently, the self-attention module in ViT, with its powerful global modeling capability, has been widely used to SR tasks [38]. For instance, Cai et al. [39] proposed a lightweight multi-stage spectral-wise transformer network (MST++), where the self-attention module computes the relationship between different channels. Building upon MST++, Wu et al. [40] proposed a multistage spatial–spectral fusion network (MSFN) for SR from MSIs. This network leverages spatial correlation and spectral self-similarity, incorporating two fundamental building blocks: a spectral module that employs spectral-wise multi-head self-attention, as introduced in [39], and a spatial module based on window-based multi-head self-attention [41].

3. Proposed Method

3.1. Overall Pipeline

The pipeline of the proposed MS2Net is displayed in Figure 2. Generally, the proposed pipeline is divided into three parts: data preprocessing, spectral–spatial feature extraction, and feature fusion and reconstruction. Specifically, to make the training samples contain both spectral signatures and spatial information, we used a sliding window to obtain 3-D patches. Then, these 3-D patches are sent to the initial convolutional layer to extract low-level features. Next, the extracted shallow features are fed to a 3-D MHR module to mine multiscale spectral and spatial features. Next, the multiscale spectral–spatial joint features are fed into the spectral and spatial branches to extract deep-level spectral and spatial features, respectively. In each branch, we use the SSDR module to enhance the feature flow. Additionally, the RCSA module is inserted in each branch to focus on the features of interest for SR in both channel and spatial dimensions. Then, the features from the two branches are adaptively fusion-weighted by the AWFF module. Finally, the feature maps are fed into a convolutional layer to obtain the final reconstructed HSI. The detailed configurations of the proposed MS2Net are listed in

Table 1. Detailed configurations of the proposed MS2Net based on the PU dataset, where # is the the number of feature maps.

Module	Input Shape	Layer Operations	Kernel Size	Padding	Stride	Filters	Output Shape
Input	(5, 5, 4, 1)	Conv-3D and PReLU	(1, 1, 1)	(0, 0, 0)	(1, 1, 1)	100	(5, 5, 4, 100)
3-D MHR	(5, 5, 4, 100)	Split	/	/	/	/	#4 (5, 5, 4, 25)
	(5, 5, 4, 25)	/	/	/	/	/	(5, 5, 4, 25)
	(5, 5, 4, 25)	Conv-3D and PReLU	(3, 3, 3)	(1, 1, 1)	(1, 1, 1)	25	(5, 5, 4, 25)
	#2 (5, 5, 4, 25)	Add and Conv-3D and PReLU	(3, 3, 3)	(1, 1, 1)	(1, 1, 1)	25	(5, 5, 4, 25)
	#2 (5, 5, 4, 25)	Add and Conv-3D and PReLU	(3, 3, 3)	(1, 1, 1)	(1, 1, 1)	25	(5, 5, 4, 25)
	#4 (5, 5, 4, 25)	Concatenate	/	/	/	/	(5, 5, 4, 100)
Spectral-SSDR	(5, 5, 4, 100)	Conv-3D and PReLU	(1, 1, 3)	(0, 0, 1)	(1, 1, 1)	100	(5, 5, 4, 100)
	(5, 5, 4, 100)	Conv-3D and PReLU	(1, 1, 3)	(0, 0, 1)	(1, 1, 1)	100	(5, 5, 4, 100)
	(5, 5, 4, 100)	Conv-3D and PReLU	(1, 1, 3)	(0, 0, 1)	(1, 1, 1)	100	(5, 5, 4, 100)
	#4 (5, 5, 4, 100)	Concatenate	/	/	/	/	(5, 5, 4, 400)
	(5, 5, 4, 400)	Conv-3D and PReLU	(1, 1, 1)	(0, 0, 0)	(1, 1, 1)	100	(5, 5, 4, 100)
	#2 (5, 5, 4, 100)	Add	/	/	/	/	(5, 5, 4, 100)
	(5, 5, 4, 100)	Conv-3D and PReLU	(1, 1, 4)	(0, 0, 0)	(1, 1, 1)	100	(5, 5, 100)
RCSA	(5, 5, 100)	Attention	/	/	/	/	(5, 5, 100)
	#2 (5, 5, 100)	Multiplication	/	/	/	/	(5, 5, 100)
Spatial-SSDR	(5, 5, 4, 100)	Conv-3D and PReLU	(3, 3, 1)	(1, 1, 0)	(1, 1, 1)	100	(5, 5, 4, 100)
	(5, 5, 4, 100)	Conv-3D and PReLU	(3, 3, 1)	(1, 1, 0)	(1, 1, 1)	100	(5, 5, 4, 100)
	(5, 5, 4, 100)	Conv-3D and PReLU	(3, 3, 1)	(1, 1, 0)	(1, 1, 1)	100	(5, 5, 4, 100)
	#4 (5, 5, 4, 100)	Concatenate	/	/	/	/	(5, 5, 4, 400)
	(5, 5, 4, 400)	Conv-3D and PReLU	(1, 1, 1)	(0, 0, 0)	(1, 1, 1)	100	(5, 5, 4, 100)
	#2 (5, 5, 4, 100)	Add	/	/	/	/	(5, 5, 4, 100)
	(5, 5, 4, 100)	Conv-3D and PReLU	(1, 1, 4)	(0, 0, 0)	(1, 1, 1)	100	(5, 5, 100)
RCSA	(5, 5, 100)	Attention	/	/	/	/	(5, 5, 100)
	#2 (5, 5, 100)	Multiplication	/	/	/	/	(5, 5, 100)
AWFF	#2 (5, 5, 100)	Feature Fusion	/	/	/	/	(5, 5, 100)
Output	(5, 5, 100)	Conv-2D and PReLU	(1, 1)	(0, 0)	(1, 1)	103	(5, 5, 103)

.

3.2. 3-D MHR Module

To effectively extract multiscale spectral–spatial features on a fine-grained level and simultaneously enhance the range of receptive fields, the 3-D MHR module, as shown in Figure 2, is introduced and incorporated into the head of the proposed MS2Net. Specifically, the structure of 3-D MHR module is shown in Figure 3.

As depicted in Figure 3, the 3-D MHR module comprises four scales, with the symbol “⊕” indicating the addition operation. Given input tensor $\mathbf{X}\in {\mathbb{R}}^{C\times D\times H\times W}$, where $\mathbf{X}$ denote the MSI patch and C, D, H, and W denotes the channel, depth, height, and width, respectively. We first split $\mathbf{X}$ into four feature subsets along the channel dimension, and each subset is denoted as ${\mathbf{X}}_i\in {\mathbb{R}}^{C/4\times D\times H\times W}$, where $i\in \{1,2,\dots ,4\}$. Next, except for ${\mathbf{X}}_1$, each ${\mathbf{X}}_i$ is sent to a 3 × 3 × 3 convolution layer, denoted by ${\mathit{Conv}}_i()$. The extracted feature maps of ${\mathit{Conv}}_i()$ are denoted by ${\mathbf{Y}}_i$. Subsequently, the feature subset ${\mathbf{X}}_i$ is added with the output of ${\mathit{Conv}}_{i-1}()$ and then fed into ${\mathit{Conv}}_i()$. In this way, the multiscale hierarchical residual features ${\mathbf{Y}}_i$ can be defined as follows:

$${\mathbf{Y}}_i=\left\{\begin{array}{cc}{\mathbf{X}}_i\hfill & i=1\hfill \\ Con{v}_i({\mathbf{X}}_i)\hfill & i=2\hfill \\ Con{v}_i({\mathbf{X}}_i+{\mathbf{Y}}_{i-1})\hfill & 2<i\le 4\hfill \end{array}\right.$$

(1)

According to this hierarchical residual structure, each convolution layer ${\mathit{Conv}}_i()$ can receive the feature maps from the previous layer, so the receptive field of ${\mathbf{X}}_i$ is larger than that of ${\mathbf{X}}_j(j\le i)$. Additionally, we use the PReLU activation function instead of the ReLU activation function to bring more nonlinear capabilities to the network. Finally, the concatenation operation at the end of this module combines the feature maps at different receptive fields.

3.3. SSDR Module

Generally, the deeper networks always have higher reconstruction quality. Nevertheless, simply stacking convolutional layers to a certain level can lead to performance bottlenecks. ResNet [16] first proposed a skip connection to alleviate this bottleneck, guaranteeing network performance even when the number of layers is deepened. Moreover, DenseNet inherits and improves the skip connection to make all preceding layers connected to each other, further improving its performance in the SR task. However, not all connections between layers have positive feedback on the results, and the large memory footprint and slow inference time become a drawback of DenseNet [17]. To address this drawback, we propose an SSDR module to reduce computation costs and extract deep spatial and spectral features. As shown in Figure 2, the SSDR module is implemented in the spectral SSDR module and the spatial SSDR module. In the spectral SSDR module, we adopt 1 × 1 × 3 spectral convolution kernels to extract discriminative spectral features. The following equations can describe the procedure of the spectral SSDR module:

$${\mathbf{X}}_1=\delta \left({\mathrm{Conv}}_1^{1\times 1\times 3}\left({\mathbf{X}}_0\right)\right)$$

(2)

$${\mathbf{X}}_2=\delta \left({\mathrm{Conv}}_2^{1\times 1\times 3}\left({\mathbf{X}}_1\right)\right)$$

(3)

$${\mathbf{X}}_3=\delta \left({\mathrm{Conv}}_2^{1\times 1\times 3}\left({\mathbf{X}}_2\right)\right)$$

(4)

$${\mathbf{X}}_{\mathrm{spectral}}={\mathbf{X}}_0+\delta \left({\mathrm{Conv}}_4^{1\times 1\times 1}[{\mathbf{X}}_0,{\mathbf{X}}_1,{\mathbf{X}}_2,{\mathbf{X}}_3]\right)$$

(5)

where ${\mathbf{X}}_0$, ${\mathbf{X}}_1$, ${\mathbf{X}}_2$ denote the input feature maps of the first, second, and third convolution layer inputs, respectively. ${\mathbf{X}}_3$ denotes the third convolutional layer output. ${\mathbf{X}}_{\mathrm{spectral}}$ represents the extracted deep spectral features. $\delta$ denotes the PReLU activation function. The procedure of the spatial SSDR module is the same as that for the spectral SSDR. Specifically, we apply 3 × 3 × 1 spatial convolution kernels to mine the rich spatial contextual features ${\mathbf{X}}_{\mathrm{spatial}}$. The following equations can describe the procedure of the spatial SSDR module:

$${\mathbf{X}}_1=\delta \left({\mathrm{Conv}}_1^{3\times 3\times 1}\left({\mathbf{X}}_0\right)\right)$$

(6)

$${\mathbf{X}}_2=\delta \left({\mathrm{Conv}}_2^{3\times 3\times 1}\left({\mathbf{X}}_1\right)\right)$$

(7)

$${\mathbf{X}}_3=\delta \left({\mathrm{Conv}}_2^{3\times 3\times 1}\left({\mathbf{X}}_2\right)\right)$$

(8)

$${\mathbf{X}}_{\mathrm{spatial}}={\mathbf{X}}_0+\delta \left({\mathrm{Conv}}_4^{1\times 1\times 1}[{\mathbf{X}}_0,{\mathbf{X}}_1,{\mathbf{X}}_2,{\mathbf{X}}_3]\right)$$

(9)

Furthermore, the feature maps output by each convolutional layer in the module have the same size and can be connected in the channel dimension. In each SSDR module, every convolutional layer outputs k feature maps, which means that k new feature maps can be obtained. Referring to [32], we set k to 100 in this module, which can achieve a better reconstruction performance.

3.4. RCSA Module

To fully leverage critical features and capture long-range global dependencies, we introduce a self-attention mechanism [38] to design the RCSA module. As shown in Figure 4, given an input feature map ${\mathbf{X}}_{\mathrm{in}}\in {\mathbb{R}}^{C\times H\times W}$, C refers to the number of channels and H and W refer to the input feature map spatial dimensions. First, to introduce the global contextual information in the attention map, ${\mathbf{X}}_{\mathrm{in}}$ is fed into the construction module. This module contains two parallel branches, each containing a global average pooling layer and a linear layer. To obtain richer global contextual information, we use two unequal pooling kernel sizes of $H\times 1$ and $1\times W$ in two branches to generate two new feature maps, i.e., ${\widehat{\mathbf{A}}}_{\mathrm{w}}\in {\mathbb{R}}^{C\times 1\times W}$ and ${\widehat{\mathbf{A}}}_{\mathrm{h}}\in {\mathbb{R}}^{C\times H\times 1}$, respectively. Next, ${\widehat{\mathbf{A}}}_{\mathrm{w}}$ and ${\widehat{\mathbf{A}}}_{\mathrm{h}}$ are repeated along the height and width to form global features ${\mathbf{A}}_{\mathrm{w}}\in {\mathbb{R}}^{C\times H\times W}$ and ${\mathbf{A}}_{\mathrm{h}}\in {\mathbb{R}}^{C\times H\times W}$ in the horizontal and vertical directions, respectively. Furthermore, ${\mathbf{A}}_{\mathrm{w}}$ and ${\mathbf{A}}_{\mathrm{h}}$ are cut along the H and W dimensions to generate a group of slices with the size of ${\mathbb{R}}^{C\times W}$ and ${\mathbb{R}}^{C\times H}$, respectively. After that, these group slices are merged to form the final global context feature map $\mathbf{A}\in {\mathbb{R}}^{(H+W)\times C\times S}$, where S = H = W. At the same time, we also cut the ${\mathbf{X}}_{\mathrm{in}}$ along the H dimension, generating a set of H slices with a size of ${\mathbb{R}}^{C\times W}$. Similarly, we cut the ${\mathbf{X}}_{\mathrm{in}}$ along the W dimension with a size of ${\mathbb{R}}^{C\times H}$. These two types of features are merged to generate the features $\mathbf{B}\in {\mathbb{R}}^{(H+W)\times S\times C}$. In the same way, the feature map $\mathbf{C}\in {\mathbb{R}}^{(H+W)\times C\times S}$ is generated. Then, we perform an affinity operation between $\mathbf{A}$ and $\mathbf{B}$ to generate the attention map $\mathbf{M}\in {\mathbb{R}}^{(H+W)\times C\times C}$. The affinity operation is represented as

$${\mathbf{M}}_{i,j}={\displaystyle \frac{\exp \left({\mathbf{A}}_i·{\mathbf{B}}_j\right)}{{\sum }_{i=1}^C\exp \left({\mathbf{A}}_i·{\mathbf{B}}_j\right)}}$$

(10)

where ${\mathbf{M}}_{i,j}$ means the degree of correlation between the ith and jth channels at a specific spatial location. Then, we perform a matrix multiplication between $\mathbf{M}$ and $\mathbf{C}$, reshaping the results to two groups, and each group has a size of ${\mathbb{R}}^{C\times H\times W}$. Next, these two groups’ feature maps are added to form long-range contextual information. Finally, we multiply it by a learnable scale parameter $\alpha$ and perform an element-wise sum with input feature ${\mathbf{X}}_{\mathrm{in}}$ to obtain the final output ${\mathbf{X}}_{\mathrm{out}}\in {\mathbb{R}}^{C\times H\times W}$. The mathematical expression is as follows:

$${\mathbf{X}}_{\mathrm{out},j}=\alpha \sum _{i=1}^C{\mathbf{M}}_{i,j}·{\mathbf{C}}_j+{\mathbf{X}}_{\mathrm{in},j}$$

(11)

in which $\alpha$ is initialized to be 0 and can be updated during the training process.

3.5. AWFF Module

Due to the different feature extraction methods of the two branches, the two branch features contribute differently to the SR task. To sufficiently fuse spectral and spatial features from different branches, we designed an AWFF module that can adaptively fuse spectral and spatial features. Specifically, we assigned weight coefficients $\mathit{\theta }$ to each branch, thus making the features are better integrated, mathematically expressed as follows:

$${\mathbf{F}}_{\mathrm{Joint}}=\mathit{\theta }·{\mathbf{F}}_{\mathrm{Spe}}+(1-\mathit{\theta })·{\mathbf{F}}_{\mathrm{Spa}}$$

(12)

where ${\mathbf{F}}_{\mathrm{joint}}$, ${\mathbf{F}}_{\mathrm{Spe}}$, and ${\mathbf{F}}_{\mathrm{Spa}}$ denote the weight fused, spectral, and spatial branch feature maps. $\mathit{\theta }$ is an adaptive weight that is updated by back-propagation. Moreover, to obtain a stable and smooth reconstruction result, we chose mean-square error as the loss function of the proposed network. The loss function is defined as follows:

$$\mathrm{Loss}(\mathit{\tau })={\displaystyle \frac{1}{N}}\sum _{n=1}^N{\parallel {\mathbf{Y}}_{\mathrm{GT}}^n-{\mathbf{Y}}_{\mathrm{HSI}}^n\parallel }^2$$

(13)

where ${\mathbf{Y}}_{\mathrm{GT}}^n$ and ${\mathbf{Y}}_{\mathrm{HSI}}^n$ are denoted as the n-th pixel value of ground-truth MSI patches and reconstructed HSI patches, respectively. N denotes the number of pixels in one training patch. $\mathit{\tau }$ represents the parameters of the proposed network.

4. Experiments

4.1. Dataset Description

To thoroughly estimate the effectiveness of the proposed MS2Net, we used two types of MSI datasets in the experiments, one type being simulated MSI datasets (i.e., the Pavia University dataset and the Indian Pines dataset) and the other real-world MSI datasets (i.e., the Chongqing dataset and the Jiaxing dataset) [42,43]. It is worth emphasising that the real-world datasets are recently collected high-quality MSI datasets that are very challenging in experiments.

• Pavia University (PU) Dataset: The first dataset was acquired by the reflective optics system imaging spectrometer airborne sensor over the University of Pavia, Italy. The spatial size of this dataset is 610 × 340 and includes 207,400 pixels, and it contains 9 labeled land-cover types. The ground sampling distance (GSD) is 1.3 m. Every pixel has 115 bands with a spectral wavelength ranging from 430 to 860 nm. After dropping the noise-contaminated bands, the number of spectral bands used for the experiment was 103. To match the requirements of the SR experimental setup, the corresponding MSI was simulated by the black-SWIR1 (i.e., 450–520 nm, 520–600 nm, 630–690 nm, and 760–900 nm) spectral response functions (SRFs) of Quick Bird. The size of the simulated MSI dataset is 610 × 340 × 4. As shown in

  Table 2. Training and test sets of the four datasets.

  Datasets	Training Set		Test Set
  	MSI	HSI	MSI	HSI
  Pavia University
  Indian Pines
  Chongqing
  Jiaxing

  , the training set size is 305 × 340 × 4, and the remaining regions were used as the test set.
• Indian Pines (IP) Dataset: The second dataset was photographed by the airborne visible/infrared imaging dpectrometer sensor over the Indian Pines test site in Northwestern Indiana, USA. The spatial size of this dataset is 145 × 145 and includes a total of 21,025 pixels, and it contains 16 labeled land-cover types. The GSD is 20 m. Every pixel has 220 bands with a spectral wavelength ranging from 400 to 2500 nm. To match the requirements of the SR experimental setup, the corresponding MSI was simulated by the SRFs of Sentinel-2. The size of the simulated MSI dataset is 145 × 145 × 13. As shown in Table 2, the training set size is 75 × 145 × 13, and the remaining regions were used as the test set.
• Chongqing (CQ) Dataset: The third dataset includes the real data collected by the ZY-1 02D satellite in Chongqing, China, which contains paired HSI and MSI for the same region and time. The advanced hyperspectral imager has a GSD of 30 m, while the multispectral imager provides MSIs with a GSD of 10 m. To make the GSD of the MSIs uniform with those of the HSIs, we performed a spatial downsampling operation on the MSI. We selected MSI and HSI with a size of 400 × 1000, containing 400,000 pixels. Every pixel of the HSI has 94 bands with a spectral wavelength ranging from 395 to 1341 nm. The MSI includes eight bands with a spectral wavelength ranging from 452 to 1047 nm. As shown in Table 2, the training set and test set sizes are 400 × 400 × 8 and 400 × 600 × 8, respectively.
• Jiaxing (JX) Dataset: The last dataset was collected by the ZY-1 02D satellite in Jiaxing, China. The GSDs for HSI and MSI are the same as those for the Chongqing dataset. We also performed a spatial downsampling operation on the MSI to ensure that the GSDs of the MSI and HSI were consistent. In this paper, we selected MSI and HSI with a size of 500 × 500, containing a total of 250,000 pixels. Each pixel has 76 bands for HSI and 8 bands for MSI. As shown in Table 2, the training set and test set sizes are 250 × 500 × 8 and 250 × 500 × 8, respectively.

4.2. Experimental Setup

All the experiments were performed on a deep learning workstation with a 2× Intel Xeon E5-2680 v4 processor (2.4 GHz), manufactured by Intel Corporation in Santa Clara, CA, USA, 128 GB of DDR4 RAM, and 8× NVIDIA GeForce RTX 2080Ti super graphical processing unit (GPU) with 11 GB of memory. The software environment was CUDA v11.2, PyTorch 1.10, and Python 3.8 [44]. For the training setup of the proposed MS2Net, we applied the Adam optimizer to update the parameters for 200 epochs. The initial learning rate was set to 0.0001. The batch size was set to 32. Due to the limited computing power, we randomly selected patch pairs from MSI and HSI to train the network. The patch size of the simulated dataset was set to 5 × 5 and 15 × 15 for the real-world dataset. Furthermore, to purposefully demonstrate the effectiveness of our proposed MS2Net, we selected eight representative SR methods for comparison: one sparse representation-based method and seven DL-based methods. All these comparison methods are briefly described in the following:

• J-SLoL [9]: This method performs SR by learning the low-rank HS and MS dictionaries and their corresponding sparse representations.
• AWAN [27]: The backbone of this network is composed of several dual residual attention modules, where the long and short skip connections ensure adequate feature flow. Adaptive weighted channel attention is proposed to make the network focus on representative channels. In addition, the PReLU replaces the ReLU activation function to increase the nonlinear expressiveness.
• HSCNN [15]: This network consists of three stages to perform the SR task. It first applies 1 × 1 convolution for feature extraction; then, it stacks multiple densely connected blocks with a convolution kernel size of 3 × 3 for feature mapping. Finally, the 1 × 1 convolution is used for spectral reconstruction.
• SSJSR [45]: This method contains two sub-networks, namely, spatial super-resolution sub-network and SR sub-network. In this experiment, only the SR sub-network is adopted for comparison. This spectral sub-network consists of five 3-D convolutional layers with a kernel size of $3\times 3\times 3$, and the network ends with a sub-pixel convolutional layer with a kernel size of $3\times 1\times 1$ to reconstruct the HSI.
• SSRAN [31]: This network utilizes three identical dual-branch residual blocks to extract spatial and spectral features of MSI, simultaneously. The bottom branch employs two 2-D convolutional layers with a kernel size of $1\times 1$ to extract the spectral information. Two 2-D convolutional layers with a kernel size of $3\times 3$ are used for the top branch to extract the spatial information. Similarly, the neighboring spectral attention is introduced at the end of each block.
• MST++ [39]: This network mainly consists of several single-stage spectral-wise Transformer (SST) modules. Each SST adopts a U-shaped structure consisting of an encoder and a decoder to extract multi-resolution contextual information. The fundamental unit of both the encoder and decoder is the spectral-wise attention (SA) block. Unlike conventional Transformer architectures, the SA block introduces spectral-wise multi-head self-attention to replace standard self-attention, reducing computational complexity while preserving performance.
• RepCPSI [23]: This network mainly consists of several polymorphic residual context restructuring (PRCR) modules. The PRCR module is similar to a basic residual structure, with the difference that it uses polymorphic convolution to extract features and that it equips a lightweight coordinate-preserving proximity spectral-aware attention block to enhance representational capabilities.
• MSFN [40]: Similar to MST++, the main structure of this network mainly consists of multiple single-stage spatial–spectral fusion networks (SSFNs). SSFN also has the structure of the U-Net to extract multiscale features. In addition, it uses the codec of Swin Transformer as the basic unit for feature extraction.

4.3. Evaluation Metrics

To objectively evaluate the quality of the proposed MS2Net-reconstructed HSI, five evaluation metrics were utilized in our experiments. The first metric is the root mean square error (RMSE) [46], which measures the estimated error between the reconstructed HSI $\widehat{\mathbf{X}}\in {\mathbb{R}}^{H\times W\times B}$ and the ground-truth HSI $\mathbf{X}\in {\mathbb{R}}^{H\times W\times B}$. The RMSE of a single band can be calculated as follows:

$$\mathrm{RMSE}\left({\mathbf{X}}^i,{\widehat{\mathbf{X}}}^i\right)=\sqrt{{\displaystyle \frac{|{\mathbf{X}}^i·{\widehat{\mathbf{X}}}^i{|}_F^2}{HW}}}$$

(14)

where ${\mathbf{X}}^i$ and ${\widehat{\mathbf{X}}}^i$ represent the ith band reconstructed image and ground-truth image, respectively, and they are scaled to the range [0, 255]. The average RMSE of all bands is used as the final result. A smaller RMSE value means that a better reconstruction result is obtained.

The second metric is the peak signal-to-noise ratio (PSNR) [47], which measures the pixel difference between the $\mathbf{X}$ and $\widehat{\mathbf{X}}$. The PSNR of a single band can be defined as follows:

$$\mathrm{PSNR}\left({\mathbf{X}}^i,{\widehat{\mathbf{X}}}^i\right)=20·{\log }_{10}({\displaystyle \frac{255}{\mathrm{RMSE}}})$$

(15)

The average PSNR of all bands is considered the final result. A large PSNR means that the reconstructed HSI spatial information is less distorted.

The third metric is the spectral angle mapper (SAM) [48], which evaluates the similarity in shape between two spectral vectors in degrees. The spectral similarity between the reconstructed pixel ${\widehat{X}}_j^i$ and ground-truth pixel $X_j^i$ is given by

$$\mathrm{SAM}\left({\mathbf{X}}_j^i,{\widehat{\mathbf{X}}}_j^i\right)=\arccos \left({\displaystyle \frac{{\widehat{\mathbf{X}}}_j^i{\mathbf{X}}_j^i}{\parallel {\widehat{\mathbf{X}}}_j^i{\parallel }_2\parallel {\mathbf{X}}_j^i{\parallel }_2}}\right)$$

(16)

The final SAM result is obtained by calculating the average value of the whole spatial domain pixels. If the angle is 0, there is no spectral change.

The fourth metric is the relative global dimensional error in synthesis (ERGAS) [49], which reflects the overall quality of the reconstructed HSI. The ERGAS of a single band is computed as follows:

$$\mathrm{ERGAS}\left({\mathbf{X}}^i,{\widehat{\mathbf{X}}}^i\right)=100\sqrt{{\left({\displaystyle \frac{\mathrm{RMSE}({\mathbf{X}}^i,{\widehat{\mathbf{X}}}^i)}{{\rho }_{{\widehat{\mathbf{X}}}^i}}}\right)}^2}$$

(17)

where ${\rho }_{{\widehat{\mathbf{X}}}^i}$ indicates the mean value of ${\widehat{\mathbf{X}}}^i$, and the RMSE$({\mathbf{X}}^i,{\widehat{\mathbf{X}}}^i)$ is the mean value of RMSE. The average ERGAS of all bands is considered the final result. A smaller ERGAS value means that a high-quality HSI is reconstructed.

The last metric is the structural similarity index (SSIM) [50], which is an improvement of the universal image quality index. The SSIM of a single band is shown in the following:

$$\mathrm{SSIM}\left({\mathbf{X}}^i,{\widehat{\mathbf{X}}}^i\right)={\displaystyle \frac{\left(2{\rho }_{{\mathbf{x}}^i}{\rho }_{{\widehat{\mathbf{x}}}^i}+c_1\right)\left(2{\sigma }_{{\mathbf{x}}^i{\widehat{\mathbf{x}}}^i}+c_2\right)}{\left({\rho }_{{\mathbf{x}}^i}^2+{\rho }_{{\widehat{\mathbf{x}}}^i}^2+c_1\right)\left({\sigma }_{{\mathbf{x}}^i}^2+{\sigma }_{{\widehat{\mathbf{x}}}^i}^2+c_2\right)}}$$

(18)

where ${\rho }_{{\mathbf{X}}^i}$ and ${\rho }_{{\widehat{\mathbf{X}}}^i}$ are the average value of the image ${\mathbf{X}}^i$ and ${\widehat{\mathbf{X}}}^i$, respectively; ${\sigma }_{{\mathbf{x}}^i}^2$ and ${\sigma }_{{\widehat{\mathbf{x}}}^i}^2$ are the standard deviations of the image ${\mathbf{X}}^i$ and ${\widehat{\mathbf{X}}}^i$, respectively; and $c_1$ and $c_2$ are constants. A SSIM value close to 1 implies good reconstructed quality.

4.4. Experimental Results and Analysis

To verify the superiority of the proposed method at all levels, we quantitatively and qualitatively evaluated the reconstruction results of the proposed method with eight comparative methods on four datasets. First, as shown in

Table 3. Reconstruction performance comparison of different methods on the IP dataset.

Methods	Indian Pines Dataset
	RMSE (↓ 0)	PSNR (↑ ∞)	SAM (↓ 0)	SSIM (↑ 1)	EGRAS (↓ 0)
J-SLoL [9]	2.927	38.374	2.071	0.961	3.404
AWAN [27]	2.864	44.936	1.929	0.955	3.452
HSCNN [15]	2.374	47.246	1.659	0.969	2.866
SSJSR [45]	2.261	48.785	1.573	0.971	2.673
SSRAN [31]	2.701	45.629	1.915	0.962	3.336
MST++ [39]	2.088	49.148	1.453	0.975	2.474
RepCPSI [23]	2.616	44.710	1.849	0.959	3.511
MSFN [40]	2.025	49.635	1.410	0.977	2.385
MS2Net	1.542	52.642	1.045	0.986	1.823

The best results in each column are highlighted in bold, and the second-best results are underlined.

, we quantitatively assessed the reconstruction results on the IP dataset. Generally, the proposed MS2Net achieves the best reconstruction results in five evaluation metrics compared with other methods. Concretely, the reconstruction accuracy obtained by DL-based methods is much higher than that obtained by sparse representation-based methods (i.e., J-SLoL) in terms of PSNR since the DL-based method does not destroy the spatial information of the MSIs. Furthermore, compared with the second-best DL-based method (i.e., MSFN), the proposed MS2Net decreases RMSE by 23.85%, SAM by 23.74%, and EGRAS by 32.07%. As for PSNR and SSIM, the proposed MS2Net is 5.71% and 0.91% better, respectively, in these two metrics than the second-best method. Additionally, the proposed MS2Net shares similarities with the SSRAN network structure, as both are designed to simultaneously extract spectral and spatial information. However, due to the simple structure of the SSRAN, its effectiveness needs to be further improved. In contrast, given the well-designed network architecture, the proposed MS2Net achieves the desired results in all metrics.

To visually demonstrate the reconstruction performance, the error maps of different methods for five representative sampling bands (i.e., 450 nm, 550 nm, 650 nm, 806 nm, and 2370 nm) on the IP dataset are shown in Figure 5. As shown in Figure 5, the proposed MS2Net achieves the minimum reconstruction error in all five sampling bands. Specifically, all methods are poorly reconstructed in the 650 nm in the IP dataset because this band is at the edge of the MSI spectral response function and provides limited spectral information for the reconstruction task. Moreover, each method has different reconstruction results for different land-cover types. For example, the land-cover type of the red rectangle is the soybean–mintill, with a smooth and uniform spatial distribution. Due to the spatial information disrupted by J-SLoL, the edge texture information is lost in the reconstruction result of this region. On the contrary, the reconstruction error in this region is relatively small for those DL-based methods due to their consideration of spatial features. In addition, the green rectangle contains a mixture of land-cover types, such as wood and other vegetation. All comparative methods have poor reconstruction results in this region. Due to the elaborate multiscale feature extraction module, the proposed MS2Net significantly outperforms the other comparison methods in this region on all five sampling bands.

Second, the reconstruction results of different methods on the PU dataset are shown in

Table 4. Reconstruction performance comparison of different methods on the PU dataset.

Methods	Pavia University Dataset
	RMSE (↓ 0)	PSNR (↑ ∞)	SAM (↓ 0)	SSIM (↑ 1)	EGRAS (↓ 0)
J-SLoL [9]	3.247	42.357	3.647	0.975	10.456
AWAN [27]	2.381	41.819	2.690	0.979	7.392
HSCNN [15]	2.049	43.648	2.418	0.982	6.480
SSJSR [45]	2.269	42.472	2.593	0.980	7.149
SSRAN [31]	2.557	41.045	2.864	0.977	7.732
MST++ [39]	2.121	43.058	2.448	0.981	6.581
RepCPSI [23]	2.159	42.947	2.540	0.981	6.645
MSFN [40]	2.357	41.773	2.665	0.979	7.224
MS2Net	1.827	45.275	2.178	0.984	5.819

The best results in each column are highlighted in bold, and the second-best results are underlined.

. Due to the limited spectral information remaining after the spectral downsampling of the PU dataset, the reconstruction results of all methods are degraded in all five metrics compared to the IP dataset. However, the proposed MS2Net still leads in all metrics. Specifically, compared with the second-best method (i.e., HSCNN), the reconstruction results of the proposed MS2Net decreased by 10.83%, 9.92%, and 10.20% for RMSE, SAM, and EGRAS, respectively. Furthermore, regarding PSNR and SSIM, the proposed MS2Net improves by 3.72% and 0.2%, respectively. More visually, the four representative sampling bands error maps of different methods (i.e., 450 nm, 550 nm, 650 nm, and 806 nm) on the PU dataset are shown in Figure 6. As illustrated in Figure 6, the proposed MS2Net reconstructs HSI more accurately in all four representative bands and shows higher fidelity than other methods. Specifically, the reconstruction error is minor for land-cover types with continuous spatial distribution, such as roads and artificial buildings. In contrast, reconstructing sparsely distributed land-cover types, such as trees and roofs, is more challenging. Nevertheless, the MS2Net achieves outstanding reconstruction accuracy even for these land-cover types.

Third, we validated all methods on the real-world datasets. The CQ dataset has a GSD of 30 m, meaning the individual pixels contain various land-cover types. This results in a complex spectral signature recorded by a single pixel. These complex spectral signatures pose a significant challenge to the reconstruction task. The reconstruction results of different methods on the CQ dataset are listed in

Table 5. Reconstruction performance comparison of different methods on the CQ dataset.

Methods	Chongqing Dataset
	RMSE (↓ 0)	PSNR (↑ ∞)	SAM (↓ 0)	SSIM (↑ 1)	EGRAS (↓ 0)
J-SLoL [9]	11.217	27.973	6.862	0.582	29.763
AWAN [27]	2.673	40.734	2.452	0.974	7.361
HSCNN [15]	2.614	40.895	2.337	0.973	7.056
SSJSR [45]	2.748	40.619	2.495	0.972	7.449
SSRAN [31]	3.016	39.390	2.865	0.966	9.036
MST++ [39]	3.083	39.530	2.485	0.969	7.709
RepCPSI [23]	2.572	41.341	2.324	0.976	6.544
MSFN [40]	3.133	39.182	2.655	0.967	8.346
MS2Net	2.211	42.535	1.922	0.983	5.670

The best results in each column are highlighted in bold, and the second-best results are underlined.

. From Table 5, we can learn that J-SLoL achieves the worst reconstruction accuracy. Other DL-based methods achieved acceptable reconstruction results. The RepCPSI method achieves the second-best results due to its deeper network layers. In particular, compared with the RepCPSI, the reconstruction results of the proposed MS2Net decreased by 14.03%, 17.29%, and 13.35% for RMSE, SAM, and EGRAS, respectively. Furthermore, regarding PSNR and SSIM, the proposed MS2Net performs 2.81% and 0.71% better, respectively. Visually, the four representative sampling bands error maps of different methods (i.e., 359 nm, 524 nm, 670 nm, and 962 nm) on the CQ dataset are displayed in Figure 7. As expected, the proposed MS2Net has a good reconstruction result and a smaller reconstruction error than other methods.

To further validate the robustness of the proposed method, we finally conducted experiments on the recently released JX dataset.

Table 6. Reconstruction performance comparison of different methods on the JX dataset.

Methods	Jiaxing Dataset
	RMSE (↓ 0)	PSNR (↑ ∞)	SAM (↓ 0)	SSIM (↑ 1)	EGRAS (↓ 0)
J-SLoL [9]	6.411	33.879	3.574	0.896	18.337
AWAN [27]	3.403	39.962	1.310	0.963	7.123
HSCNN [15]	3.033	41.645	1.136	0.972	5.413
SSJSR [45]	3.104	41.249	1.215	0.969	5.919
SSRAN [31]	3.154	40.875	1.283	0.969	6.360
MST++ [39]	3.090	42.196	1.028	0.968	6.221
RepCPSI [23]	2.921	41.275	1.008	0.971	6.585
MSFN [40]	3.172	40.223	1.151	0.964	7.922
MS2Net	2.209	43.932	0.737	0.984	4.443

The best results in each column are highlighted in bold, and the second-best results are underlined.

presents the reconstruction results of various methods on the JX dataset. Generally, we can observe from Table 6 that the proposed MS2Net consistently outperforms other methods across all evaluation metrics. Specifically, the proposed MS2Net achieves the lowest RMSE of 2.209, which is 24.37% lower than the second-best method (RepCPSI, 2.921). It also attains the highest PSNR of 43.932, representing a 4.11% improvement over MST++ (42.196). For SAM, the proposed MS2Net achieves a value of 0.737, which is 26.88% lower than RepCPSI (1.008), indicating significantly reduced spectral distortion. In terms of SSIM, the proposed MS2Net achieves 0.984, slightly surpassing HSCNN (0.972) by 1.23%. Finally, the lowest EGRAS is achieved by the proposed MS2Net with 4.443, which is a 17.92% improvement over HSCNN (5.413). Figure 8 presents the error maps of four representative sampling bands (395 nm, 550 nm, 1200 nm, and 2200 nm) for different methods on the JX dataset. As shown in Figure 8, the proposed MS2Net achieves more accurate HSI reconstruction across all four bands, demonstrating superior fidelity compared to the other methods. Moreover, Figure 9, Figure 10, Figure 11 and Figure 12 show the spectral curves of three typical land-cover types randomly selected for four datasets. Evidently, the reconstructed spectral curves of the proposed MS2Net on the three data sets are closer to the ground-truth spectral responses.

5. Discussion

5.1. Analysis of Ablation Experiments

In this subsection, we conduct extensive ablation experiments across four different datasets to evaluate the effectiveness of each module in the proposed MS2Net. Specifically, “W/O 3-D MHR” refers to MS2Net without the 3-D MHR module, “W/O RCSA” indicates the removal of the RCSA module, “W/O Spe-RCSA” denotes the exclusion of the spectral branch RCSA module, and “W/O Spa-RCSA” represents the absence of the spatial branch RCSA module. Additionally, “W/O Spe-Net” refers to MS2Net without the spectral branch, “W/O Spa-Net” excludes the spatial branch, and “W/O AWFF” indicates the absence of the AWFF module. Quantitative results of different ingredients of the proposed MS2Net on four datasets are presented in

Table 7. Ablation experimental results of different modules in the proposed MS2Net on the IP dataset.

Models	Indian Pines Dataset
	RMSE (↓ 0)	PSNR (↑ ∞)	SAM (↓ 0)	SSIM (↑ 1)	EGRAS (↓ 0)
W/O 3-D MHR	1.668	52.080	1.139	0.984	1.964
W/O RCSA	1.723	50.625	1.192	0.982	2.078
W/O Spe-RCSA	1.612	52.392	1.102	0.985	1.896
W/O Spa-RCSA	1.604	52.561	1.093	0.985	1.912
W/O Spe-Net	1.656	52.015	1.143	0.984	1.946
W/O Spa-Net	1.558	52.371	1.060	0.985	1.850
W/O AWFF	1.551	52.355	1.055	0.986	1.849
MS2Net	1.542	52.642	1.045	0.986	1.823

The best results in each column are highlighted in bold, and the second-best results are underlined.

,

Table 8. Ablation experimental results of different modules in the proposed MS2Net on the PU dataset.

Models	Pavia University Dataset
	RMSE (↓ 0)	PSNR (↑ ∞)	SAM (↓ 0)	SSIM (↑ 1)	EGRAS (↓ 0)
W/O 3-D MHR	1.841	45.216	2.180	0.984	5.849
W/O RCSA	1.866	44.976	2.236	0.984	5.898
W/O Spe-RCSA	1.835	45.231	2.192	0.984	5.866
W/O Spa-RCSA	1.833	45.238	2.193	0.984	5.832
W/O Spe-Net	1.868	44.915	2.214	0.984	5.941
W/O Spa-Net	1.864	45.017	2.227	0.984	5.913
W/O AWFF	1.859	45.040	2.211	0.984	5.914
MS2Net	1.827	45.275	2.178	0.984	5.819

The best results in each column are highlighted in bold, and the second-best results are underlined.

,

Table 9. Ablation experimental results of different modules in the proposed MS2Net on the CQ dataset.

Models	Chongqing Dataset
	RMSE (↓ 0)	PSNR (↑ ∞)	SAM (↓ 0)	SSIM (↑ 1)	EGRAS (↓ 0)
W/O 3-D MHR	2.729	40.624	2.534	0.971	7.049
W/O RCSA	2.383	41.786	2.189	0.978	6.293
W/O Spe-RCSA	2.725	40.806	2.519	0.972	6.966
W/O Spa-RCSA	2.665	40.679	2.259	0.976	6.956
W/O Spe-Net	2.844	40.454	2.611	0.969	7.151
W/O Spa-Net	3.024	39.802	2.816	0.964	7.950
W/O AWFF	2.611	41.082	2.338	0.975	6.616
MS2Net	2.211	42.535	1.922	0.983	5.670

The best results in each column are highlighted in bold, and the second-best results are underlined.

and

Table 10. Ablation experimental results of different modules in the proposed MS2Net on the JX dataset.

Models	Jiaxing Dataset
	RMSE (↓ 0)	PSNR (↑ ∞)	SAM (↓ 0)	SSIM (↑ 1)	EGRAS (↓ 0)
W/O 3-D MHR	3.037	42.081	0.907	0.971	5.248
W/O RCSA	2.386	43.143	0.980	0.979	4.932
W/O Spe-RCSA	3.063	41.372	1.155	0.970	5.739
W/O Spa-RCSA	3.047	41.416	1.136	0.971	5.748
W/O Spe-Net	3.096	41.230	1.191	0.969	5.894
W/O Spa-Net	3.973	38.427	1.952	0.945	9.524
W/O AWFF	2.352	43.544	0.959	0.979	4.541
MS2Net	2.209	43.932	0.737	0.984	4.443

The best results in each column are highlighted in bold, and the second-best results are underlined.

.

First, we focus on the ablation results of the 3-D MHR module, i.e., W/O 3-D MHR. From the results of the ablation experiments of this module on four datasets, the reconstruction results are largely improved due to its ability to extract multiscale spatial–spectral features for reconstruction tasks. Particularly, the land-cover distribution is complex for the CQ and JX dataset and variable in scale. The quality of model reconstruction decreases significantly after removing the 3-D MHR module. Second, we pay attention to the results of the ablation experiments with the attention mechanisms. It is observed from the ablation experiments of the attention mechanism on all four datasets that the reconstruction results of the proposed MS2Net are reduced after removing the attention mechanism (i.e., W/O RCSA model). The W/O Spe-RCSA and W/O Spa-RCSA models perform better than the W/O RCSA models on the IP and PU datasets. However, for the CQ and JX dataset, the W/O RCSA model performs better than the W/O Spe-RCSA and W/O Spa-RCSA models. This indicates that only inserting attention mechanisms in the spectral or spatial branch for datasets with complex feature distribution does not achieve good reconstruction results. Third, we focus on the impact of each branch on the reconstruction results. The ablation results show that the influence of spatial and spectral features on the reconstruction results differs across datasets. For example, for the IP and PU datasets, the W/O Spa-Net model performs better than the W/O Spe-Net model. In contrast, for the CQ and JX datasets, the W/O Spe-Net model performs better than the W/O Spa-Net model. This is reasonable because the CQ and JX datasets have low spatial resolution (i.e., 30 m), and a single pixel in the image contains many mixed pixels, resulting in a relatively complex composition of single-pixel spectral signatures for these datasets. The above analysis shows that good reconstruction accuracy cannot be achieved using only a single spatial or spectral branch. Both spatial and spectral features need to be extracted for the reconstruction task to obtain satisfactory results. Furthermore, we can also observe from the ablation experiment results that, across different datasets, spectral and spatial information has varying degrees of influence on the reconstruction results. For this reason, we propose an AWFF module to fuse these two types of features. To verify the effectiveness, we performed ablation experiments on the AWFF module. The ablation experimental results show that the module can effectively aggregate two features and improve the reconstruction quality.

5.2. Analysis of Model Complexity

In this subsection, we compare the complexity of the models based on parameters, floating-point operations (FLOPs), and runtimes, all evaluated with a batch size of 1. The results are presented in

Table 11. Parameters (M), FLOPs (GMac), and runtime (ms) of deep learning methods on four datasets.

Datasets	Metrics	AWAN [27]	HSCNN [15]	SSJSR [45]	SSRAN [31]	MST++ [39]	RepCPSI [23]	MSFN [40]	MS2Net
IP	Parameters (M)	1.971	1.986	1.116	0.286	52.985	2.221	121.553	0.846
	FLOPs (GMac)	0.049	0.050	0.174	0.007	0.741	0.055	1.540	0.160
	Runtime (ms)	7.280	10.946	0.983	3.326	16.592	11.347	55.421	4.786
PU	Parameters (M)	1.844	1.910	0.507	0.278	11.675	2.130	26.761	0.655
	FLOPs (GMac)	0.046	0.048	0.047	0.007	0.164	0.053	0.340	0.053
	Runtime (ms)	6.953	10.513	0.955	3.290	16.024	10.559	53.082	4.634
CQ	Parameters (M)	1.839	1.907	0.533	0.278	9.737	2.127	22.310	0.734
	FLOPs (GMac)	0.046	0.048	0.094	0.007	0.137	0.053	0.284	0.100
	Runtime (ms)	6.867	10.107	0.912	3.266	16.133	10.057	51.390	4.594
JX	Parameters (M)	1.822	1.896	0.505	0.277	6.395	2.114	14.617	0.732
	FLOPs (GMac)	0.045	0.047	0.093	0.007	0.091	0.053	0.186	0.100
	Runtime (ms)	6.695	10.002	0.883	3.228	15.960	9.967	50.643	4.473

The highest and second-highest values in each row are highlighted in bold and underlined, respectively.

. As shown in Table 11, the AWAN and HSCNN methods adopt deep network structures, resulting in a higher number of parameters and FLOPs across all datasets. Specifically, AWAN has nearly 2M parameters and relatively low FLOPs, while HSCNN uses slightly more parameters but achieves comparable FLOPs. In contrast, the SSJSR method employs a simplified network architecture with only five 3-D convolution layers, which significantly reduces its parameters. However, this simplicity comes at the cost of higher FLOPs due to the increased computational demands of 3-D operations. Meanwhile, the SSRAN method has the lowest overall complexity, as it leverages lightweight 1-D and 2-D convolution operations to achieve minimal parameters, FLOPs, and runtime. Although RepCPSI is a CNN-based lightweight SR scheme, it introduces a new convolutional operator that is slightly more complex and thus has higher parameters and FLOPs than SSRAN. In contrast, the proposed MS2Net incorporates advanced operations that enhance feature extraction and representation while maintaining a manageable level of complexity. This allows the proposed MS2Net to achieve significantly better performance than both SSRAN and RepCPSI, with only a slight increase in computational demand. Furthermore, the recently introduced MST++ and MSFN methods, though designed as lightweight Transformer-based architectures, still exhibit relatively high complexity. Take the IP dataset as an example: compared to MSFN, the proposed MS2Net achieves reductions of 99.30% in parameters, 89.61% in FLOPs, and 91.36% in runtime, respectively, demonstrating its significant computational efficiency while maintaining strong performance. Based on the above analysis, it is evident that the proposed MS2Net strikes an optimal balance between computational efficiency and performance. The convergence of the proposed MS2Net on four datasets is shown in Figure 13. We can observe from Figure 13 that the proposed MS2Net achieves stable and efficient convergence across all datasets. In the initial epoch, the loss decreases rapidly, which indicates that the proposed MS2Net effectively learns the underlying features. As the epoch increases, the convergence becomes smooth and stable, which reflects the robustness of the network and its good generalization ability.

5.3. Analysis of Classification Validation

In this subsection, we design a classification experiment on two datasets with labeled categories to verify the quality of HSI reconstructed by different models. The SVM is selected as the basic classifier. The quantitative classification results for the IP and PU datasets are presented in

Table 12. Classification results of all competing SR methods on IP dataset.

Class	Color	J-SLoL [9]	AWAN [27]	HSCNN [15]	SSJSR [45]	SSRAN [31]	MST++ [39]	RepCPSI [23]	MSFN [40]	MS2Net	R-HSI
C1		70.87	82.72	88.44	94.22	87.11	93.64	83.61	91.83	90.66	87.55
C2		0.83	28.35	54.09	57.78	43.44	45.26	48.37	73.60	56.96	61.06
C3		86.69	95.36	92.63	87.25	90.08	93.31	82.58	81.50	92.63	92.63
C4		99.01	91.47	90.11	95.52	90.58	95.15	96.22	96.31	98.35	97.41
C5		0	92.86	100	66.66	86.66	66.66	42.85	40.00	93.33	86.66
C6		0	62.11	47.91	52.08	42.70	76.84	59.57	67.67	55.20	56.25
C7		96.77	92.04	94.82	93.93	95.22	93.25	95.02	92.15	95.46	96.11
C8		94.74	96.77	92.10	97.89	90.52	98.41	98.38	98.97	98.42	98.42
C9		96.23	88.00	90.69	91.19	92.67	95.30	94.30	94.00	95.16	98.63
OA (%)		84.73	85.94	88.57	89.38	87.88	90.44	88.98	90.26	91.61	92.59
AA (%)		60.57	81.07	83.42	81.83	79.89	84.20	77.88	81.78	86.24	86.08
k × 100		80.03	82.16	85.52	86.55	84.60	87.90	85.91	87.71	89.36	90.59

The best results in each column are highlighted in bold, and the second-best results are underlined.

and

Table 13. Classification results of all competing SR methods on PU dataset.

Class	Color	J-SLoL [9]	AWAN [27]	HSCNN [15]	SSJSR [45]	SSRAN [31]	MST++ [39]	RepCPSI [23]	MSFN [40]	MS2Net	R-HSI
C1		83.31	85.68	81.92	88.28	83.35	85.02	86.11	84.97	88.12	92.00
C2		97.18	97.38	98.13	97.38	97.38	97.54	97.24	97.77	97.51	98.59
C3		75.24	80.03	74.20	83.03	76.22	81.05	81.22	78.27	81.24	89.29
C4		74.93	80.49	88.05	86.30	83.21	89.68	85.93	86.63	88.72	93.92
C5		63.31	60.50	66.83	59.08	65.09	75.05	70.67	71.00	74.22	90.66
C6		84.88	86.57	85.60	83.80	83.68	68.41	71.84	67.25	88.11	92.53
C7		77.92	86.39	85.71	81.12	83.29	83.17	83.54	69.29	81.06	87.59
C8		99.23	99.69	100	99.84	99.86	99.83	99.47	99.84	99.85	99.69
OA (%)		86.47	87.84	88.48	88.01	87.66	89.25	88.67	87.39	90.22	94.82
AA (%)		82.01	84.59	85.05	84.86	84.00	84.97	84.50	81.88	87.35	93.03
k × 100		80.87	82.81	83.78	83.11	82.61	84.97	84.09	82.33	86.30	92.82

The best results in each column are highlighted in bold, and the second-best results are underlined.

, respectively, where accuracy, overall accuracy (OA), average accuracy (AA), and kappa coefficient (k) are reported. Generally, the proposed MS2Net achieves the classification accuracy closest to the real HSI (R-HSI) on the two datasets. More visually, the full-pixel classification maps for the two datasets with different methods are shown in Figure 14 and Figure 15. From Figure 14 and Figure 15, it can be seen that the classification map of MS2Net is more similar to the R-HSI classification map. Specifically, taking the IP dataset as an example, it can be seen from Figure 14 that the proposed MS2Net has a smoother classification performance for the region of class 9 compared to other methods. Furthermore, for the green rectangular region in Figure 14, the classification performance of HSI reconstructed by all methods is noisy in this region due to its inclusion of feature mixing. In contrast, the proposed MS2Net outperforms other comparative methods for classification performance in this region.

6. Conclusions

In this paper, we proposed a novel MS2Net framework to achieve the SR task from MSIs. Specifically, the 3-D MHR module was utilized to extract the multiscale spectral–spatial features of MSIs. To further enhance feature extraction, we designed a dual-branch structure, where each branch utilized SSDR modules with different convolutional kernel sizes to effectively capture spatial and spectral features. At the same time, we also introduced the attention mechanism, i.e., RCSA module, in each branch to direct the network to focus on the features of interest and to enhance its ability to capture global features. To further optimize the reconstruction quality, we proposed an AWFF module to ensure effective fusion of features extracted from the two branches. A series of ablation experiments verified the effectiveness of each module in the proposed MS2Net. Moreover, the classification validation experiments further confirmed the usability of the proposed MS2Net-reconstructed HSIs.

Although the proposed MS2Net has demonstrated excellent SR capabilities, there is still room for improvement in terms of model complexity. Furthermore, the reconstruction performance of the proposed MS2Net may be affected when applied to complex geographical features in different scenarios (e.g., diverse terrain, vegetation, or human-made objects). Consequently, in future work, we aim to explore the following aspects. On the one hand, a cross-modal self-supervised learning framework can be investigated, such as combining multiple remote sensing data types (e.g., MSI, LiDAR, SAR, etc.) for representation learning and leveraging the complementarity between different modalities to enhance SR performance. On the other hand, the exploration of lightweight models represents a promising direction for practical deployment, particularly in scenarios with limited computational resources.