Hyperspectral Image Super-Resolution Method Based on Spectral Smoothing Prior and Tensor Tubal Row-Sparse Representation

Abstract

Due to the limited hardware conditions, hyperspectral image (HSI) has a low spatial resolution, while multispectral image (MSI) can gain higher spatial resolution. Therefore, derived from the idea of fusion, we reconstructed HSI with high spatial resolution and spectral resolution from HSI and MSI and put forward an HSI Super-Resolution model based on Spectral Smoothing prior and Tensor tubal row-sparse representation, termed SSTSR. Foremost, nonlocal priors are applied to refine the super-resolution task into reconstructing each nonlocal clustering tensor. Then per nonlocal cluster tensor is decomposed into two sub tensors under the tensor t-prodcut framework, one sub-tensor is called tersor dictionary and the other is called tensor coefficient. Meanwhile, in the process of dictionary learning and sparse coding, spectral smoothing constraint is imposed on the tensor dictionary, and $L_{1,1,2}$ norm based tubal row-sparse regularizer is enforced on the tensor coefficient to enhance the structured sparsity. With this model, the spatial similarity and spectral similarity of the nonlocal cluster tensor are fully utilized. Finally, the alternating direction method of multipliers (ADMM) was employed to optimize the solution of our method. Experiments on three simulated datasets and one real dataset show that our approach is superior to many advanced HSI super-resolution methods.

1. Introduction

Hyperspectral image (HSI), as a three-dimensional data cube with two spatial dimensions and one spectral dimension, contains rich spatial and spectral information of ground objects. Due to this advantage, it has been widely used in many fields such as denoising [1], unmixing [2], classification [3,4], and object detection [5]. However, limited by the hardware conditions of the sensor, there is a trade-off between the spatial resolution and spectral resolution of HSIs. That is, HSIs usually have lower spatial resolution and higher spectral resolution, while multispectral images (MSIs) have lower spectral resolution and higher spatial resolution. Therefore, it is urgent and necessary to fuse the information of HSI and MSI to obtain an HSI with higher spatial and spectral resolutions, thereby improving the accuracy of practical applications.

In recent decades, scholars have proposed many methods for hyperspectral and multispectral fusion. According to the characteristics and theoretical basis of HSI super-resolution, those related work can be divided into four categories: the methods based on pan-sharpening, the methods based on matrix decomposition, the methods based on deep learning, and the methods based on tensor decomposition.

1.1. Fusion Based on Pan-Sharpening

The early fusion method is to fuse the lower-resolution MSI with the higher-resolution panchromatic (PAN) image, called pansharpening. The typical techniques include component projection-substitution (CP) [6] and multiresolution analysis (MRA) [7]. CP methods include intensity hue saturation (IHS) transformation [8], principal component analysis (PCA) transformation [9] and Gram–Schmidt (GS) transformation [10]. The main idea of those methods is to replace the component containing spatial information in PAN image with the same element for MSI. The representative methods based on MRA include wavelet transform [11], Laplacian pyramid (LP) [12]. The main idea is to inject the high-resolution spatial structure particulars of the PAN into the MSI. Inspired by this, many methods extend the pan-sharpening way to HSI and MSI fusion, such as hypersharpening [13,14]. However, If there is a large gap between the spatial resolution of HSIs and MSIs, the fusion results will be accompanied by varying degrees of distortion.

1.2. Fusion Based on Matrix Decomposition

Based on the matrix decomposition, it is assumed that the matrix of a high-resolution HSI (HR-HSI) expanded along spectral dimension can be decomposed into the product of a spectral basis matrix and a coefficient matrix, and the fusion problem is transformed into the estimation of a spectral basis matrix and a coefficient matrix. Specifically, based on the linear mixed spectral model [15], Yokoya et al. [16] proposed a coupled non-negative matrix decomposition method to alternately update the hyperspectral endmember and the abundance matrix with high spatial resolution. Having the idea that there is a strong correlation between bands of an HSI, Simões et al. [17] estimated spectral features in a low-dimensional subspace by using the low-rank feature between bands of the HSI, thus improving the fusion accuracy and efficiency. Similarly, Zhang et al. [18] fully explored the multi-manifold structure and low-rank structure of spectral bands, and proposed an method based on fusion on group spectral embedding (GSE). More recently, the sparse representation methods have shown good capabilities in estimating the spectral basis matrix and coefficient matrix to obtain the desired results. The work of [19,20,21,22,23,24,25,26,27] regarded spectral basis as an overcomplete dictionary to obtain sparsity, and enforced sparsity constraints to the coefficient matrix. Akhtar et al. [19] proposed a Bayesian sparse coding method. Dong et al. [20] proposed a non-negative structure sparse representation method, and utilized the clustering sparsity of HSI for fusion task. Later, Xue et al. [26] proposed a structured sparse representation model by utilizing higher-level spectrum and spatial prior. In general, the premise of matrix decomposition-based methods is to expand HR-MSI and LR-HSI into matrices along spectral dimensions, which often ignore the original structure of hyperspectral images. Although the accuracies are improved compared to pansharpening methods, however, matrix factorization-based methods have higher computational complexity and parameter settings.

1.3. Fusion Based on Deep Learning

In recent years, deep learning has made major breakthroughs in the application of image processing. Dong et al. [28] applied convolutional neural networks (CNN) to image super-resolution for the first time. Since then, a lot of work has been carried out around deep learning methods. For instance, Palsson et al. [29] proposed a 3D convolutional network-based fusion algorithm, which regarded the fusion problem as a spectral super-resolution task of multispectral images. First, the down-sampled multispectral image and the original hyperspectral image were used to learn between the images. Then the learned model was utilized to complete the super-resolution of MSI. Zhang et al. [30] focused more on reconstructing the missing spatial and spectral information of the image, and proposed SSR-NET based on the spatial and spectral edge loss. Yang et al. [31] established a two-branch convolutional neural network architecture to extract multispectral images and hyperspectral images separately, and combine features to achieve super-resolution goals. Compared with traditional machine learning methods that rely on the design of strong handcrafted features, deep learning methods are data-driven, and directly learn prior knowledge from a large amount of data. For example, Wei et al. [32] proposed a deep recursive network based on a deep structure prior and achieved excellent performance. Because the three-dimensional (3D) convolution operation can preserve the spatial-spectral correlation of the HSI better than the one-dimensional (1D) and two-dimensional (2D) convolution operations, Wang et al. [33] proposed to extract spectral and spatial information of source images through 1D convolution and 2D convolution while Hu et al. [34] proposed a multi-scale feature fusion aggregation network based on 3D convolution, which had better reconstruction results. In addition, the non-blind method relies on the fully known point spread function (PSF) and spectral response function (SRF) to simulate the degradation process. Some super-resolution methods based on deep learning can adaptively simulate these two functions. From this perspective, Zheng et al. [35] adaptively learn the parameters of PSF and SRF through two special convolutional layers and a self-encoding network. Inevitably, deep learning methods have some shortcomings. Compared with traditional method models, the neural network used in deep learning is poorly interpretable, and it relies on a large number of data sets for training, also requires high hardware conditions.

1.4. Fusion Based on Tensor Decomposition

Recently, tensor decomposition has rapidly emerged in the fields of high-dimensional image denoising, image completion, and compressed sensing. Since HSIs possess a three-dimensional tensor structure that integrates spatial and spectral information, tensor decomposition has become a hot spot for HSI super-resolution [36]. Li et al. [37] took the lead in applying Tucker decomposition to hyperspectral image fusion, alternately estimating dictionaries of the three modes and a core tensor. In addition, Dian et al. [38] added a non-local prior to estimate each non-local cluster, reducing the computational cost of each iteration, and also discussed the effect of image reconstruction under blind conditions. On this basis, Wan et al. [39] stacked each non-local cluster into a fourth-order tensor and discussed the cluster sparsity. In addition, CANDECOMP/PARAFAC decomposition (CPD) can also capture the dependencies between different dimensions, and Kanatsoulis et al. [40] utilized the coupled CPD model to handle the HSI-MSI fusion task. However, based on the fusion model of Tucker decomposition or CPD, although the size of each modal dictionary is obviously reduced, the interaction between spatial information and spectral information is also weakened by the matrixing operation in the decomposition process. Moreover, the tensor train decomposition model and tensor ring decomposition model have a more vital ability to mine the internal structure of the data. These two decomposition models can effectively maintain the inherent low-rank structure of the tensor while avoiding the loss of original information. For example, Dian et al. [41] applied a regular term to the tensor train decomposition factor and achieved good reconstruction results. Then, Dian et al. [42] proposed a method based on low-rank subspace to estimate the spectral dictionary and coefficients. Later, Xu et al. [43] proposed a non-convex rank constrained fusion model of tensor ring factors, which imposed tensor nuclear norm constraints on decomposition factors. Inspired by t-prodcut based tensor sparse representation [44], Xu et al. [45] proposed a new HSI super-resolution framework. However, the model proposed in [45] lacked some considerations of the inherent properties of HSIs. For example, it did not make full use of the essential characteristics of the tensor decomposition factor and only used the $L_1$ norm to solve the tensor coefficients, which would lead to loss of primary information and sub-optimal solutions in practical applications. In addition, the model adopted orthogonal constraints on the dictionary tensor, so that the spectrum of the linear combination of these base spectra did not satisfy the smooth property. Therefore, there are still too much room left for improvement.

In this paper, we focus more on the properties of the reconstructed image itself. Combining the characteristics of hyperspectral images, we carefully design the constraints of spectral smoothing and tubal row-sparse on the framework mentioned in [45] and propose a new HSI super-resolution model. In each iteration process, the reconstruction image is made closer to the structural properties of the image itself. The advantages of our method are mainly reflected in two aspects. First, we use a nonlocal patch-wise way to deal with the fusion problem, instead of directly processing it on the whole image. Second, we pay more attention to the internal structure and properties of the tensor factorization. On the one hand, we impose a spectral smoothing constraint on the tensor spectral dictionary, and on the other hand, we impose the $L_{1,1,2}$ norm [46] on the tensor coefficient factor to capture the intrinsic structure of it. Compared with current super-resolution methods, our contributions are as follows.

• We approach the fusion problem in a patch-wise way instead of directly processing the images. Specifically, to fully exploit the spatial self-similarity of HSI, we use a clustering method on the source image and construct multiple nonlocal tensor patches. On this basis, we apply the tensor sparse representation model to the reconstruction of each nonlocal tensor patch. In this way, the efficiency of our method is improved.
• Furthermore, based on the tensor sparse representation model, we focus more on the properties of hyperspectral images. To make the reconstructed image closer to the original properties of HSI, we impose a spectral smoothing constraint on the tensor dictionaries to promote the spectral smoothness of reconstructed images. Meanwhile, it was noted that we use the $L_{1,1,2}$ norm [46] to characterize the tubal row-sparsity exhibited by the coefficient tensor.
• We perform effective convex approximation for each term of the model and use ADMM [47] to optimize the solution of the model. Comparative experiments conducted on multiple simulated data sets and one real data set validate that the proposed method is superior to the current advanced competitors.

The remainder of this paper is organized as follows. Section 2 presents the basic notations with HSI super-resolution problem, as well as some of the related work, and our proposed method. Section 3 presents the optimization algorithm. In Section 4, we show the results of comparing our method with other methods in three simulated data sets and one real data set. In Section 5, we discuss parameter selection and the effectiveness and superiority of our designed constraints. Section 6 presents the conclusions.

2. Materials and Methods

2.1. Notions and Definitions

In this section, we will introduce the symbols and definitions [44] involved in this paper. It is important to note that we use lowercase letters for vectors, capital letters for matrices, and Euler Script letters for tensors.

Definition 1

(Mode-n unfolding). The mode-n unfolding matrix of the P-dimensional tensor $\mathcal{X} \in {\mathbb{R}}^{f_1\times f_2\times \cdots \times f_P}$ is denoted as $X_{\left(p\right)} \in {\mathbb{R}}^{f_p\times f_1{f}_2,\dots ,f_{p-1}{f}_{p+1},\dots ,f_P}$, which is also represented by $unfol{d}_p\left(\mathcal{X}\right) = X_{\left(p\right)}$, and $\mathcal{X} = fol{d}_p\left(X_{\left(p\right)}\right)$. It takes the $n_{th}$ dimension fiber as the column of the matrix, mapping the tensor elements into the matrix elements. Noted that $\mathcal{X}(:,i,j)$, $\mathcal{X}(i,:,j)$ and $\mathcal{X}(i,j,:)$ are the column, row, and tubal fibers of $\mathcal{X}$, respectively.

Definition 2

(Mode-n Tensor-Matrix Product). The mode-n tensor-matrix product of a tensor $\mathcal{X} \in {\mathbb{R}}^{I_1\times I_2\times }$${}^{\cdots \times I_N}$ and a matrix $\mathbf{A} \in {\mathbb{R}}^{P_n\times I_n}$ is a tensor $\mathcal{Y} \in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_{n-1}}$${}^{\times P_n\times I_{n+1}\times \dots \times I_N}$ denoted by $\mathcal{X}{\times }_{k}A$ and its elements are computed by

$${\left(\mathcal{X}{\times }_k\mathbf{A}\right)}_{i_1,\dots ,i_{k-1},j,i_{k+1},\dots ,i_N} = \sum _{i_k}{x}_{i_1\dots i_{k-1}{i}_k{i}_{k+1}\dots i_N}{a}_{p_n{i}_n}$$

(1)

Here, we have $\mathcal{Y} = \mathcal{X}{\times }_k\mathbf{A}\iff Y_{\left(k\right)} = \mathbf{A}{X}_{\left(k\right)}$.

Definition 3

(t-prodcut). The t-prodcut between $\mathcal{D} \in {\mathbb{R}}^{d_1\times d_2\times d_3}$ and $\mathcal{A} \in {\mathbb{R}}^{d_2\times d_4\times d_3}$ is $\mathcal{J} \in {\mathbb{R}}^{d_1\times d_4\times d_3}$, where $\mathcal{J}(i,j,:) = {\sum }_{m=1}^r\mathcal{D}(i,m,$$:) \ast \mathcal{A}(m,j,:)$, and ∗ represents a circular convolution operator. To better understand, we represent the t-prodcut using the following operator. First, we use $D^{\left(i\right)}$ to represent $\mathcal{D}(:,:,i)$, $i \in \left[d_3\right]$, each $D^{\left(i\right)}$ is a frontal slice of $\mathcal{D}$. Then, we define the block circular matrix blkcirc $\left(\mathcal{D}\right) \in {\mathbb{R}}^{d_1{d}_3\times d_2{d}_3}$ as

$$blkcirc\left(\mathcal{D}\right) = \left[\begin{array}{cccc}{D}^{\left(1\right)}& D^{\left(d_3\right)}& \cdots & D^{\left(2\right)}\\ D^{\left(2\right)}& D^{\left(1\right)}& \cdots & D^{\left(d_3\right)}\\ ⋮& ⋮& ⋮& ⋮\\ D^{\left(d_3\right)}& D^{\left(d_3-1\right)}& \cdots & D^{\left(1\right)}\end{array}\right]$$

(2)

Next, we define the operator of unfolding and folding of the frontal slices of $\mathcal{D}$ as

$$Unfold{}_V\left(\mathcal{D}\right) = \left[\begin{array}{c}{D}^{\left(1\right)}\\ D^{\left(2\right)}\\ ⋮\\ D^{\left(d_3\right)}\end{array}\right],Fold{}_V\left(Unfold{}_V\left(\mathcal{D}\right)\right) = \mathcal{D}$$

(3)

At this point, the t-prodcut between $\mathcal{D}$ and $\mathcal{A}$ is $\mathcal{J} = \mathcal{D}\ast \mathcal{A} = {Fold}_V\left(blkcirc\left(\mathcal{D}\right)·\right.$

$\left.{Unfold}_V\left(\mathcal{A}\right)\right) \in {\mathbb{R}}^{d_1\times d_4\times d_3}$. To solve the tensor-tensor product optimization problem quickly, we introduce a significant tool called the Discrete Fourier Transform (DFT). Let $\overline{\mathcal{D}} \in {\mathbb{R}}^{d_1\times d_2\times d_3}$ represents transformation form of DFT of $\mathcal{D} \in {\mathbb{R}}^{d_1\times d_2\times d_3}$ along the 3^rd mode, we define $\overline{D} \in {\mathbb{R}}^{d_1{d}_3\times d_2{d}_3}$ as follows:

$$\overline{D} = \mathit{blockdiag}\left(\overline{\mathcal{D}}\right) = \left[\begin{array}{cccc}{\overline{D}}^{\left(1\right)}\hfill & \hfill & & \\ \hfill & {\overline{D}}^{\left(2\right)}\hfill & \hfill & \\ \hfill & \hfill & \ddots \hfill & \\ \hfill & \hfill & & {\overline{D}}^{\left(d_3\right)}\hfill \end{array}\right]$$

(4)

The function of blockdiag $(·)$ operator above is mapping the tensor $\overline{\mathcal{D}}$ to the block matrix arranged diagonally. Then we will use the following Lemma to explain why we can use DFT to solve our problem quickly.

Lemma 1

([48]). Assume that $\mathcal{D} \in {\mathbb{R}}^{d_1\times d_2\times d_3}$, $\mathcal{A} \in {\mathbb{R}}^{d_2\times d_4\times d_3}$ are two random tensors. Let $\mathcal{J} = \mathcal{D} \ast \mathcal{A}$. Then, we introduce the theorem of property as follows.

• ${\parallel \mathcal{J}\parallel }_F^2 = \frac{1}{d_3}{\parallel \overline{J}\parallel }_F^2$
• $\mathcal{J} = \mathcal{D}\ast \mathcal{A}$is equivalent to$\overline{J} = \overline{D}\overline{A}$, thus, we have${\overline{J}}^{\left(j\right)} = {\overline{D}}^{\left(j\right)}{\overline{A}}^{\left(j\right)}$, $j\in \left[d_3\right]$

Definition 4

($L_{1,1,2}$ norm and tubal row-sparsity). For a tensor $\mathcal{A} \in {\mathbb{R}}^{d_1\times d_2\times d_3}$, we define $\mathcal{A}(i,j,:)$ as a tube of $\mathcal{A}$, and define the tubal row-sparse of $\mathcal{A}$ as the number of non-zero tubes of $\mathcal{A}$. Here, we use $L_{1,1,2}$ norm to compute the tubal sparsity [49]. The definition of $L_{1,1,2}$ norm is given as follows.

$${\parallel \mathcal{A}\parallel }_{1,1,2} = \sum _{i,j}{\parallel \mathcal{A}(i,j,:)\parallel }_F$$

(5)

which calculates the summation of the $L_2$ norm of all tubes of $\mathcal{A}$.

2.2. Preliminaries

2.2.1. Problem Formulation

To facilitate understanding, we build a preliminary observation model, and use Euler Script letters to represent two input images and one output image, respectively.

Through the fusion of LR-HSI $\mathcal{Y} \in {\mathbb{R}}^{w\times h\times C}$ and HR-MSI $\mathcal{Z} \in {\mathbb{R}}^{W\times H\times c}$, we obtain the target image HR-HSI $\mathcal{X} \in {\mathbb{R}}^{W\times H\times C}$, where W, w and H, h represent the dimensions of the spatial width and height, respectively. C and c represent the dimension of the spectral. It is noted that $w<<W$, $h<<H$ and $c<<C$. As mentioned at Definition 1, $X_{\left(3\right)} \in {\mathbb{R}}^{C\times WH}$, $Y_{\left(3\right)} \in {\mathbb{R}}^{C\times wh}$, and $Z_{\left(3\right)} \in {\mathbb{R}}^{c\times WH}$ indicate matrices of $\mathcal{X} \in {\mathbb{R}}^{W\times H\times C}$, $\mathcal{Y} \in {\mathbb{R}}^{w\times h\times C}$ and $\mathcal{Z} \in {\mathbb{R}}^{W\times H\times c}$ expanded along spectral mode individually. Therefore, we establish a preliminary representation relationship.

$$Y_{\left(3\right)} = X_{\left(3\right)}BH + N_{hs}$$

(6)

$$Z_{\left(3\right)} = R{X}_{\left(3\right)} + N_{ms}$$

(7)

where $B \in {\mathbb{R}}^{WH\times WH}$ represents the spatial blurring operator and $H \in {\mathbb{R}}^{WH\times wh}$ is often assumed to be a down-sampling matrix. $N_{hs}$ represents the independent and identically distributed (i.i.d.) noise of LR-HSI. $R \in {\mathbb{R}}^{c\times C}$ represents the spectral response of multispectral sensor. $N_{ms}$ indicates the independent and identically distributed (i.i.d.) noise of HR-MSI.

Based on the above description, we build a preliminary model as follows.

$$\underset{X_{\left(3\right)}}{argmin}\left\{\frac{1}{2}{∥Y_{\left(3\right)} - X_{\left(3\right)}BH∥}_F^2 + \frac{\delta }{2}{∥Z_{\left(3\right)} - R{X}_{\left(3\right)}∥}_F^2 + \lambda \varphi \left(X_{\left(3\right)}\right)\right\}$$

(8)

where $\delta$ is a parameter that measures the relationship between the two fidelity terms. There may be multiple solutions to solve this problem, resulting in severely ill-posed of the problem. Therefore, we carry out appropriate regularization constraints on the problem according to the prior knowledge of hyperspectral images. $\varphi \left(X_{\left(3\right)}\right)$ represents the constraint on the spatial and spectral information of potential $X_{\left(3\right)}$ and $\lambda$ represents the coefficient of the constraint term.

2.2.2. Tensor Sparse Representation Based on t-Product

Because of the similarity and redundancy between the bands of hyperspectral data, sparse representation has a strong advantage in super-resolution reconstruction model. For high-dimensional image data, traditional methods based on matrix decomposition usually embed high-dimensional data into the matrix space and use traditional dictionary learning methods to complete restoration or denoising tasks. However, this matrixization will break the original multi-dimensional structure of the image, Zhang et al. [46] proposed the tensor dictionary learning and considered the sparsity of the tensor coefficient based on the t-prodcut mentioned in Definition 3, and offered a tensor combination sparse representation model. The model is as follows.

$$\underset{\mathcal{D},\mathcal{A}}{argmin}\frac{1}{2}{\parallel \mathcal{X} - \mathcal{D}\ast \mathcal{A}\parallel }_F^2 + \lambda {\parallel \mathcal{A}\parallel }_1$$

(9)

where $\mathcal{X} \in {\mathbb{R}}^{a\times n\times b}$ indicates a third-order tensor which is stacked by n images of size $a\times b$. $\mathcal{D} \in {\mathbb{R}}^{a\times r\times b}$ represents a dictionary which also is a third-order tensor and per lateral slice of the dictionary indicates an atom. So, r expresses the number of atoms. $\mathcal{A} \in {\mathbb{R}}^{r\times n\times b}$ represents the tensor sparse coefficients. $\lambda$ is a sparse regularization parameter. According to the algebraic form of tensor-tensor product, each lateral slice of $\mathcal{X} \in {\mathbb{R}}^{a\times n\times b}$ can be linearly combined by tensor dictionary and tensor coefficient. Here, we present a linear combination representation of each lateral slice below as shown in Figure 1.

$$\begin{array}{c}\hfill \mathcal{X}(:,i,:) = \mathcal{D} \times \mathcal{A}(:,i,:) = \mathcal{D}(:,1,:) \times \mathcal{A}(1,i,:) + \dots + \mathcal{D}(:,r,:) \times \mathcal{A}(r,i,:)\end{array}$$

(10)

2.3. Proposed Method

Given the tensor linear combination sparse representation model based on t-prodcut, we proposed a super-resolution method based on this model. To take full advantage of the nonlocal self-similarity of hyperspectral images, we first performed nonlocal clustering on the up-sampled LR-HSI and HR-MSI, and transformed the problem into the reconstruction of each cluster of HR-HSI. Then per nonlocal cluster tensor was decomposed into two sub tensors under the tensor t-prodcut framework, and the potential properties of each sub tensor were explored. Specifically, the tensor dictionary was constrained by the prior knowledge of spectral smoothing (spectrum continuity) of HSIs. In addition, the $L_{1,1,2}$ norm was utilized to well characterize the tubal-row sparsity of tensor coefficients. Finally, the ADMM algorithm was employed to iteratively solve our model. The flowchart of the proposed method is shown in Figure 2.

2.3.1. Nonlocal Cluster Tensor

To capture strong low rankness of the patches in HSIs, we incorporate a nonlocal self-similarity prior to the model. Briefly, we divide the image into multiple patches by a clustering algorithm and stack these patches with similar spectral and spatail features into clusters. That is, we change the super-resolution task of the whole image into the reconstruction of each cluster. Here, we employ [49] to sort all the segmented patches into a highly smooth one-dimensional sequence, in which the two adjacent patches are highly similar. All you need to do is initializing the number of each cluster on the sequence.

Since HR-HSI is unknown, we cannot group it directly. Therefore, we cluster and group HR-HSI based on the spatial information and location of HR-MSI. So we first divide HR-MSI $\mathcal{Z} \in {\mathbb{R}}^{W\times H\times c}$ into a group of overlapping tensor patches ${\left\{{\mathcal{J}}_i\right\}}_{1<i<N} \subset {\mathbb{R}}^{v_w\times v_h\times c}$, where $v_w$ and $v_h$ indicate the width and height of the patch of the partitioning tensor, and N represents the number of patches of all the partitioning blocks. Then we group these tensor patches to form multiple clusters. The elements of per cluster are tensor patches with dimension of $v_w \times v_h \times c$. We use ${\left\{{\mathcal{J}}_i^k\right\}}_{1<i<n} \subset {\mathbb{R}}^{v_w\times v_h\times c}$ to represent the $k_{th}$ cluster with n tensor patches. Then, we expand each element of each cluster into a matrix along the spectral dimension of the tensor, and stack each matrix to form a new third-order tensor. We use ${\mathcal{Z}}^k \in {\mathbb{R}}^{c\times v_w{v}_h\times m_k}$ to represent this new third-order tensor, and we have ${\mathcal{Z}}^k(:,:,i) = {unfold}_3\left({\mathcal{J}}_i^k\right)$. Finally, we rewrite the fusion problems with the tensor notions as follows.

$$\begin{array}{c}\underset{\mathcal{X}}{argmin}\left\{\frac{1}{2}{\parallel \mathcal{Y} - \mathcal{X}\mathcal{B}\mathcal{H}\parallel }_F^2 + \frac{\delta }{2}{\parallel \mathcal{Z} - \mathcal{R}\mathcal{X}\parallel }_F^2\right\}\\ \mathrm{s}.\mathrm{t}.\mathcal{Y} = {\left({\displaystyle \sum _k}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _k}{U}_k^T{\mathcal{Y}}^k\\ \mathcal{Z} = {\left({\displaystyle \sum _k}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _k}{U}_k^T{\mathcal{Z}}^k\end{array}$$

(11)

Here, $U_k$ represents the operator that extracts the $k_{th}$ nonlocal cluster tensor from HR-MSI or LR-HSI, that is, ${\mathcal{Z}}^k = U_k\mathcal{Z},{\mathcal{Y}}^k = U_k\mathcal{Y}$. $\mathcal{B}$ and $\mathcal{H}$ represent spatial blurring kernel and downsampling operator, respectively. However, in practice, we use mode-n tensor-matrix product for computation, where $\mathcal{X}\mathcal{B}\mathcal{H}$ is equivalent to $BH{X}_{\left(3\right)}$. In addition, each pixel intensity of HR-MSI is equal to the spectral response times the pixel intensity in the corresponding HR-HSI. So when we use t-product to compute $\mathcal{Z} = \mathcal{R}\mathcal{X}$, we have $Z^{k^{\left(i\right)}} = R{X}^{k^{\left(i\right)}}$, $i = 1,\dots ,m_k$ to compute each slice individually, as shown in the following equation.

$$\left[\begin{array}{c}{Z}^{k^{\left(1\right)}}\\ Z^{k^{\left(2\right)}}\\ ⋮\\ Z^{k^{\left(m_k\right)}}\end{array}\right] = \left[\begin{array}{cccc}R& 0& \cdots & 0\\ 0& R& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & R\end{array}\right]\left[\begin{array}{c}{X}^{k^{\left(1\right)}}\\ X^{k^{\left(2\right)}}\\ ⋮\\ X^{k\left(m_k\right)}\end{array}\right]$$

(12)

2.3.2. Spectral Smooth Prior on Nonlocal Cluster Tensor

In HSIs, the nonlocal similarity between different patches not only represents the spatial structure similarity, but also represents spectral similarity. So, we apply t-prodcut to the reconstruction process of nonlocal cluster tensor. In this case, similar patches are likely to belong to the same type of ground material at the same coordinates, so they should be represented on the same basis. Based on Section 2.2.2, each nonlocal cluster tensor is reconstructed separately as follows.

$$\begin{array}{c}\underset{\mathcal{X},{\mathcal{D}}^k,{\mathcal{A}}^k}{argmin}\left\{\frac{1}{2}{\parallel \mathcal{Y} - \mathcal{X}\mathcal{B}\mathcal{H}\parallel }_F^2 + {\displaystyle \sum _{k=1}^K}\left(\frac{\delta }{2}∥{\mathcal{Z}}^k - \mathcal{R} \ast {\mathcal{D}}^k \ast {\mathcal{A}}^k∥ + \lambda {∥{\mathcal{A}}^k∥}_1\right)\right\}\\ \mathrm{s}.\mathrm{t}.\mathcal{Z} = {\left({\displaystyle \sum _k}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _k}{U}_k^T{\mathcal{Z}}^k\end{array}$$

(13)

For each nonlocal cluster tensor $\mathcal{X}$, we have to estimate two sub tensors: tensor dictionary ${\mathcal{D}}^k$ and coefficient tensor ${\mathcal{A}}^k$. In this section, we only discuss the properties of tensor dictionary. From Equation (10), it is easy to find that each lateral slice of the tensor is a linear combination of the corresponding tensor dictionary ${\mathcal{D}}^k$ and coefficient tensor ${\mathcal{A}}^k$. This means that these lateral slices can be regarded as the basis of the spectral space ${\mathcal{D}}^k$. That is, continuous bases tend to generate continuous data. For hyperspectral data, spectral smoothing is an important property. We assume that if we enforce the spectrum of each base of the dictionary to be smooth enough, then the spectrum of the nonlocal cluster tensor reconstructed from these bases will also be smooth enough. To confirm this hypothesis, we reconstructed one nonlocal cluster tensor by applying spectral smoothing constraint and without applying spectral smoothing constraint, respectively. From Figure 3, there are obvious differences between the spectral curves at the same spatial location, the original curve shows a smooth change. Therefore, we impose spectral smoothing constraints on tensor dictionary for all nonlocal cluster tensors. Then, (13) can be rewritten as follows.

$$\begin{array}{c}\underset{\mathcal{X},{\mathcal{D}}^k,{\mathcal{A}}^k}{argmin}\left\{\frac{1}{2}{\parallel \mathcal{Y} - \mathcal{X}\mathcal{B}\mathcal{H}\parallel }_F^2 + {\displaystyle \sum _{k=1}^K}\left(\frac{\delta }{2}∥{\mathcal{Z}}^k - \mathcal{R} \ast {\mathcal{D}}^k \ast {\mathcal{A}}^k∥ + \tau {∥{\mathcal{D}}^k \times {}_{1}M∥}_F^2 + \lambda {∥{\mathcal{A}}^k∥}_1\right)\right\}\\ \mathrm{s}.\mathrm{t}.\mathcal{Z} = {\left({\displaystyle \sum _k}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _k}{U}_k^T{\mathcal{Z}}^k\end{array}$$

(14)

where M represents a first-order difference matrix, $\tau$ is a regularization parameter and ${\times }_k$ is the k-mode product between tensor and matrix.

2.3.3. Tubal Sparsity Constraint with Sparse Representation Model for Nonlocal Cluster Tensor

Since the optimization problem of $L_0$ norm is not convex [50], many scholars proposed to use a convex relaxation technique to transform it into a convex optimization problem. Xu’s work [45], as well as Li’s [37] and Dian’s [38] work, only uses $L_1$ norm [51] to constrain the tensor coefficients. However, it ignores the underlying structural properties of tensor coefficient factors, which can easily lead to loss of image information and sub-optimal solutions. Therefore, we explore the intrinsic structural properties of tensor coefficients from two aspects. On the one hand, based on tensor sparse representation model of t-prodcut, the actual operation is the convolution operation between the lateral slices of the tensor dictionary and tubes of the tensor coefficients. Therefore, we reckon that it is easier to obtain an optimal solution by capturing the sparsity presented by all tubes of the tensor coefficient. On the other hand, the group sparse signals are more seemly to emerge after nonlocal clustering than completely random sparse signals. As illustrated in Figure 4, the tensor coefficients exhibit a row sparsity. So, we use the $L_{1,1,2}$ norm mentioned in Definition 4 to characterize the tubal row-sparsity of tensor coefficients.

So, we rewrite Equation (14) as follows.

$$\begin{array}{c}\underset{\mathcal{X},{\mathcal{D}}^k,{\mathcal{A}}^k}{argmin}\left\{\frac{1}{2}{\parallel \mathcal{Y} - \mathcal{X}\mathcal{B}\mathcal{H}\parallel }_F^2 + {\displaystyle \sum _{k=1}^K}\left(\frac{\delta }{2}∥{\mathcal{Z}}^k - \mathcal{R} \ast {\mathcal{D}}^k \ast {\mathcal{A}}^k∥ + \right.\right.\\ \left.\left.\tau {∥{\mathcal{D}}^k \times {}_{1}M∥}_F^2 + \lambda {∥{\mathcal{W}}^k \odot {\mathcal{A}}^k∥}_{1,1,2}\right)\right\}\\ \mathrm{s}.\mathrm{t}.\mathcal{Z} = {\left({\displaystyle \sum _k}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _k}{U}_k^T{\mathcal{Z}}^k\end{array}$$

(15)

where ${\mathcal{W}}^k$ represents a weight tensor that better promote the tubal sparsity, and ⊙ represents the operation of component-wise multiplication.

3. Optimization Algorithm

Having ADMM algorithm in mind, we introduce two new variables to decouple $\mathcal{D}$ and $\mathcal{A}$, so we can express (14) as the following Lagrangian.

$$\begin{array}{cc}\hfill \mathcal{L}\left(\mathcal{X}\right)& =\frac{1}{2}{\parallel \mathcal{Y} - \mathcal{X}\mathcal{B}\mathcal{H}\parallel }_F^2\hfill \\ \hfill & + {\displaystyle \sum _{k=1}^K}\left(\frac{\delta }{2}{∥{\mathcal{Z}}^k - \mathcal{R} \ast {\mathcal{D}}^k \ast {\mathcal{A}}^k∥}_F^2 + \lambda {∥{\mathcal{W}}^k\odot {\mathcal{E}}^k∥}_{1,1,2}\right) + \tau {∥{\mathcal{C}}^k \times {}_{1}M∥}_F^2\hfill \\ \hfill & +{\displaystyle \sum _{k=1}^K}〈{\mathcal{P}}_2^k,{\mathcal{D}}^k - {\mathcal{C}}^k〉 + \frac{\sigma }{2}{\displaystyle \sum _{k=1}^K}{∥{\mathcal{D}}^k - {\mathcal{C}}^k∥}_F^2\hfill \\ \hfill & + \sum _{k=1}^K〈{\mathcal{P}}_3^k,{\mathcal{A}}^k - {\mathcal{E}}^k〉 + \frac{\sigma }{2}{\displaystyle \sum _{k=1}^K}{∥{\mathcal{A}}^k - {\mathcal{E}}^k∥}_F^2\hfill \\ \hfill & +〈{\mathcal{P}}_1,\mathcal{X} - {\left({\displaystyle \sum _{k=1}^K}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _{k=1}^K}{U}_k^T\left({\mathcal{D}}^k \ast {\mathcal{A}}^k\right)〉\hfill \\ \hfill & +\frac{\sigma }{2}{∥\mathcal{X} - {\left({\displaystyle \sum _{k=1}^K}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _{k=1}^K}{U}_k^T\left({\mathcal{D}}^k \ast {\mathcal{A}}^k\right)∥}_F^2\hfill \end{array}$$

(16)

where ${\mathcal{P}}_1$, ${\mathcal{P}}_2^k$, and ${\mathcal{P}}_3^k$ are the so-called Lagrange multipliers, $\sigma$ is a penalty parameter. Then we solve this optimization algorithm by alternatively updating each variable with the others fixed. It should be noted that each cluster tensor is optimized and solved separately, and ${\mathcal{C}}^k$, ${\mathcal{D}}^k$, ${\mathcal{E}}^k$ and ${\mathcal{A}}^k$ are all updated independently. The optimization process is as follows.

 (1)

Update${\mathcal{C}}^k$

$$\underset{{\mathcal{C}}^k}{argmin}\left\{\frac{\sigma }{2}{∥{\mathcal{C}}^k - {\mathcal{D}}^k - {\mathcal{P}}_2^k/\sigma ∥}_F^2 + \tau {∥{\mathcal{C}}^k \times {}_{1}M∥}_F^2\right\}$$

(17)

In order to facilitate the solution, the tensors in the equation are expanded along $1 - mode$. The optimal of ${\mathcal{C}}^k$ along $1 - mode$ is

$$C_{\left(1\right)}^k = {\left[\sigma I + \tau \left(M^{T}M\right)\right]}^{-1}\left(\sigma D_{\left(1\right)}^k + P_2^k\right)$$

(18)

It is worth noting that $D_{\left(1\right)}^k$ and $P_2^k$ are the unfolded matrices of ${\mathcal{C}}^k$ and ${\mathcal{P}}_2^k$ along the first dimension, respectively. After getting $C_{\left(1\right)}^k$, we fold it to tensor form, where ${\mathcal{C}}^k = {fold}_1\left(C_{\left(1\right)}^k\right)$.

 (2)

Update${\mathcal{D}}^k$

$$\begin{array}{cc}\hfill \underset{{\mathcal{D}}^k}{argmin}& \frac{\delta }{2}{\parallel {\mathcal{Z}}^k - \mathcal{R} \ast {\mathcal{D}}^k \ast {\mathcal{A}}^k\parallel }_F^2 + \frac{\sigma }{2}{∥{\mathcal{X}}^k - {\mathcal{D}}^k \ast {\mathcal{A}}^k + {\mathcal{P}}_1^k/\sigma ∥}_F^2\hfill \\ & +\frac{\sigma }{2}{∥{\mathcal{D}}^k - {\mathcal{C}}^k + {\mathcal{P}}_2^k/\sigma ∥}_F^2\hfill \end{array}$$

(19)

Each ${\mathcal{D}}^k$ can be solved seperately. In this case, ${\mathcal{P}}_1^k = U_k{\mathcal{P}}_1$. According to Lemma 1, we know that ${\mathcal{D}}^k \ast {\mathcal{A}}^k$ is equivalent to ${\overline{D^k}}^{\left(i\right)}{\overline{A^k}}^{\left(i\right)}$, $i \in \left[m_k\right]$. Therefore, Equation (19) can be effectively solved in Fourier domain. Its unique solution is equivalent to solving the general Sylvester Equation (20).

$$\begin{array}{cc}\hfill & \left(\delta {\overline{R}}^{\left(i\right)}{\overline{R}}^{{\left(i\right)}^T} + \sigma I\right)\left({\overline{D^k}}^{\left(i\right)}{\overline{A^k}}^{\left(i\right)}{\overline{A^k}}^{{\left(i\right)}^T}\right) + \sigma {\overline{D^k}}^{\left(i\right)}\hfill \\ & = \delta {\overline{Z^k}}^{\left(i\right)}{\overline{R}}^{\left(i\right)}{\overline{A^k}}^{{\left(i\right)}^T} + \sigma {\overline{X^k}}^{\left(i\right)}{\overline{A^k}}^{{\left(i\right)}^T} + {\overline{{\mathcal{P}}_1^k}}^{\left(i\right)}{\overline{A^k}}^{{\left(i\right)}^T} + \sigma {\overline{C^k}}^{\left(i\right)}-{\overline{{\mathcal{P}}_2^k}}^{\left(i\right)}\hfill \end{array}$$

(20)

The conjugate gradient method [52] is used to solve (20) efficiently.

 (3)

Update${\mathcal{E}}^k$

The optimization problem of ${\mathcal{E}}^k$ is

$$\underset{{\mathcal{E}}^k}{argmin}\frac{\sigma }{2}{∥{\mathcal{E}}^k - {\mathcal{A}}^k - {\mathcal{P}}_3^k/\sigma ∥}_F^2 + \lambda {∥{\mathcal{W}}^k \odot {\mathcal{E}}^k∥}_{1,1,2}$$

(21)

Here, we use a soft threshold operator for the solution.

$${\mathcal{E}}^k(i,j,:) = {shrink}_{1,1,2}\left(temp{\mathcal{A}}^k(i,j,:),\left|{\mathcal{W}}^k(i,j)\right| · \frac{\lambda }{\sigma }\right)$$

(22)

where

$$\begin{array}{c}\hfill temp{\mathcal{A}}^k = {\mathcal{A}}^k + {\mathcal{P}}_3^k/\sigma \\ \hfill {\mathcal{W}}^k(i,j) = \frac{1}{{∥temp{\mathcal{A}}^k(i,j,:)∥}_2 + \gamma }\end{array}$$

(23)

If ${\psi <\parallel x\parallel }_2$, ${shrink}_{1,1,2}(x,\psi ) = \frac{{\parallel x\parallel }_2-\psi }{{\parallel x\parallel }_2}x$ , otherwise ${shrink}_{1,1,2}(x,\psi ) = 0$. The function of $\gamma$ is to avoid singularities.

 (4)

Update${\mathcal{A}}^k$

The optimization for the coefficient tensors is shown below.

$$\begin{array}{cc}\hfill & \underset{{\mathcal{A}}^k}{argmin}\left\{\frac{\delta }{2}{∥{\mathcal{Z}}^k - \mathcal{R} \ast {\mathcal{D}}^k \ast {\mathcal{A}}^k∥}_F^2 + \frac{\sigma }{2}\parallel {\mathcal{X}}^k - {\mathcal{D}}^k \ast {\mathcal{A}}^k +\right.\hfill \\ \hfill & \left.{\mathcal{P}}_1^k/\sigma ∥{}_F^2 + \frac{\sigma }{2}∥{\mathcal{A}}^k - {\mathcal{E}}^k - {\mathcal{P}}_3^k{/\sigma \parallel }_F^2\right\}\hfill \end{array}$$

(24)

Similar to ${\mathcal{D}}^k$, ${\mathcal{A}}^k$ can be effectively solved in the Fourier domain.

$$\begin{array}{cc}\hfill {\overline{{\mathcal{A}}^k}}^{\left(i\right)}& = \left(\delta {\overline{D^k}}^{{\left(i\right)}^T}{\overline{R}}^{{\left(i\right)}^T}{\overline{R}}^{\left(i\right)}{\overline{D^k}}^{\left(i\right)}\right.{\left. + \sigma {\overline{D^k}}^{{\left(i\right)}^T}{\overline{D^k}}^{\left(i\right)}\right)}^{-1}\left(\delta {\overline{D^k}}^{{\left(i\right)}^T}{\overline{R}}^{{\left(i\right)}^T}{\overline{Z^k}}^{\left(i\right)}\right.\hfill \\ \hfill & \left. + \sigma {\overline{D^k}}^{{\left(i\right)}^T}{\overline{X^k}}^{\left(i\right)} + {\overline{D^k}}^{{\left(i\right)}^T}{\overline{{\mathcal{P}}_1^k}}^{\left(i\right)} + \sigma {\mathcal{E}}^k - {{\mathcal{P}}_3^k}^{\left(i\right)}\right)\hfill \end{array}$$

(25)

 (5)

Update$\mathcal{X}$

The optimization problem of $\mathcal{X}$ is

$$\begin{array}{cc}\hfill \mathcal{X}& = \underset{\mathcal{X}}{argmin}\frac{1}{2}{∥\mathcal{Y} - \mathcal{X}\mathcal{B}\mathcal{H}∥}_F^2 + \left({\mathcal{P}}_1,\mathcal{X} - {\left({\displaystyle \sum _{k=1}^K}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _{k=1}^K}{U}_k^T\left({\mathcal{D}}^k \ast {\mathcal{A}}^k\right)\right)\hfill \\ \hfill & + \frac{\sigma }{2}{∥\mathcal{X} - {\left({\displaystyle \sum _{k=1}^K}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _{k=1}^K}{U}_k^T\left({\mathcal{D}}^k \ast {\mathcal{A}}^k\right)∥}_F^2\hfill \end{array}$$

(26)

Here, we convert it to

$$\begin{array}{cc}\hfill & \mathcal{X}\left(\mathcal{B}\mathcal{H}{\mathcal{H}}^T{\mathcal{B}}^T + \right.\sigma \mathcal{D}) = \mathcal{Y}{\mathcal{H}}^T{\mathcal{B}}^T - {\mathcal{P}}_1 + \sigma {\left({\displaystyle \sum _{k=1}^K}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _{k=1}^K}{U}_k^T\left({\mathcal{D}}^k \ast {\mathcal{A}}^k\right)\hfill \end{array}$$

(27)

Here, we use the conjugate gradient method to solve (27) efficiently.

 (6)

Update Lagrange multipliers

$$\begin{array}{c}{\mathcal{P}}_1 = {\mathcal{P}}_1 + \sigma \left(\mathcal{X} - {\left({\displaystyle \sum _{k=1}^K}{U}_k^T{U}_k\right)}^{-1}{\displaystyle \sum _{k=1}^K}{U}_k^T\left({\mathcal{D}}^k\ast {\mathcal{A}}^k\right)\right)\\ {\mathcal{P}}_2^k = {\mathcal{P}}_2^k + \sigma \left({\mathcal{D}}^k - {\mathcal{C}}^k\right),k=1\dots ,K\\ {\mathcal{P}}_3^k = {\mathcal{P}}_3^k + \sigma \left({\mathcal{C}}^k - {\mathcal{E}}^k\right),k=1\dots ,K\end{array}$$

(28)

where $\sigma = s\sigma$, $s>1$ and the process of SSTSR is shown in Algorithm 1.

Algorithm 1 The proposed SSTSR method for HSI super-resolution.

Require: LR-HSI $\mathcal{Y}$, HR-MSI $\mathcal{Z}$, $v_w$, $v_h$, $\mathcal{B}$, $\mathcal{R}$, $\mathcal{H}$, $\delta$, $\lambda$, $\tau$ Ensure: HR-HSI ${\mathcal{X}}_{\left(i\right)}$ 1: Initialization:$i = 1,i_{max} = 15$, ${\mathcal{X}}_{\left(\mathbf{0}\right)} = \mathcal{Y}{\mathcal{H}}^T{\mathcal{B}}^T$, ${\mathcal{P}}_{1^{\left(0\right)}} = 0,{\mathcal{P}}_{2^{\left(0\right)}}^k = 0,{\mathcal{P}}_{3^{\left(0\right)}}^k = 0$, ${\mathcal{D}}_{\left(0\right)}^k,{\mathcal{A}}_{\left(0\right)}^k,{\mathcal{C}}_{\left(0\right)}^k,{\mathcal{E}}_{\left(0\right)}^k$ are randomly initialized with the scale dimensions, $\sigma = 1$, $s = 1.01$, tol=0.001. 2: While not converged and $i < i_{max}$ do 3: For$cluste{r}_k = 1:Ma{x}_k$ 4: Update${\mathcal{E}}_{\left(i\right)}^k$ by (22) 5: Update${\mathcal{C}}_{\left(i\right)}^k$ by (18) 6: Update${\mathcal{D}}_{\left(i\right)}^k$ by (20) with CG 7: Update${\mathcal{A}}_{\left(i\right)}^k$ by (25) 8: Update${\mathcal{P}}_{2^{\left(i\right)}}^k$, ${\mathcal{P}}_{3^{\left(i\right)}}^k$ by (28) 9: End for 10: Update${\mathcal{X}}_{\left(i\right)}$ by (27) with CG 11: Update${\mathcal{P}}_{1^{\left(i\right)}}$ by (28), $\sigma = s\sigma$ 12: Check the convergence condition 13: ${∥{\mathcal{X}}_{\left(i\right)} - {\left({\sum }_{k=1}^K{\mathbf{U}}_k^T{\mathbf{U}}_k\right)}^{-1}{\sum }_{k=1}^K{\mathbf{U}}_k^T\left({\mathcal{D}}_{\left(i\right)}^k\ast {\mathcal{A}}_{\left(i\right)}^k\right)∥}_F^2$<tol, ${∥{\mathcal{X}}_{\left(i\right)} - {\mathcal{X}}_{(i-1)}∥}_F^2$<tol 14: $\parallel {\left({\sum }_{k=1}^K{U}_k^T{U}_k\right)}^{-1}{\sum }_{k=1}^K{U}_k^T\left({\mathcal{D}}_{\left(i\right)}^k\ast {\mathcal{A}}_{\left(i\right)}^k\right)-$${\left({\sum }_{k=1}^K{U}_k^T{U}_k\right)}^{-1}{\sum }_{k=1}^K{U}_k^T\left({\mathcal{D}}_{(i-1)}^k \ast {\mathcal{A}}_{(i-1)}^k\right){\parallel }_F^2$ 15: <tol 16: Update iteration 17: $i = i + 1$ 18: End While

4. Results

4.1. Synthetic Dataset

Data set 1: The first data (http://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes, accessed on 29 January 2022) set was formed by ROSIS sensor capturing scenes from the University of Pavia (PU) over northern Italy. The initial data size was $610 \times 340 \times 115$, but considering the presence of water vapor absorption bands in the data and in order to show more spatial details, we sampled the data as the reference image, which has size of $300 \times 300 \times 103$. We simulated LR-HSI by using a $9 \times 9$ Gaussian filter with standard deviation 2.12 to blur it from the reference image, then we sampled it with down sampling ratio of 5. We used the spectral response function of IKONOS multispectral sensor to simulate the generation of HR-MSI. After that, the independent identically distributed noise of corresponding SNR was applied. For LR-HSI, the SNR was set to 30 dB, and for HR-MSI, the SNR was arranged to 40 dB.

Data set 2: The second data set (https://engineering.purdue.edu/~biehl/MultiSpec/hyperspectral.html, accessed on 29 January 2022) was formed by HYDICE sensor capturing scenes from the National Mall over Washington, D.C (WDC). Similarly, there were bands with low SNR in this data, so the final size left was $1280 \times 307 \times 191$, and then we chose $300 \times 300 \times 191$ as the reference image. We used the same simulation process as the first data set to generate LR-HSI and HR-MSI.

Data set 3: The last data set (2013 IEEE GRSS Data Fusion Contest: http://www.grss-ieee.org/community/technical-committees/data-fusion, accessed on 29 January 2022) was Houston (HOS) captured by ITRES CASI-1500 sensor, which was offered by the 2013 IEEE GRSS Data Fusion Competition. The original size of Houston was $349 \times 1905 \times 144$. Similarly, we selected the bands and sampled the data with rich details of features. So, the final size of reference image was $300 \times 300 \times 103$. We used the same method to generate LR-HSI together with HR-MSI as we did for the first data set.

4.2. Quantitative Metrics

We used six quantitative index [53] to judge our superresolution output.

(1) PSNR: The Peak Signal to-Noise Ratio. PSNR is the most regular objective evaluation index of images and the larger PSNR is, the less image distortion is. Since the data are hyperspectral images with multiple bands, we calculate the average PSNR of all bands.

(2) SAM: The spectral angle mapper. Sam computes the average angle between the estimated image spectrum and the reference image spectrum.

(3) CC: The Correlation Coefficient.

(4) ERGAS: The relative dimensionless global error in synthesis. ERGAS mainly evaluates the spectral quality of all fusion bands.

(5) SSIM: Structural Similarity. It mainly measures the similarity of the estimated image and reference image.

(6) UIQI: The Universal Image Quality Index. The UIQI calculates the average value of all image patches for the estimated image and reference image.

4.3. Compared Methods

We choose some classical and recent advanced methods based on tensor decomposition to compare with our proposed method, including Hysure [17], CSTF [37], NPTSR [45], LTTR [41], NLSTF [38] and LTMR [42]. The parameters of these comparison algorithms are the best parameters described in the reference literature as far as possible.

All methods except for CSTF have the same spectral response function and spatial blur kernels in the experiments. For CSTF, we follow the assumption about the separability of the downsampling operator along spatial modes.

For the parameters used in the comparison algorithms, we follow the reference literature as much as possible, and have adjusted them appropriately. For Hysure, the number of subspace bands is 10 and the dimensions of estimated method choose VCA, ${\lambda }_m = 1$, ${\lambda }_{\varphi } = 5\times {10}^{-4}$. For CSTF, $n_w = n_h = 500$, $n_s = 15$, $\lambda = {10}^{-5}$. For LTMR, $K = 200$, $patchsize = 10$, $L = 10$, $\lambda = {10}^{-3}$. For NLSTF, $n_w = n_h = 10$, $n_s = 10$, $K = 150$, $\lambda = {10}^{-7}$, ${\lambda }_1 = {10}^{-4}$, ${\lambda }_2 = {\lambda }_3 = {10}^{-5}$. For LTTR, $K = 460$, $\lambda = 4\times {10}^{-4}$. For NPTSR, $\lambda = {10}^{-3}$, $\beta = {10}^3$, $N_k = 3$, $r = 10$.

In addition, we also compare the proposed SSTSR method with the two deep learning methods, i.e., HAM-MFN [54] and CNN-Fus [55], the details are described in Section 4.4.

4.4. Experimental Results on Synthetic Datasets

In this section, we present the results generated by all methods on three simulated datasets, i.e., PU, WDC, and HOS. We also show the comparison of quantitative indicators in Table 1 and Table 2, where the best values are boldface. In addition, we demonstrate the superiority of our method in the following three aspects.

(1)

Visual effects of reconstructed images. Figure 5, Figure 6 and Figure 7 list the fusion results of different methods on three datasets, i.e., PU, WDC, and HOS, respectively. To deepen the visual effect, we pseudo-color the experimental results while magnifying the representative local information. In addition, with the aid of ground truth, the comparison of residual images is supplemented, in which the dark blue residual image indicates better reconstruction effect. As can be seen from Figure 5, Figure 6 and Figure 7, the results of CSTF, LTMR, LTTR, and NLSTF all show color distortion compared with the ground truth. From the residual image, the result of our method is bluer and smoother. It fully verifies that our proposed method can obtain images with better spatial structure details.

(2)

Spectral curve and spectral curve residual. In addition, we also compare the spectral quality of the reconstructed images. Figure 8a shows the spectral curve of the reconstructed image at pixel (90, 90) of the PU dataset and the residual spectral curve of the reconstructed image with the ground truth. Similarly, Figure 8b,c also compare the spectral curves at the pixel (100, 200) of the WDC and the pixel (100, 100) of the HOS, respectively. It is clear from Figure 8 that the spectral curves of the reconstructed images of our method on the three datasets are closer to the ground truth spectral curves, and the residual curves are also closer to the zero-horizontal line. This also demonstrates the effectiveness of the spectral smoothing constraints imposed in our method. Compared with other methods, our proposed SSTSR method can obtain images with higher spectral quality.

(3)

Quantitative indicators and time complexity comparison. As can be seen from Table 1, on the PU dataset, our method achieves a leading position in all indicators, and on the WDC dataset, although the three indicators of SAM, CC, and ERGAS slightly lag behind Hysure and NPTSR, our PSNR, SSIM, UIQI values are still leading, and on the HOS dataset, all indicators of our method once again rank first. It needs to be mentioned that all methods have similar SSIM values on HOS dataset, so the results of this indicator are not listed. Taken together, the average PSNR of our method on the three datasets is 0.63 dB, 2.85 dB, 4.53 dB, 9.84 dB, 2.33 dB, 0.33 dB higher than Hysure, CSTF, LTMR, LTTR, NLSTF, NPTSR, respectively, which verifies the superiority of our method. In addition, we also give a comparison of PSNR in each band for all methods on the three datasets in Figure 9. As can be seen from Figure 9, our method outperforms other methods in most bands. Besides, the measurement of the ERGAS index indicates the spectral quality of the reconstructed image and the smaller the value, the better the spectral quality. The ERGAS value of our method is also state-of-the-art on three datasets. Although the designed regular terms can improve the performance of the method, they also consume more computing time. Our method has no advantage in the comparison of time complexity, so in future work, we will focus on optimizing our method to reduce the time complexity.

We also selected two deep learning methods [54,55] for comparison. Unfortunately, there was a lack of relevant codes. We directly quoted the results in the reference literature for quantitative comparison. For a fair comparison, we set the same conditions for the experiments. (1) The size and scope of HR-HSI and HR-MSI were the same. (2) We simulated and generated LR-HSI and HR-MSI according to the spatial-spectral degradation method described in the reference literature. (3) We selected the same quantitative indicator function. Specifically, compared with the HAM-MFN [54], the selected PU size was $260 \times 340 \times 103$, the upper left corner coordinate was (351, 1), the spectral degradation was simulated by the randomly generated spectral response function, and the spatial degradation was simulated by bicubic linear interpolation and the downsampling ratio was 4. Compared with the CNN-Fus [55], the selected PU size was $610 \times 340 \times 103$, the upper left corner coordinate was (0, 0), the spectral degradation was simulated by the spectral response function of the IKONOS sensor to generate HR-MSI, and the $7 \times 7$ Gaussian filter with a standard deviation of 2 was used to simulate spatial degradation with a downsampling ratio of 4. As can be seen from Table 2, our method also showed certain advantages.

Table 1. Quantitative and complexity comparison in simulated PU, WDC and HOS data sets.

Dataset	Index	Best Values	Hysure [17]	CSTF    [37]	LTMR [42]	LTTR [41]	NLSTF [38]	NPTSR [45]	SSTSR
PU	PSNR	+∞	43.26	40.76	40.71	35.53	40.70	43.55	43.84
	SAM	0	2.5971	2.7189	3.7590	5.6304	2.8811	2.4149	2.4080
	CC	1	0.9946	0.9943	0.9868	0.9769	0.9915	0.9950	0.9951
	ERGAS	0	1.1519	1.2062	1.6003	2.3986	1.5421	1.1248	1.1071
	SSIM	1	0.9379	0.9308	0.9053	0.8261	0.9390	0.9438	0.9444
	UIQI	1	0.9257	0.9165	0.8896	0.8017	0.9271	0.9328	0.9335
	TIME	0	43	37	220	176	20	280	371
WDC	PSNR	+∞	46.05	44.64	43.32	36.77	43.28	46.43	46.81
	SAM	0	6.6306	6.0177	6.8458	8.2989	10.5458	5.0079	5.0167
	CC	1	0.9177	0.9115	0.8417	0.6580	0.8167	0.9097	0.9150
	ERGAS	0	5.7105	7.4531	12.6333	30.8024	10.6421	9.0399	6.4304
	SSIM	1	0.7302	0.6652	0.5595	0.3564	0.6451	0.7576	0.7703
	UIQI	1	0.6797	0.7113	0.5006	0.3289	0.6105	0.7192	0.7286
	TIME	0	43	48	210	352	25	489	673
HOS	PSNR	+∞	50.24	47.19	43.81	39.63	50.46	50.49	50.80
	SAM	0	1.4110	1.2761	3.4890	5.1802	1.3703	1.2914	1.2746
	CC	1	0.9980	0.9981	0.9890	0.9779	0.9982	0.9982	0.9983
	ERGAS	0	0.6727	0.6437	1.7494	2.2988	0.6246	0.6173	0.6111
	SSIM	1	-	-	-	-	-	-	-
	UIQI	1	0.9819	0.9810	0.9289	0.8562	0.9831	0.9838	0.9835
	TIME	0	43	35	80	174	25	271	341

Table 1. Quantitative and complexity comparison in simulated PU, WDC and HOS data sets.

Dataset	Index	Best Values	Hysure [17]	CSTF    [37]	LTMR [42]	LTTR [41]	NLSTF [38]	NPTSR [45]	SSTSR
PU	PSNR	+∞	43.26	40.76	40.71	35.53	40.70	43.55	43.84
	SAM	0	2.5971	2.7189	3.7590	5.6304	2.8811	2.4149	2.4080
	CC	1	0.9946	0.9943	0.9868	0.9769	0.9915	0.9950	0.9951
	ERGAS	0	1.1519	1.2062	1.6003	2.3986	1.5421	1.1248	1.1071
	SSIM	1	0.9379	0.9308	0.9053	0.8261	0.9390	0.9438	0.9444
	UIQI	1	0.9257	0.9165	0.8896	0.8017	0.9271	0.9328	0.9335
	TIME	0	43	37	220	176	20	280	371
WDC	PSNR	+∞	46.05	44.64	43.32	36.77	43.28	46.43	46.81
	SAM	0	6.6306	6.0177	6.8458	8.2989	10.5458	5.0079	5.0167
	CC	1	0.9177	0.9115	0.8417	0.6580	0.8167	0.9097	0.9150
	ERGAS	0	5.7105	7.4531	12.6333	30.8024	10.6421	9.0399	6.4304
	SSIM	1	0.7302	0.6652	0.5595	0.3564	0.6451	0.7576	0.7703
	UIQI	1	0.6797	0.7113	0.5006	0.3289	0.6105	0.7192	0.7286
	TIME	0	43	48	210	352	25	489	673
HOS	PSNR	+∞	50.24	47.19	43.81	39.63	50.46	50.49	50.80
	SAM	0	1.4110	1.2761	3.4890	5.1802	1.3703	1.2914	1.2746
	CC	1	0.9980	0.9981	0.9890	0.9779	0.9982	0.9982	0.9983
	ERGAS	0	0.6727	0.6437	1.7494	2.2988	0.6246	0.6173	0.6111
	SSIM	1	-	-	-	-	-	-	-
	UIQI	1	0.9819	0.9810	0.9289	0.8562	0.9831	0.9838	0.9835
	TIME	0	43	35	80	174	25	271	341

Table 2. Comparison of quantitative indicators between the proposed method and the deep learning methods.

Dataset	Method	PSNR	SAM	ERGAS	SSIM
PU (260 × 340 × 103)	HAMMFN [54]	40.8632	2.5308	1.8052	0.9776
	SSTSR	44.0746	2.3627	1.1156	0.9428
Dataset	Method	PSNR	SAM	UIQI	SSIM
PU (610 × 340 × 103)	CNN-Fus [55]	43.0170	2.2350	0.9920	0.9870
	SSTSR	43.6980	2.4149	0.9932	0.9607

Table 2. Comparison of quantitative indicators between the proposed method and the deep learning methods.

Dataset	Method	PSNR	SAM	ERGAS	SSIM
PU (260 × 340 × 103)	HAMMFN [54]	40.8632	2.5308	1.8052	0.9776
	SSTSR	44.0746	2.3627	1.1156	0.9428
Dataset	Method	PSNR	SAM	UIQI	SSIM
PU (610 × 340 × 103)	CNN-Fus [55]	43.0170	2.2350	0.9920	0.9870
	SSTSR	43.6980	2.4149	0.9932	0.9607

Table 3. The necessity of using spectral smooth and tubal row-sparse constraints.

Constraints	PSNR	SAM	ERGAS	CC	UIQI	SSIM
Spectral Smooth	46.57	5.1505	6.6625	0.9131	0.7243	0.7621
ine Tubal Sparsity	46.50	5.0497	8.7855	0.9076	0.7164	0.7511
ine Both Constraints	46.81	5.0167	6.3404	0.9150	0.7286	0.7703

Table 3. The necessity of using spectral smooth and tubal row-sparse constraints.

Constraints	PSNR	SAM	ERGAS	CC	UIQI	SSIM
Spectral Smooth	46.57	5.1505	6.6625	0.9131	0.7243	0.7621
ine Tubal Sparsity	46.50	5.0497	8.7855	0.9076	0.7164	0.7511
ine Both Constraints	46.81	5.0167	6.3404	0.9150	0.7286	0.7703

4.5. Experimental Results on Real Dataset

The real data set was captured and formed from the scenes over Paris. Among them, the multi-spectral data was provided by The ALI Instrument, and the hyperspectral data was captured and formed by Hyperion. The size of HSI was $24 \times 24 \times 128$, the size of MSI was $72 \times 72 \times 9$. For the estimation of spectral response function and blurring kernel, we employed the algorithm proposed in [17] according to the input data. Similarly, for CSTF, we used a separable blurring kernel to guarantee the maximum fair comparison. Since there was no ground truth maps, we did not list the quantitative results. In Figure 10, we illustrated the images of band 90 of the competing algorithms for visual comparison, and it was clear that the results of the proposed SSTSR super-resolution method contained more fine details, as shown in the red rectangle.

5. Discussion

5.1. Parameters Selection

There are five important parameters for our proposed method, including $\delta$, $\lambda$, $\tau$, $m_k$ and r. $\delta$ weighs the relationship between the two fidelity terms. $\lambda$ and $\tau$ are penalty factors for the two regular terms. $m_k$ determines the number of each cluster, and r determines the size of the tensor dictionary.

Figure 11a–c show the psnr values as a function of parameter $\delta$ and parameter $\lambda$ on the simulated three data sets, respectively. For these experiments, $\delta$ was selected from [100 500 1000 1500 2000] and $\lambda$ was chosen from [0.001 0.005 0.01 0.015 0.02 0.03 0.04 0.05]. It can be seen from Figure 11a–c that the values of parameters $\delta$ and $\lambda$ do have a great influence on the experimental results. However, for the three data sets, when parameter $\delta = {10}^3$, the local optimal values of parameter $\lambda$ may be reached. Therefore, we set $\lambda = 0.03$ for PU, $\lambda = 0.01$ for WDC, $\lambda = 0.01$ for HOS, respectively. In addition, Figure 12a shows the psnr as a function of parameter $\tau$ on the WDC dataset. It could be seen that the proposed SSTSR exhibited stable performance when $\tau =$ 10,000. Furthermore, we also found that setting $\tau =$ 10,000 was also robust for both PU and HOS datasets.

Besides, we also discussed the number of nonlocal clusters $m_k$ and the number of dictionary atoms r. First, we set the range of [1 3 5 7 9 11] for $m_k$. The selection of r should not only consider the reduction of the subspace dimension, but also ensure that enough spectral information was retained. The range of r was set to [5 10 15 20 25]. As can be seen in Figure 11d–f, the PSNR of the reconstructed images of the three datasets reached the maximum value when $r = 5$. Because of the difference of similar spatial information in each dataset, the values of $m_k$ for the three datasets were slightly different, so we set $m_k = 5$ for PU, $m_k=5$ for WDC, $m_k = 9$ for HOS, respectively. In addition, for the non-local clustering algorithm used in our method, the size of the nonlocal clustering patch was set to $20 \times 20$ and the step size was set to 8.

5.2. Convergence Behavior

Taking University of Pavia and Washington DC Mall as examples, Figure 12b shows the relative error calculated by ${∥{\mathcal{X}}_{(i+1)}-{\mathcal{X}}_{\left(i\right)}∥}_F/{∥{\mathcal{X}}_{\left(i\right)}∥}_F$ of the reconstructed image as a function of iterations. It is clear that the relative error decreases sharply in the initial stage, and then continues to decrease in a relatively smooth trend until the relative error is close to 0. This verifies that our proposed SSTST method has strong convergence. When the number of iterations is 15, the tolerance threshold of the algorithm can be reached.

5.3. Effectiveness of the Spectral Smooth Prior and Tubal Sparsity Constraint

In this subsection, we discuss the influence of each regular term on the reconstructed results. First, we did the related experiments on data set 2. From Table 3, we can find that the ERGAS index with only the spectral smoothing constraint is lower than the ERGAS index without the spectral smoothing constraint, and the lower the ERGAS index, the better the fusion quality of the spectrum, which fully invalidates the effeciveness of the spectral smoothing constraint. By adding tube row-sparseness constraint on the basis, the PSNR increases by 0.3 dB, which validates that the constraints imposed on both dictionaries and coefficients are reasonable and effective.

6. Conclusions

In this article, we propose a SSTSR model for HSI super-resolution, which employs the tensor sparse representation framework based on t-product. The proposed SSTSR method puts forward continuity constraints on tensor dictionary to improve the spectral smoothness of HSI, and introduces tubal row-sparse constraints on tensor coefficients to exploit their inherent sparse structure. It fully considers the relationship between nonlocal modes and tensor decomposition factors. Besides, ADMM algorithm is used to efficiently solve our model. Extensive experimental results show that the SSTSR method can significantly improve the spatial resolution of LR-HSI while preserving the spectral curves.

In the near future, we will improve the performance of our method in three aspects. First, because the spectral reflectance of ground objects is non-negative, limiting the non-negativity of the dictionary has practical physical significance for hyperspectral image super-resolution. Second, our method relies on the estimation of the spectral subspace of the reconstructed nonlocal clustering tensor, and we subsequently consider speeding up the algorithm by adaptively choosing the subspace spectral dictionary size. Furthermore, we will refer to more efficient hyperparameter optimization methods to reduce the computational cost.