Hyperspectral and Multispectral Image Fusion Using Coupled Non-Negative Tucker Tensor Decomposition

Abstract

Fusing a low spatial resolution hyperspectral image (HSI) with a high spatial resolution multispectral image (MSI), aiming to produce a super-resolution hyperspectral image, has recently attracted increasing research interest. In this paper, a novel approach based on coupled non-negative tensor decomposition is proposed. The proposed method performs a tucker tensor factorization of a low resolution hyperspectral image and a high resolution multispectral image under the constraint of non-negative tensor decomposition (NTD). The conventional matrix factorization methods essentially lose spatio-spectral structure information when stacking the 3D data structure of a hyperspectral image into a matrix form. Moreover, the spectral, spatial, or their joint structural features have to be imposed from the outside as a constraint to well pose the matrix factorization problem. The proposed method has the advantage of preserving the spatio-spectral structure of hyperspectral images. In this paper, the NTD is directly imposed on the coupled tensors of the HSI and MSI. Hence, the intrinsic spatio-spectral structure of the HSI is represented without loss, and spatial and spectral information can be interdependently exploited. Furthermore, multilinear interactions of different modes of the HSIs can be exactly modeled with the core tensor of the Tucker tensor decomposition. The proposed method is straightforward and easy to implement. Unlike other state-of-the-art approaches, the complexity of the proposed approach is linear with the size of the HSI cube. Experiments on two well-known datasets give promising results when compared with some recent methods from the literature.

1. Introduction

Hyperspectral imagery utilizes a broad range of the electromagnetic spectrum to obtain information of the imaged scene, allowing better identification of materials or detection of processes. As each band of a HSI contains the spectral response to a narrow interval of the electromagnetic spectrum, it is necessary to collect reflectance from a wider area on the scene, decreasing the spatial resolution of HSIs. HSI acquisition instruments have extensive limitations for capturing high-resolution images, and there is always a tradeoff between spectral resolution and spatial resolution.

Hence, we always need spatial information provided from outside; using an auxiliary image consisting of panchromatic image, multispectral image or an RGB one is a well-known approach. In recent years, there have been several attempts which fuse a high resolution multispectral image (HRMSI) with a low resolution hyperspectral image (LRHSI) [1,2,3,4,5,6,7,8,9,10,11,12,13] to produce a high spatio-spectral resolution HSI. Basically, all fusion approaches can be grouped in the following categories: methods using a Bayesian framework [6,11,14,15,16,17,18], matrix factorization based methods [4,5,12,19,20,21], tensor factorization based methods [1,3,9,22,23,24,25] and deep learning based methods [26,27,28].

Bayesian methods apply the observed LRHSI and HRMSI and some prior information or regularization terms to build up the statistical model for estimating a super-resolution HSI [29]. In [11], a Bayesian framework is introduced, based on a sparse representation along with the alternating direction method of multipliers (ADMM) to solve the fusion problem. In our previous work [15], smooth graph signal modeling is employed to regularize in order to incorporate the spatio-spectral joint structural features of the HSI. The method proposed in [6] uses spectral unmixing and sparse representations in a Bayesian framework to increase the spatial resolution of the HSI. The most important deficiency of the Bayesian framework is that regularization terms are required that should comprehensively represent spatial information of HSIs, which is partially lost by matricizing the HSI.

As an alternative approach, matrix factorization approaches have been widely applied to fuse LRHSI with HRMSI [4,5,12,19,20,21]. These methods describe the targeted high resolution HSI (HRHSI) as a product of a matrix of basis vectors of spectral signatures, learned from the high spectral resolution LRHSI, and a coefficient matrix, estimated from auxiliary data, such as a HRMSI. The method proposed in [4] exploits non-local self-similarity, based on a spatial and spectral sparse representation to estimate both basis vectors and coefficient matrices. Ref. [5] employed a non-negative structured sparse representation to estimate the spectral basis vectors, and a structured sparse representation is utilized to determine the coefficient matrices. In [19,20], the spatial structure of HRHSI is applied as regularization to estimate the super-resolution HSI. The method proposed in [12] applied spectral unmixing to regularize the fusion problem. This method alternately updates spectral basis vectors and coefficient matrices, subject to non- negativity and sum-to-one constraints. As HSIs are essentially non-negative data, non-negative matrix factorization (NMF) frameworks are quite compatible with the observed data. Therefore, the approach proposed in [21] alternately factorizes the LRHSI and HRMSI under non-negativity constraints, and multiplicative update rules (MUR) [30] are engaged to estimate the two factorized matrices in each stage. Of note, in Bayesian frameworks and NMF-based approaches, the 3D data structure is stacked into a matrix form, which causes a loss in the neighborhood structures, smoothness, and continuity characteristics. To avoid this, tensors or multiway arrays have been frequently used in hyperspectral data analysis for the purposes of image classification [31,32,33], data compression [34,35], change detection [36,37], target and anomaly detection [36,38], and denoising [39,40]. Additionally, exploiting the capability of multilinear algebra on the multiway array representations allows more flexibility in choosing constraints and describing data structures. Moreover, more general features can be extracted from the data compared with the matrix-based approaches. More recently, tensor-based representations have been widely used for HSI super-resolution [1,9,22,24,41]. The representation of a HSI as a tensor of order three is a structural and natural model without any loss of information. In most of these methods, a low rank tensor representation is exploited to estimate the HRHSI. It offers the benefits of extreme noise and memory usage reduction, and extraction of discriminative features [41]. Accordingly, two well-known tensor decompositions, the Canonical Polyadic decomposition (CPD) and the Tucker decomposition, are frequently used to estimate HRHSIs [9,22,24,25,31,42]. The former decomposes a tensor into the sum of component rank-one tensors, while the later factors three-dimensional data into the multiplication (mode-n products) of four fractions: a core tensor and three dictionary matrices corresponding to each mode (see Figure 1). In [22], a low rank tensor-train representation is incorporated for HSI super-resolution. In [23], nonlocal sparse tensor factorization is proposed to model non-local self-similarity of HSIs. In [24], a nonlocal coupled CPD framework is used to fuse hyperspectral and multispectral images. A spatio-spectral sparsity prior constraint is imposed on the core tensor in a coupled sparse tensor factorization method in [9]. In [38], a low-rank constraint is applied to the core tensor in the Tucker representation to detect anomalies in HSI. It effectively shows the superposition of a spectral background and the anomaly signatures. A coupled CP decomposition based method is proposed in [43], which fused LRHSI and HRMSI to produce a HRHSI in a tensor framework. Note that the CP-based rank of a tensor is defined as the minimum of rank-one component tensors that are summed to express that tensor, which is unknown and generally not easy to estimate. Meanwhile, in the Tucker representation there is just one component tensor which is called the core tensor. It can comprehensively express the relations between the various modes and multilinear interactions among them.

Moreover, with the recent success of deep learning techniques in various image processing tasks, an increasing research interest arose for deep learning-based image fusion. In [27] a fusion method based on convolutional neural networks (CNN) was proposed, in which the two CNN branches are devoted to spectral features of the LR-HSI and spatial neighborhood features of the HR-MSI, respectively. In [28], an unsupervised deep approach was proposed for a blind fusion of HRMSI and LRHSI, considering no prior assumptions on spatial and spectral degradation functions. Instead, they are modeled with deep network structures. The largest deficiency of the deep learning-based methods is their requirement of huge amounts of training data, which is practically not available. Furthermore, deep learning-based methods have limited generalization ability regarding the different sensor characteristics such as spectral range and the observational models.

AS HSI is naturally non-negative data, it has a strong prior knowledge to apply to the tensor factorization method, which is expected to further improve the performance of the fusion. In this paper, we extend NMF to a tensor framework, applying a non-negative tensor decomposition. Contrary to the conventional NMF-based methods, where the spatial, spectral, or their joint structural features must be additionally imposed as constraints, in this paper, the spatio-spectral joint structures of HSIs are preserved without having to impose constraints. We optimally exploit the spectral properties of the LRHSI and the spatial properties of the HRMSI, by directly applying a coupled (non-negative) Tucker decomposition to both images. We will refer to the proposed method as coupled non-negative Tucker decomposition (CNTD).

Using the Tucker tensor representation, the proposed method comprehensively models multilinear modes of the HSI, where the core tensor precisely expresses the relations between the various modes. Therefore we proposed this tensor framework to benefit from its ability to preserve spatio-spectral joint structures. Regarding the huge amount of data in hyperspectral image processing and analysis, computation efficiency is a significant factor. Thus, we propose an algorithm which is straightforward and easy to implement, with a complexity, linear with the size of the hyperspectral data cube. The main contributions of this paper are highlighted below.

• The application of non-negativity priors to the Tucker tensor decomposition of LRHSI and HRMSI, to estimate spectral and spatial mode-dictionaries in a Tucker model, respectively. To the best of our knowledge, this is the first time that a non-negative Tucker decomposition is used to represent hyperspectral images in a HSI fusion framework.
• The preservation of spatio-spectral joint structures of HSIs without prior knowledge requirements and much lower information losses than matrix frameworks.
• The construction of an algorithm with lower-order complexity than the state-of-the-art.

The remainder of this paper is organized as follows. Some preliminaries on tensors are given in Section 2. Section 3 formulates the HSI-MSI fusion framework. The proposed coupled non-negative tensor decomposition (CNTD) method for super-resolution HSI is introduced in Section 4. The complexity of the proposed method is elaborated in Section 5. Section 6 describes some experimental results on two well-known datasets, Pavia University and Indian Pines. Finally, conclusions and future work are described in Section 7.

2. Preliminaries on Tensors

Let us denote a tensor of order N as $\mathcal{X}\in {ℝ}^{I_1\times I_2\times \dots \times I_N}$, having N indices $i_1,i_2,\dots ,i_N$ and its members by $x_{i_1{i}_2\dots i_N}$ where $1\le i_n\le I_n$. Tensor matricization unfolds a tensor of order N into a matrix. The mode-$n$ matricization of $\mathcal{X}$ reorders the elements of $\mathcal{X}$ to form the matrix ${{\rm X}}_{\left(n\right)}\in {ℝ}^{I_n\times I_{n+1}{I}_{n+2}\dots I_N{I}_1{I}_2\dots I_{n-1}}$. Tensor matricization can be regarded as an extension to matrix vectorization.

The mode-n product of tensor $\mathcal{X}\in {ℝ}^{I_1\times I_2\times \dots \times I_N}$ and matrix $A\in {ℝ}^{J_n\times I_n}$, is defined by $\mathcal{M}=\mathcal{X}{\times }_{n}A\in {ℝ}^{I_1\times I_2\times \dots \times J_n\times \dots \times I_N}$, and entries are calculated by:

$${𝓂}_{i_1\dots i_{n-1}{J}_n{i}_{n+1}\dots i_N}={\displaystyle \sum }_{i_n}{𝓍}_{i_1\dots i_{n-1}{i}_n{i}_{n+1}\dots i_N}{a}_{j_n{i}_n}$$

(1)

The mode-n product $\mathcal{X}{\times }_{n}A$ can also be denoted in matrix form as $M_{\left(n\right)}=A{{\rm X}}_{\left(n\right)}$. The mode-n product has two important properties: (i) the order of multiple mode-n products with different modes is arbitrary:

$$\left(\mathcal{X}{\times }_{m}A\right){\times }_{n}B=(\mathcal{X}{\times }_{n}B){\times }_{m}A \left(m\ne n\right)$$

(2)

and for multiple mode-n products with the same modes, the order is relevant:

$$\left(\mathcal{X}{\times }_{n}A\right){\times }_{n}B=\mathcal{X}{\times }_n\left(BA\right)$$

(3)

The scalar product of two tensors $\mathcal{X},\mathcal{Y}$ indicated as $<\mathcal{X},\mathcal{Y}>={\displaystyle \sum }_{i_1,i_2,\dots ,i_N}{𝓍}_{i_1,i_2,\dots ,i_N}{𝓎}_{i_1,i_2,\dots ,i_N}$. The Frobenius norm of a tensor $\mathcal{X}$ is indicated as ${\Vert \mathcal{X}\Vert }_F=\sqrt{<\mathcal{X},\mathcal{X}>}.$

The Tucker decomposition of $\mathcal{X}\in {ℝ}^{I_1\times I_2\times \dots \times I_N}$ is expressed as mode products of a core tensor $\mathcal{U}\in {ℝ}^{K_1\times K_2\times \dots \times K_N}$ and N mode matrices $V^{\left(n\right)}\in {ℝ}^{I_n\times K_n}$, which is expressed as:

$$\mathcal{X}=\mathcal{U}{\times }_1{V}^{\left(1\right)}{\times }_2{V}^{\left(2\right)}\dots {\times }_N{V}^{\left(N\right)}$$

(4)

which has an element-wise form given by:

$${𝓍}_{i_1\dots i_N}={\displaystyle \sum }_{k_1\dots k_N}{𝓊}_{k_1\dots k_N}{v}_{i_1{k}_1}^{\left(1\right)}{v}_{i_2{k}_2}^{\left(2\right)}\dots v_{i_N{k}_N}^{\left(N\right)}$$

(5)

The mode-n matricization form of $\mathcal{X}$ is expressed by Kronecker products ($⨂$) of the mode-n matricization of the core tensor and mode matrices, as:

$$\begin{gathered} X_{\left(n\right)}=V^{\left(n\right)}{U}_{\left(n\right)}[V^{\left(n-1\right)}⨂\dots ⨂V^{\left(2\right)}⨂V^{\left(1\right)}⨂V^{\left(N\right)} \\ ⨂\dots ⨂V^{\left(n+2\right)}⨂V^{\left(n+1\right)} \end{gathered}$$

(6)

where $U_{\left(n\right)}$ is the mode-n matricization of the core tensor $\mathcal{U}.$ The Kronecker product of two matrices $A\in {ℝ}^{I\times J}$ and $B\in {ℝ}^{K\times L}$ is a matrix denoted by $A⨂B\in {ℝ}^{IK\times JL}$, defined as:

$$A⨂B=\left[\begin{array}{cccc}{\mathit{a}}_{11}\mathbf{B}& {\mathit{a}}_{12}\mathbf{B}& \dots & {\mathit{a}}_{1\mathit{J}}\mathbf{B}\\ {\mathit{a}}_{21}\mathbf{B}& {\mathit{a}}_{22}\mathbf{B}& \dots & {\mathit{a}}_{2\mathit{J}}\mathbf{B}\\ ⋮& ⋮& \ddots & ⋮\\ {\mathit{a}}_{\mathit{I}1}\mathbf{B}& {\mathit{a}}_{\mathit{I}2}\mathbf{B}& \dots & {\mathit{a}}_{\mathit{I}\mathit{J}}\mathbf{B}\end{array}\right]$$

(7)

The following Kronecker product properties and the vectorization operation ($vec(·)$) are used in this paper:

$$\begin{gathered} vec\left(UA{V}^T\right)=\left(V⨂U\right)vec\left(A\right) \\ {\left(V⨂U\right)}^T=V^T⨂U^T \\ \left(V⨂U\right)\left(A⨂B\right)=VA⨂UB \end{gathered}$$

(8)

All basic notations are presented in

Table 1. Basic Notation.

Notation	Description
$\mathcal{X}$	Tensor
X	Matrix
$𝓍$	Tensor element
$\mathbf{𝓍}$	Spectral vector of tensor
X	Scaler
${\times }_n$	Mode-n product
$⨂$	Kronecker product
$⊛$	Hadamard product
${{\rm X}}_{\left(n\right)}$	Mode-n matricization of tensor X
$X^{\left(n\right)}$	n mode matrix in Tucker decomposition

.

3. HSI-MSI Fusion Problem Formulation

In this paper, the HRHSI, LRHSI, and HRMSI are denoted as tensors of order three. The target HRHSI is denoted by $\mathcal{Z}\in {ℝ}^{W\times H\times S}$, where $W$, $H$ and $S$ are the width, height and number of spectral bands, respectively. The LRHSI and HRMSI are denoted by ${\mathcal{Y}}_h\in {ℝ}^{w\times h\times S}$ and ${\mathcal{Y}}_m\in {ℝ}^{W\times H\times s}$, respectively, with $w$ ≪ $W$, $h$ ≪ $H$ and $s$ ≪ $S$. In this paper, we aim at estimating a HRHSI in a fusion framework, based on the observations ${\mathcal{Y}}_h$ and ${\mathcal{Y}}_m$. In this section, we briefly describe the matrix factorization-based fusion scheme and then detail the tensor decomposition-based scheme.

3.1. Matrix Factorization-Based Fusion Scheme

In the matrix factorization-based fusion scheme, each spectral signature of the HRHSI is considered to be a linear mixture of a small number of basis vectors:

$$Z_{\left(3\right)}=EA$$

(9)

where, $Z_{\left(3\right)}\in {ℝ}^{S\times WH}$ is the mode-three matricization of $\mathcal{Z}$. $E\in {ℝ}^{S\times k}$ contains the basis vectors in its columns and $A\in {ℝ}^{k\times WH}$ is the coefficient matrix. Similarly ${\mathit{Y}}_{{\mathit{h}}^{\left(3\right)}}\in {ℝ}^{S\times wh}$ and ${\mathit{Y}}_{{\mathit{m}}^{\left(3\right)}}\in {ℝ}^{s\times WH}$ are the mode-three matricization of ${\mathcal{Y}}_h$ and ${\mathcal{Y}}_m$, respectively. Conventionally, they can be considered as spectral and spatial down-sampled versions of the target HRHSI. Thus:

$$Y_{{\mathit{h}}^{\left(3\right)}}=Z_{\left(3\right)}M$$

(10)

where, $M\in {ℝ}^{WH\times wh}$ is the point spread function (PSF) matrix and the spatial down-sampling in the hyperspectral sensor. It can be separated regarding width and height modes [9]:

$$M={(P_{\mathbf{2}}⨂P_{\mathbf{1}})}^{\mathit{T}}$$

(11)

where $P_1\in {ℝ}^{\mathit{w}\times W}$ and $P_2\in {ℝ}^{\mathit{h}\times H}$ are spatial separable down-sampling operators of the width and height spatial modes, respectively.

Similarly:

$$Y_{{\mathit{m}}^{\left(3\right)}}=P_3{Z}_{\left(3\right)}$$

(12)

where $P_3\in {ℝ}^{s\times S}$ is a matrix, modeling spectral down-sampling in the multispectral sensor. Therefore, the spectral response functions (SRF) of the multispectral sensor is included within its rows.

Commonly, in the HSI-MSI fusion problem formulation, the SRF and PSF are assumed to be known [4,9,11,17,25]. the proposed approach in [19] is also used to approximate the SRF and PSF from observed data. Following (10):

$$P_3{Y}_{{\mathit{h}}^{\left(3\right)}}=P_3{Z}_{\left(3\right)}M$$

(13)

which, according to (12), can also be written as:

$$P_3{Y}_{{\mathit{h}}^{\left(3\right)}}=Y_{{\mathit{m}}^{\left(3\right)}}M$$

(14)

In, an optimization framework, based on quadratic regularizing, is presented to estimate $P_3$ and $M$ from the observed LRHSI and HRMSI:

$$\underset{P_{3,M}}{\min }\Vert P_3{Y}_{{\mathit{h}}^{\left(3\right)}}-Y_{{\mathit{m}}^{\left(3\right)}}{M\Vert }^2+{\mathbf{\alpha }}_{\mathbf{1}}{\Gamma }_1\left(P_3\right)+{\mathbf{\alpha }}_{\mathbf{2}}{\Gamma }_2\left(M\right)$$

(15)

where ${\Gamma }_1(·)$ and ${\Gamma }_2(·)$ are quadratic regularizers, ${\mathit{\alpha }}_{\mathbf{1}}$ and ${\mathit{\alpha }}_{\mathbf{2}}$ are the respective regularization parameters. See [19] for more details.

3.2. Tensor Decomposition-Based Fusion Scheme

As HSIs are naturally 3D data, the tensor is a more efficient representation than a matrix form, and we can benefit from its ability to exploit intrinsic structures of the HSI and multilinear interactions between its different modes. Hence, in this paper, the target HRHSI $\mathcal{Z}$ is formally expressed by:

$$\mathcal{Z}=\mathcal{C}{\times }_{1}W{\times }_{2}H{\times }_{3}S$$

(16)

which is called the Tucker representation, where $W\in {ℝ}^{W\times n_w}$, $H\in {ℝ}^{H\times n_h}$ and $S\in {ℝ}^{S\times n_s}$ are width, height and spectral dictionary matrices, respectively. $n_w$, $n_h$ and $n_s$ are the number dictionary atoms of each mode, and $\mathcal{C}\in {ℝ}^{n_w\times n_h\times n_s}$ is the core tensor that denotes the interactions between several modes.

Mode-n $\left(n=1,2,3\right)$ matricizations of $\mathcal{Z}$ are given by:

$$\begin{gathered} Z_{\left(1\right)}=W{C}_{\left(1\right)}{\left(S⨂H\right)}^T \\ {Z}_{\left(2\right)}=H{C}_{\left(2\right)}{\left(S⨂W\right)}^T \\ {Z}_{\left(3\right)}=S{C}_{\left(3\right)}{\left(H⨂W\right)}^T \end{gathered}$$

(17)

Similarly, both LRHSI and HRMSI are represented by:

$${\mathcal{Y}}_h=\mathcal{C}{\times }_1{W}_{\mathit{h}}{\times }_2{H}_{\mathit{h}}{\times }_{3}S+{\mathcal{E}}_{\mathit{h}}$$

(18)

$${\mathcal{Y}}_m=\mathcal{C}{\times }_{1}W{\times }_{2}H{\times }_3{S}_{\mathit{m}}+{\mathcal{E}}_{\mathit{m}}$$

(19)

where $W_{\mathit{h}}\in {ℝ}^{\mathit{w}\times {\mathit{n}}_{\mathit{w}}}$, $H_{\mathit{h}}\in {ℝ}^{\mathit{h}\times {\mathit{n}}_{\mathit{h}}}$ and ${\mathit{S}}_{\mathit{m}}\in {ℝ}^{\mathit{s}\times {\mathit{n}}_{\mathit{s}}}$ are width, height, and spectral dictionary matrices, respectively, and ${\mathcal{E}}_h\in {ℝ}^{w\times h\times S}$ and ${\mathcal{E}}_m\in {ℝ}^{W\times H\times s}$ denote Independent-Identically-Distributed (i.i.d.) noise of ${\mathcal{Y}}_h$ and ${\mathcal{Y}}_m$, respectively. As the LRHSI and HRMSI are the spatially and spectrally down-sampled form of the HRHSI, respectively, one has:

$$W_h=P_{1}W$$

(20)

$$H_h=P_{2}H$$

(21)

$$S_m=P_{3}S$$

(22)

4. Proposed CNTD Approach

The goal of a LRHSI and HRMSI fusion framework is to estimate a high spatio-spectral target HSI. Since $w\ll W$, $h\ll H$ and $s\ll S$, the super resolution problem is severely ill-posed and prior information is needed to regularize the fusion problem. Orthogonality and statistical independency of basis vectors in the Tucker representation, and sparsity, smoothness, and non-negativity of HSIs are some constraints that help to find a unique solution for the super-resolution problem of HSIs [44,45,46].

In this paper, we propose a new algorithm based on coupled non-negative Tucker decomposition (CNTD). The proposed method performs Tucker tensor decomposition of the LRHSI and HRMSI, subject to NTD. The original NMF method inherently loses spatio-spectral joint structure information when unfolding the 3D data into matrix form. Therefore, in this paper, we impose NTD to both tensors of the HSI and MSI directly. The CNTD method effectively combines multiple data tensors, where the intrinsic spatio-spectral joint structures of HSI can be represented without loss and interdependently exploited. The proposed CNTD approach is depicted in Figure 2, illustrating the fusion of the spatial information of HRMSI and the spectral information of LRHSI to produce the HRHSI.

Considering (16), (18), and (19), the LRHSI and HRMSI fusion problem is formulated by the following constrained least square optimization problems:

$$\begin{gathered} \underset{\mathcal{C},W_{\mathit{h}},H_{\mathit{h}},S}{\min } \Vert {\mathcal{Y}}_h-\mathcal{C}{\times }_1{W}_{\mathit{h}}{\times }_2{H}_{\mathit{h}}{\times }_3{S\Vert }_F^2 \\ s.t. \mathcal{C},W_{\mathit{h}},H_{\mathit{h}},S\ge 0 \end{gathered}$$

(23)

$$\begin{gathered} \underset{\mathcal{C},W, H, S_{\mathit{m}}}{\min }\Vert {\mathcal{Y}}_m-\mathcal{C}{\times }_{1}W{\times }_{2}H{\times }_3{S}_{\mathit{m}}{\Vert }_F^2 \\ s.t. \mathcal{C},W,H,{\mathit{S}}_{\mathit{m}}\ge 0 \end{gathered}$$

(24)

Hyperspectral and multispectral data fusion, based on NTD, is executed by the estimation of the corresponding dictionaries and the core tensor. The non-negative Tucker tensor decomposition is non-convex in its entirety, but it is convex in each of its factors. We easily derive updated rules for each mode-dictionary matrix by matricizing the Tucker model into its corresponding modes. Therefore, these can be considered as conventional NMF for which we use a block coordinate descent scheme. Therefore, the cost functions are optimized with respect to each factor while keeping the others fixed. Traditionally in gradient descent, the learning rates are positive, but since the subtraction of terms in the updated rules can lead to negative elements, Lee and Seung [30] proposed to use adaptive learning rates to avoid subtraction and thus the production of negative elements. Like the conventional NMF, the optimization algorithm is easy to implement and computationally efficient. NTD attempts to decompose a non-negative data tensor into the multilinear products of a non-negative core tensor and non-negative mode-dictionary matrices [47]. To minimize the predefined optimization problems (23) and (24), the multiplicative update rule (MUR) is applied, and can be directly achieved by NMF multiplicative rule, for which the convergence to local optima under the non-negativity constraints has been proven in [30,48].

4.1. Updating Mode-Dictionary Matrices

Updating algorithms for each mode-dictionary matrix can be easily derived by matricizing the Tucker model into its corresponding modes. Lee and Seung’s multiplicative update rule has been a widely known approach owing to the simplicity of its implementation [30]. We use an extended MUR for the mode-dictionary matrices of ${\mathcal{Y}}_h$. The first mode matricization of ${\mathcal{Y}}_h$ is given by:

$$Y_{\mathit{h}}{}_{\left(1\right)}\approx \underset{\begin{array}{c}1st fraction of the\\ conventional NMF\\ \triangleq \alpha \end{array}}{\underbrace{ W_{\mathit{h}} }}\underset{\begin{array}{c}2nd fraction of the\\ conventional NMF\\ \triangleq \beta \end{array}}{\underbrace{C_{\left(1\right)}{\left(S⨂H_{\mathit{h}}\right)}^T}}$$

(25)

where $Y_{\mathit{h}}{}_{\left(1\right)}$ and $C_{\left(1\right)}$ are the first mode matricization of LRHSI (${\mathcal{Y}}_h$) and the core tensor ($\mathcal{C}$), respectively. Equation (25) can be rewritten as $Y_{\mathit{h}}{}_{\left(1\right)}\approx \mathsf{\alpha }\mathsf{\beta }$, just like the conventional NMF, for which the updating rules are given by [49]:

$$\begin{gathered} \mathsf{\alpha }\leftarrow \mathsf{\alpha }⊛\frac{Y_{\mathit{h}}{}_{\left(1\right)}{\mathsf{\beta }}^{\mathit{T}}}{\mathsf{\alpha }\mathsf{\beta }{\mathsf{\beta }}^{\mathit{T}}} \\ \mathsf{\beta }\leftarrow \mathsf{\beta }⊛\frac{{\mathsf{\alpha }}^{\mathit{T}}{Y}_{\mathit{h}}{}_{\left(1\right)}}{{\mathsf{\alpha }}^{\mathit{T}}\mathsf{\alpha }\mathsf{\beta }} \end{gathered}$$

(26)

Equation (25) can be treated as the conventional NMF, where each fraction is updated using MUR (26):

$$W_{\mathit{h}}\leftarrow W_{\mathit{h}}⊛\frac{Y_{\mathit{h}}{}_{\left(1\right)}{\left[C_{\left(1\right)}{\left(S⨂H_{\mathit{h}}\right)}^{\mathit{T}}\right]}^{\mathit{T}}}{W_{\mathit{h}}{C}_{\left(1\right)}{\left(S⨂H_{\mathit{h}}\right)}^{\mathit{T}}{\left[C_{\left(1\right)}{\left(S⨂H_{\mathit{h}}\right)}^{\mathit{T}}\right]}^{\mathit{T}}}$$

(27)

where the fraction line denotes the element-wise division.

Similarly, the second mode matricization of ${\mathcal{Y}}_h$ is given by:

$$Y_{\mathit{h}}{}_{\left(2\right)}\approx \underset{\begin{array}{c}1st fraction of the\\ conventional NMF\end{array}}{\underbrace{ H_{\mathit{h}} }}\underset{\begin{array}{c}2nd fraction of the\\ conventional NMF\end{array}}{\underbrace{C_{\left(2\right)}{\left(S⨂W_{\mathit{h}}\right)}^{\mathit{T}}}}$$

(28)

and the second mode-dictionary matrix is updated as:

$$H_{\mathit{h}}\leftarrow H_{\mathit{h}}⊛\frac{Y_{\mathit{h}}{}_{\left(2\right)}{\left[C_{\left(2\right)}{\left(S⨂W_{\mathit{h}}\right)}^{\mathit{T}}\right]}^{\mathit{T}}}{H_{\mathit{h}}{C}_{\left(2\right)}{\left(S⨂W_{\mathit{h}}\right)}^{\mathit{T}}{\left[C_{\left(2\right)}{\left(S⨂W_{\mathit{h}}\right)}^{\mathit{T}}\right]}^{\mathit{T}}}$$

(29)

Finally, the third mode matricization of ${\mathcal{Y}}_{\mathit{h}}$ is given by $Y_{\mathit{h}}{}_{\left(3\right)}\approx S{C}_{\left(3\right)}{\left(H_{\mathit{h}}⨂W_{\mathit{h}}\right)}^{\mathit{T}}$. Therefore, the spectral mode-dictionary ($S$) is updated as:

$$S\leftarrow S⊛\frac{Y_{\mathit{h}}{}_{\left(3\right)}{\left[C_{\left(3\right)}{\left(H_{\mathit{h}}⨂W_{\mathit{h}}\right)}^{\mathit{T}}\right]}^{\mathit{T}}}{S{C}_{\left(3\right)}{\left(H_{\mathit{h}}⨂W_{\mathit{h}}\right)}^{\mathit{T}}{\left[C_{\left(3\right)}{\left(H_{\mathit{h}}⨂W_{\mathit{h}}\right)}^{\mathit{T}}\right]}^{\mathit{T}}}$$

(30)

4.2. Updating Core Tensor

Applying (8) to (25):

$$vec\left(Y_{\mathit{h}}{}_{\left(1\right)}\right)=vec\left(W_{\mathit{h}}{C}_{\left(1\right)}{\left(S⨂H_{\mathit{h}}\right)}^{\mathit{T}}\right)=\underset{\begin{array}{c}1st fraction of the\\ conventional NMF\end{array}}{\underbrace{\left(S⨂H_{\mathit{h}}⨂W_{\mathit{h}}\right)}}\underset{\begin{array}{c}2nd fraction of the \\ conventional NMF\end{array}}{\underbrace{vec\left(C_{\left(1\right)}\right)}}$$

(31)

which can be treated as the conventional NMF as well. Incorporating MUR to calculate the core tensor ($\mathcal{C}$):

$$\mathit{v}\mathit{e}\mathit{c}\left({\mathit{C}}_{\left(1\right)}\right)\leftarrow \mathit{v}\mathit{e}\mathit{c}\left({\mathit{C}}_{\left(1\right)}\right)⊛\frac{{\left(S⨂H_{\mathit{h}}⨂W_{\mathit{h}}\right)}^{\mathit{T}}vec(Y_{\mathit{h}}{}_{\left(1\right)})}{{\left(S⨂H_{\mathit{h}}⨂W_{\mathit{h}}\right)}^{\mathit{T}}\left(S⨂H_{\mathit{h}}⨂W_{\mathit{h}}\right)vec\left(C_{\left(1\right)}\right)}$$

(32)

Applying (8) to the numerator of (32):

$$\begin{array}{l}{\left(S⨂H_{\mathit{h}}⨂W_{\mathit{h}}\right)}^{\mathit{T}}vec(Y_{\mathit{h}}{}_{\left(1\right)})=\left({\left(S⨂H_{\mathit{h}}\right)}^T⨂W_{\mathit{h}}{}^T\right)vec(Y_{\mathit{h}}{}_{\left(1\right)})\\ =vec\left(W_{\mathit{h}}{}^T{Y}_{\mathit{h}}{}_{\left(1\right)}{\left(S^{\mathit{T}}⨂H_{\mathit{h}}{}^{\mathit{T}}\right)}^{\mathit{T}}\right)\\ =vec\left({({\mathcal{Y}}_{\mathit{h}}{\times }_1{W}_{\mathit{h}}{}^{\mathit{T}}{\times }_2{H}_{\mathit{h}}{}^{\mathit{T}}{\times }_3{S}^{\mathit{T}})}_{\left(1\right)}\right)\end{array}$$

(33)

and its denominator is given by:

$$\begin{array}{l}{\left(S⨂H_{\mathit{h}}⨂W_{\mathit{h}}\right)}^{\mathit{T}}\left(S⨂H_{\mathit{h}}⨂W_{\mathit{h}}\right)vec\left(C_{\left(1\right)}\right)\\ =\left({\left(S⨂H_{\mathit{h}}\right)}^{\mathit{T}}⨂W_{\mathit{h}}{}^{\mathit{T}}\right)\left(\left(S⨂H_{\mathit{h}}\right)⨂W_{\mathit{h}}\right)vec\left(C_{\left(1\right)}\right)\\ =\left[\left({\left(S⨂H_{\mathit{h}}\right)}^{\mathit{T}}\left(S⨂H_{\mathit{h}}\right)\right)⨂\left(W_{\mathit{h}}{}^{\mathit{T}}{W}_{\mathit{h}}\right)\right]vec\left(C_{\left(1\right)}\right)\\ =vec\left(\left(W_{\mathit{h}}{}^{\mathit{T}}{W}_{\mathit{h}}\right)C_{\left(1\right)}\left(S^{\mathit{T}}S⨂{\mathit{H}}_{\mathit{h}}{}^{\mathit{T}}{\mathit{H}}_{\mathit{h}}\right)\right)\\ =vec\left({\left(\mathcal{C}{\times }_1{W}_h^T{W}_h{\times }_2{H}_h^T{H}_h{\times }_3{S}^{T}S\right)}_{\left(1\right)}\right)\end{array}$$

(34)

As a result, the core tensor $\mathcal{C}$ is updated by:

$$\mathcal{C}\leftarrow \mathcal{C}⊛\frac{{\mathcal{Y}}_h{\times }_1{W}_h^T{\times }_2{H}_h^T{\times }_3{S}^T}{\mathcal{C}{\times }_1{W}_h^T{W}_h{\times }_2{H}_h^T{H}_h{\times }_3{S}^{T}S}$$

(35)

When performing MUR for ${\mathcal{Y}}_m$ in a similar way, the following updating relations for ${\mathcal{Y}}_m$ are obtained:

$$W\leftarrow W⊛\frac{Y_{\mathit{m}}{}_{\left(1\right)}{\left[C_{\left(1\right)}{\left(S_{\mathit{m}}⨂H\right)}^{\mathit{T}}\right]}^{\mathit{T}}}{W{C}_{\left(1\right)}{\left(S_{\mathit{m}}⨂H\right)}^{\mathit{T}}{\left[C_{\left(1\right)}{\left(S_{\mathit{m}}⨂H\right)}^{\mathit{T}}\right]}^{\mathit{T}}}$$

(36)

$$H\leftarrow H⊛\frac{Y_{\mathit{m}}{}_{\left(2\right)}{\left[C_{\left(2\right)}{\left(S_{\mathit{m}}⨂W\right)}^{\mathit{T}}\right]}^{\mathit{T}}}{H{C}_{\left(2\right)}{\left(S_{\mathit{m}}⨂W\right)}^{\mathit{T}}{\left[C_{\left(2\right)}{\left(S_{\mathit{m}}⨂W\right)}^{\mathit{T}}\right]}^{\mathit{T}}}$$

(37)

$$S_{\mathit{m}}\leftarrow S_{\mathit{m}}⊛\frac{Y_{\mathit{m}}{}_{\left(3\right)}{\left[C_{\left(3\right)}{\left(H⨂W\right)}^{\mathit{T}}\right]}^{\mathit{T}}}{S_{\mathit{m}}{C}_{\left(3\right)}{\left(H⨂W\right)}^{\mathit{T}}{\left[C_{\left(3\right)}{\left(H⨂W\right)}^{\mathit{T}}\right]}^{\mathit{T}}}$$

(38)

$$\mathcal{C}\leftarrow \mathcal{C}⊛\frac{{\mathcal{Y}}_m{\times }_1{W}^{\mathit{T}}{\times }_2{H}^{\mathit{T}}{\times }_3{S}_{\mathit{m}}{}^T}{\mathcal{C}{\times }_1{\times }_1{W}^{\mathit{T}}W{\times }_2{H}^{\mathit{T}}H{\times }_3{S}_{\mathit{m}}{}^T{S}_{\mathit{m}}}$$

(39)

As the LRHSI and HRMSI contain, respectively, spectral and spatial information of the target image, the spectral mode-dictionary $S$ is initialized using the former and the spatial mode-dictionary matrices of the height H and width W are initialized using the latter. The spectral mode-dictionary $S$ is initialized using the simplex identification split augmented Lagrangian (SISAL) algorithm [50], which efficiently identifies a minimum volume simplex containing the LRHSI spectral vectors. The spatial mode-dictionary matrices W and H are initialized from the mode-one and -two matricization of the HRMSI, respectively, via dictionary update cycles of the KSVD method [51]. The core tensor $\mathcal{C}$ is initialized using the ADMM framework presented in [9].

To fully exploit its spectral information, the proposed algorithm CNTD starts with applying NTD to the LRHSI. $W_h$ and $H_h$ are initialized by (20) and (21), respectively, to inherit the reliable spatial information from the HRMSI, while the other variables are fixed. Then, $W_{\mathit{h}}$, $H_{\mathit{h}}$, $S$ and $\mathcal{C}$ are alternately updated by (27), (29), (30), and (35), respectively, until convergence of the objective function in (23).

The next step of the proposed algorithm is to apply NTD to the HRMSI. $S_m$ is initialized by (22), to exploit the spectral information of the LRHSI. In the optimization phase, $W$, $H$, $S_{\mathit{m}}$ and $\mathcal{C}$ are alternately updated using (36)–(39) while the other variables are fixed, until convergence of the objective function in (24). The super-resolution HSI is calculated using the estimated core tensor and mode-dictionary matrices. Algorithm 1 gives the pseudocode of the proposed CNTD algorithm.

Algorithm 1: The proposed coupled non-negative tensor decomposition method.

Input: LRHSI (${\mathcal{Y}}_h$), HRMSI (${\mathcal{Y}}_m$). Output: HRHSI ($\mathcal{Z}$)

Estimate PSF ($P_1$, $P_2$), SRF ($P_3$), using method from [19].

Initialize the core tensor ($\mathcal{C}$) via ADMM [9], and mode-dictionaries ($W,H,S)$ via DUC KSVD [51].

NTD for ${\mathcal{Y}}_h$ Initialize $W_h$, $H_{h}by$ (20), (21), respectively. Update $W_h$, $H_h$, $S$ and $\mathcal{C}$ alternately by (27), (29), (30) and (35), respectively until convergence of

NTD for ${\mathcal{Y}}_m$ Initialize $S_m$ by (22) Update $W$, $H$, $S_{\mathit{m}}$ and $\mathcal{C}$ alternately by (36)–(39) until convergence of the objective function in (24). Using the estimated $W$, $H$, $S$ and $\mathcal{C}$ to calculate the HRHSI ($\mathcal{Z}$) via Tucker tensor decomposition (16).

5. Computational Complexity

In this section, we analyze the computational complexity of the proposed method. According to Algorithm 1, the proposed method includes two sub-optimization problems, which engage MURs to estimate the NTD of ${\mathcal{Y}}_h$ and ${\mathcal{Y}}_m$. Each sub-optimization problem mainly contains four updating steps. In each step, the heaviest parts are the multiplications of the core matrix with the output of the Kronecker products. For optimizing the width dictionary, this term is given by $C_{\left(1\right)}{\left(S⨂H_{\mathit{h}}\right)}^{\mathit{T}}$, which has a complexity order of $\mathit{O}\left(n_w{n}_h{n}_{s}Sh\right)$. For the height dictionary, the term $C_{\left(2\right)}{\left(S⨂W_{\mathit{h}}\right)}^{\mathit{T}}$ has a complexity order of $\mathit{O}\left(n_w{n}_s{n}_{h}Sw\right)$. For the spectral dictionary, the term $C_{\left(3\right)}{\left(H_{\mathit{h}}⨂W_{\mathit{h}}\right)}^{\mathit{T}}$ has a complexity order of $\mathit{O}\left(n_w{n}_s{n}_{h}hw\right)$. Finally, the highest complexity order of the core tensor is $\mathit{O}\left(n_w{n}_s^2{n}_h^2\right)$. Similarly, the heaviest parts of the second sub-optimization step have complexity orders $\mathit{O}\left(n_w{n}_h{n}_{s}sH\right)$, $\mathit{O}\left(n_w{n}_s{n}_{h}sW\right)$, $\mathit{O}\left(n_w{n}_s{n}_{h}HW\right)$, and $\mathit{O}\left(n_w{n}_s^2{n}_h^2\right)$ for $\mathit{W}$, $\mathit{H}$, ${\mathit{S}}_m$, and $\mathcal{C}$, respectively. Given that, LR-HSI and HR-HSI are the spatial and spectral down-sampled versions of HR-HSI, respectively, $\mathrm{W}$, $\mathrm{H}$ and $\mathrm{S}$ are multiples of $\mathrm{w}$, $\mathrm{h}$ and $\mathrm{s}$, respectively. Therefore, the overall computational complexity of the proposed algorithm can be expressed as:

$$\mathit{O}\left(n_w{n}_h{n}_{s}HS\right)+\mathit{O}\left(n_w{n}_h{n}_{s}WS\right)+\mathit{O}\left(n_w{n}_h{n}_{s}WH\right)+\mathit{O}\left(n_w{n}_h^2{n}_s^2\right)$$

(40)

From (40), one can observe that the overall computational complexity is linear with the size of the HSI cube ($W$, $H$, $S$). This is just the same as that of the conventional NMF algorithm, owing to the fact that each update step of the proposed CNTD method can be considered as an NMF problem. Of note, the complexity of the proposed method outperforms the other state-of-the-art tensor factorization methods [4,9,23].

6. Experimental Observations and Results

6.1. Data Sets

The proposed method CNTD is performed on two well-known data sets, depicted in Figure 3. The first data set is the Pavia University image [52], which is captured by the Reflective-Optics-System-Imaging-Spectrometer (ROSIS) optical sensor upon the Pavia University in Italy. The reference HRHSI is a $120\times 120\times 93$ image with 1.3 m per pixel spatial resolution. The wavelength domain from $430$ to $860$ mm is removed because of low SNR and water vapor absorptions. The $30\times 30\times 93$ LRHSI is produced by applying a Gaussian spatial blurring filter on each band of the reference image, and down-sampling with a factor of four in both width and height directions. The HRMSI of size $120\times 120\times 4$ is produced by filtering the reference image with the IKONOS-like reflectance spectral response function depicted in Figure 4. (see [19] for more details about SRF and PSF).

The Indian Pines image is the second test data set. It was acquired by the NASA Aeronautics-and-Space-Administration-Airborne-Visible-Infrared-Imaging-Spectrometer (AVIRIS) [53] over the Indian Pines scene situated in North-Western Indiana. The reference image of size $120\times 120\times 224$ has a wavelength domain from $400$ nm to $2500$ nm and a $20$ m per pixel spatial resolution. We reduced its number of bands to 185 after removing the water absorption and noisy bands ($1$–$4$, $104$–$115$, $150$–$170$, $223$ and $224$). The LRHSI of size $30\times 30\times 185$ was constructed after down-sampling and application of the same blurring filters as which were applied to the first data set. The LANDSAT 7-like spectral response function, depicted in Figure 4, is engaged to produce a HRMSI of size $120\times 120\times 6$. The reference, LRHSI and HRMSI of both data sets are depicted in Figure 3.

6.2. Evaluation Criteria

The performance of the proposed method is evaluated using five different indices. The first index, which is a measure of spectral distortion, is the spectral angle mapper (SAM) in degrees. The angle between pixel spectral responses ${\widehat{\mathbf{𝔃}}}_j$ and ${\mathbf{𝔃}}_j$ of the estimated HRHSI ($\widehat{\mathcal{Z}}$) and the reference image ($\mathcal{Z}$) are calculated, then averaged over all pixels. It is expressed as:

$$\mathrm{SAM}\left(\mathcal{Z},\widehat{\mathcal{Z}}\right)=\frac{1}{WH}{\displaystyle \sum }_{j=1}^{WH}\arccos \left(\frac{{\widehat{\mathbf{𝔃}}}_j^T\widehat{\mathbf{𝔃}}}{\Vert {\widehat{\mathbf{𝔃}}}_j{\Vert }_2 \Vert {\mathbf{𝔃}}_j{\Vert }_2}\right)$$

(41)

the ideal SAM value being zero.

The second index is the root mean squared error (RMSE) that evaluates the quality of the estimated HRHSI ($\widehat{\mathcal{Z}}$), compared to the reference image ($\mathcal{Z}$). It is defined as:

$$\mathrm{RMSE}\left(\mathcal{Z},\widehat{\mathcal{Z}}\right)=\sqrt{\frac{\Vert \mathcal{Z}-{\widehat{\mathcal{Z}}\Vert }_F^2}{WHS}}$$

(42)

The third evaluation index is the relative dimensionless global error in synthesis (ERGAS), which measures the spectral quality of the estimated HRHSI, and is defined as:

$$\mathrm{ERGAS}=100\frac{WH}{wh}\sqrt{\frac{1}{S}{\displaystyle \sum }_{i=1}^S{\left(\frac{\left(\mathrm{RMSE}\left({\widehat{Z}}_{i,:},Z_{i,:}\right)\right)}{{\mu }_{Z_{i,:}}}\right)}^2}$$

(43)

where ${\widehat{Z}}_{i,:}$ and $Z_{i,:}$ denote the ith band of $\widehat{\mathcal{Z}}$ and $\mathcal{Z}$, respectively. ${\mu }_{Z_{i,:}}$ is the mean of $Z_{i,:}$. A lower ERGAS value means a lower spectral distortion between the estimated HRHSI ($\widehat{\mathcal{Z}}$) and the reference image ($\mathcal{Z}$). In the case of perfect reconstruction, it is zero.

The degree of the distortion (DD) is the fourth index, defined as:

$$\mathrm{DD}\left(\mathcal{Z},\widehat{\mathcal{Z}}\right)=\frac{1}{WHS}vec\Vert \left(\mathcal{Z}\right)-vec{\left(\widehat{\mathcal{Z}}\right)\Vert }_1$$

(44)

where $\Vert {\cdot \Vert }_1$ is ${\ell }_1$ norm, and $vec\left(\mathcal{Z}\right)$ and $vec\left(\widehat{\mathcal{Z}}\right)$ are the vectorization of tensors $\mathcal{Z}$ and $\widehat{\mathcal{Z}}$, respectively. The smaller the value of DD, the lower the spectral distortion.

The fifth index is the universal image quality index (UIQI) [54]. It is calculated by averaging over $32\times 32$ windows. The UIQI between the ith band of $\widehat{\mathcal{Z}}$ and $\mathcal{Z}$ is calculated by:

$$\mathrm{UIQI}\left(Z_i,{\widehat{Z}}_i\right)=\frac{1}{d}{\displaystyle \sum }_{j=1}^d\frac{{\sigma }_{Z_j^i{\widehat{Z}}_j^i}}{{\sigma }_{Z_j^i}{\sigma }_{{\widehat{Z}}_j^i}}\frac{2{\mu }_{Z_j^i}{\mu }_{{\widehat{Z}}_j^i}}{{\mu }_{Z_j^i}+{\mu }_{{\widehat{Z}}_j^i}}\frac{2{\sigma }_{Z_j^i}{\sigma }_{{\widehat{Z}}_j^i}}{{\sigma }_{Z_j^i}+{\sigma }_{{\widehat{Z}}_j^i}}$$

(45)

where d is the number of windows, ${\widehat{Z}}_j^i$ and $Z_j^i$ denote the jth window of the ith band of $\widehat{\mathcal{Z}}$ and $\mathcal{Z}$, respectively, ${\sigma }_{Z_j^i{\widehat{Z}}_j^i}$ is the sample covariance between $Z_j^i$ and ${\widehat{Z}}_j^i$, ${\mu }_{Z_j^i}$ and ${\sigma }_{Z_j^i}$ are the mean and standard deviations of $Z_j^i$, respectively. The UIQI index between $\widehat{\mathcal{Z}}$ and $\mathcal{Z}$ is the average UIQI value of all bands, which is:

$$\mathrm{UIQI}\left(\mathcal{Z},\widehat{\mathcal{Z}}\right)=\frac{1}{S}{\displaystyle \sum }_{i=1}^S\mathrm{UIQI}\left(Z_{i,:},{\widehat{Z}}_{i,:}\right)$$

(46)

The ideal UIQI value is one.

All of the experiments were performed using MATLAB version R2016a, and have been run by a computer with an Intel Core i5 central processing unit (CPU) at 3.4 GHz and 32 GB random access memory (RAM).

6.3. Evaluation of the Parameters

In order to evaluate the sensitivity of the proposed CNTD approach w. r. t. and its essential parameters, i.e., the number of mode (width, height, and depth) dictionary atoms $n_w$, $n_h$ and $n_s$, the proposed method has been performed for different numbers of mode-dictionary atoms. Figure 5a–c shows the RMSE of the estimated Pavia University and Indian Pines data sets as functions of the number of mode-dictionary atoms $n_w$, $n_h$ and $n_s$, respectively. As Figure 5a,b shows, the RMSE of both data sets strongly declines when $n_w$ and $n_h$ increase from 5 to 200. After that, the RMSE does not improve any further. Therefore, we chose to set the values of $n_w$ and $n_h$ to 167 for both data sets for all remaining experiments. As can be seen from Figure 5c, the RMSE for Pavia University decreases while $n_s$ increases from 5 to 40. For Indian Pines, the RMSE decreases as $n_s$ increases from 5 to 100, after which it does not decrease any further. Hence, we set $n_s$ to 30 for both aforementioned data sets. That the proposed CNTD method requires larger width and height mode-dictionaries compared to the spectral mode-dictionary is because generally the HSIs spectral vectors belong to a much lower dimension subspace.

6.4. Comparison with State of the Art Fusion Methods

The proposed fusion method is compared with state-of-the-art methods. In order to see the impact of non-negative priors, the CNTD method is compared with tensor decomposition methods CSFT [9] and NLSTF [23], which do not include non-negativity. Additionally, the proposed CNTD approach is compared with the well-known matrix framework CNMF [21], aiming to evaluate the ability of the proposed CNTD approach to preserve the spatio-spectral structure. Furthermore, our Tucker based method is compared with a CP tensor decomposition method, referred to as STEREO [43]. Finally, we have compared with the two-branched CNN method of [28].

The experiments validate the superiority of the proposed CNTD method on three aspects: the advantage of non-negative priors, the ability to preserve the spatio-spectral structure, and the computational complexity. RMSE, SAM, DD, ERGAS and UIQI for all approaches are shown in

Table 2. Quantitative metrics of the different fusion methods on the Pavia University data set.

Method	Pavia University Data Set
	RMSE	SAM	DD	ERGAS	UIQI
Ideal value	0.000	0.000	0.000	0.000	1.000
CNMF [21]	0.140	4.313	0.017	4.989	0.952
CSTF [9]	2.160	2.390	1.055	1.230	0.991
NLSTF [23]	1.452	0.964	0.846	0.520	0.993
CNN [27]	0.016	2.203	0.103	1.447	0.976
STEREO [43]	0.061	3.922	0.010	1.865	0.989
CNTD method	0.008	1.963	0.005	1.169	0.996

and

Table 3. Quantitative metrics of the different fusion methods on the Indian Pines data set.

Method	Indian Pines Data Set
	RMSE	SAM	DD	ERGAS	UIQI
Ideal value	0.000	0.000	0.000	0.000	1.000
CNMF [21]	0.054	2.142	0.008	1.789	0.954
CSTF [9]	1.533	1.363	0.997	1.082	0.974
NLSTF [23]	0.899	0.768	0.484	0.755	0.984
CNN [27]	0.013	2.270	2.090	1.060	0.820
STEREO [43]	0.042	2.303	0.007	2.538	0.932
CNTD method	0.009	1.661	0.006	1.249	0.972

for the Pavia University and Indian Pines data sets, respectively. To evaluate the computational complexity of the proposed method in comparison with the matrix-based method and the other tensor frameworks, the computation times of CNMF, CSTF, NLSTF and the proposed CNTD are shown in

Table 4. Computational time (s).

Method	Pavia University Data Set	Indian Pines Data Set
CNMF [21]	8.283	10.660
NLSTF [23]	14.017	16.503
CSTF [9]	90.191	92.660
CNTD method	6.301	9.508

. The best values are depicted in bold.

It can be observed from Table 2 and Table 3 that the proposed method outperforms the other competing methods in terms of RMSE, DD, and UIQI indices, and shows promising results for the other indices. The proposed method outperforms CNMF, CNN and STEREO in almost all of the indices on both data sets. Of note, the efficiency of the CNN method highly depends on the training sample rate, while a huge amount of hyperspectral training data is practically unavailable. Furthermore, the proposed CNTD method outperforms the CSFT for most indices on both data sets. It also has better values of RMSE, DD, and UIQI than NLSTF.

As can be observed from Table 4, the proposed method is much more computationally efficient than the competing tensor-based approaches, and is comparable to the CNMF method. This agrees with the detailed description of the computational complexity in Section V. In contrast to the proposed method, for some of the state-of-the-art methods, such as CSTF and NLTF, the complexity increases faster than that which is linear with the size of the HSI cube.

In order to validate the performance with respect to preserving spatial structures, in Figure 6, band 30 of the LRHSI and the estimated HRHSI with CNMF, CSTF, NLSTF, CNN, STEREO and the proposed CNTD are compared with the reference HRHSI. It can be observed that the proposed CNTD approach can correctly estimate most of the spatial details of the HRHSI, though there are a few distortions in the fusion results. Additionally, the error images of band 30, which reflect the differences between the estimated HRHSI and the reference image of both data sets are shown in Figure 7. The error images of the LRHSI, CNMF, CSTF, NLSTF, CNN, STEREO and the proposed CNTD are depicted. The proposed approach estimates spatial details of the HRHSI with much lower error than the competing methods. With CNMF and CNN, the edge structures of the HSI are lost, while CSTF, NLSTF and STEREO suffer from errors in homogeneous regions. The proposed approach performs better in preserving the spatial structures of HSIs at both edges and homogeneous regions.

7. Conclusions

The main objective of this paper was to extend the matrix formulation of non-negative matrix factorization to a tensor framework for the purpose of hyperspectral and multispectral image fusion. We proposed a coupled non-negative tensor decomposition approach that can be treated as a conventional NMF-based model. The proposed approach performs a tucker tensor factorization of a LRHSI and a HRMSI under the constraint of non-negative tensor decomposition. Unlike other state of the art methods, the complexity of the proposed approach is linear with the size of the HSI cube. The proposed approach gives competitive results with the state-of-the-art fusion approaches. As a future plan, we will incorporate prior information, such as spectral self-similarity, sparsity, smoothness and local consistence in the non-negative tensor decomposition, in order to find better unique basis vectors for the Tucker representation. Furthermore, in this paper we assumed the SRF and PSF to be known and also two input images considered to be registered. Therefore, we will try to overcome these limitations in our future work.