Hyperspectral Image Inpainting Based on Robust Spectral Dictionary Learning

Abstract

To address the problems of defective pixels and strips in hyperspectral images affecting subsequent processing and applications, we modeled the hyperspectral image (HSI) inpainting problem as a sparse signal reconstruction problem with incomplete observations using the theory of sparse representation, and proposed an HSI inpainting algorithm based on spectral dictionary learning. First, we studied the HSI observation model under the assumption of additive noise. We subsequently proposed a new algorithm for constructing a spectral dictionary directly from hyperspectral data by introducing an online learning optimization method and performing dictionary learning using a robust function. Afterwards, the image was sparsely encoded by applying the variable decomposition and augmented Lagrangian sparse regression method. Finally, the inpainted HSI was obtained by sparse reconstruction. The experimental results showed that compared with the existing algorithms, the algorithm proposed herein could effectively inpaint the defective HSI under different noise conditions with a shorter calculation time than those of existing methods and other dictionary learning inpainting algorithms.

1. Introduction

In the operation of hyperspectral imaging systems, sensor malfunctions can create missing pixels, missing scan lines and other anomalies in the acquired hyperspectral image (HSI), resulting in incomplete observations of the target area. This can significantly affect subsequent processing and applications of the HSI. Therefore, it is necessary to restore the complete image information in the preprocessing of the HSI. In the field of digital image processing, the process of restoring the missing data in an image is called image inpainting [1]. Since the emergence of image inpainting problems, several algorithms have been widely studied and applied, such as interpolation methods [2,3], partial differential equation methods [4,5], total variation methods [6,7,8], texture synthesis [9], and the Huber-Markov method [10,11]. All these algorithms were first proposed and applied for two-dimensional images and were mainly applied for the restoration of small areas. They were not suited for large areas of missing pixels. An improved partial differential equation method was applied to hyperspectral imagery [12] and preserving edge sharpness, but it suffered from excessive blurring. A hyperspectral image restored by the total variation method was prone to exhibiting a step effect, which was not beneficial for the subsequent processing of the image.

As sparse representation theory has been widely applied in the field of image and signal processing, its powerful processing ability has attracted the attention of many researchers [13,14]. Sparse representation methods have also been introduced into image inpainting problems [15,16,17]. The theoretical basis is the new theory of compressed sensing. Specifically, sparse or compressible signals in some transform domains can be reconstructed from a small amount of incomplete information.

To better solve the HSI inpainting problem, it was proposed that high-precision sparse reconstruction optimization could be achieved by constructing an appropriate over-complete dictionary [18]. Many scholars have studied the construction of over-complete dictionaries for image inpainting. Among them, the most classic example is that of Elad and Aharon [19]. They proposed the use of a K-SVD algorithm in image inpainting, with the over-complete dictionary obtained from a preliminary training process. In the K-SVD dictionary training process, the sparse coding in each iteration was obtained by the block coordinate relaxation algorithm (BCR), and the update of the dictionary atom was performed by the eigenvalue decomposition method. To minimize the objective cost function under certain constraints, each iteration of the dictionary learning accessed all elements in the training set, a batch processing method based on the second iteration. This feature made the process too computationally intensive to be applied to large-scale data sets, such as HSIs. In addition, the algorithm assumed that certain noise parameters, such as the variance, were known, and the image inpainting accuracy in practical applications was quite sensitive to the errors of estimated variance [17]. Zhou et al. [17] proposed a nonparametric Bayesian dictionary learning method and applied it to HSI inpainting problems. The algorithm treated dictionary learning as a factor analysis problem, in which the factor loading corresponded to dictionary atoms and adaptively exploited the potential spectral correlation between different spectral bands using beta-process factor analysis (BPFA). The dictionary set was adaptively updated by Gibbs sampling of the data to be processed and appeared as an approximation of the total posteriori probability. The method used the spectral correlation of the HSIs to achieve better inpainting. However, BPFA must acquire noise or residual variance as a priori knowledge, and each iteration step must access all the elements in the data set. Thus, this process is computationally intensive. Based on BPFA, Shen et al. [20] proposed an adaptive spectrally weighted sparse Bayesian dictionary learning method to solve the HSI inpainting problems. Through BPFA with a framework of compressive sensing, the algorithm adaptively used the potential spectral correlation between different spectral bands. Because the algorithm was improved based on the BPFA method, it also has the drawbacks of requiring noise parameters as a priori knowledge and being computationally intensive.

To quickly acquire the training dictionary, an online dictionary learning algorithm was proposed previously [21]. The algorithm only used one element or a subset of the data set for each iteration instead of all the elements in the training set. It was an effective statistical approximation of the batch processing method. The algorithm successfully reduced the computational complexity of the dictionary learning by using an online scheme, but it was still sensitive to noise and outliers. Hao et al. [22] extended the dictionary learning algorithm to the complex domain to solve the noise reduction problem of interferometric synthetic aperture radar (InSAR) images.

For actual images, the high levels of noise and the existence of outliers are still major challenges for the current online dictionary learning algorithms. At present, existing dictionary-based inpainting algorithms typically use the $l_2$ norm as the data fitting term in the objective function. Thus, the dictionary update process is easily disrupted by noise and outliers. The optimization process is not robust enough to handle outliers, and it is often difficult to obtain satisfactory results in the inpainting of HSIs.

In view of the large volume of hyperspectral data and the presence of high noise levels and outliers, we adopted an online scheme of dictionary learning in this work to introduce a robust function into the objective function of the dictionary update. Furthermore, we used the hyperspectral data of the image to be inpainted to construct an adaptive dictionary that was suitable for the image inpainting problem. The introduction of robust functions could enhance the robustness of the inpainting process against noise and outliers. In addition to the large data size, HSIs were also formed in a special way, resulting in three-dimensional cubic data. Because all the dictionary learning algorithms currently in use were developed from 2D images, the hyperspectral data are usually expanded into pixel spectral matrices for processing. However, algorithms such as BPFA and K-SVD still use processing methods for 2D images that select image blocks in the pixel spectral matrix as the training set for dictionary learning, and they do not make good use of the high correlation between the bands of hyperspectral data. For this reason, according to [23], we used pixel spectral vectors instead of the traditional image blocks as the training data for dictionary in this work, and the obtained dictionary dimension was the same as the image spectral dimension. Because the dictionary atom could be regarded as the spectral curve that constituted the HSI, the correlation between the spectral bands was better used, and the sparse coding obtained by the dictionary was more consistent with the physical meaning of the pixel. Thus, the inpainting effects were also better.

According to aforementioned analysis, in this paper, we proposed an HSI inpainting algorithm based on online robust spectral dictionary learning, termed INORSDL. The image to be restored was processed with methods based on sparse domain modeling to finally obtain a satisfactory inpainting result.

The main contributions of this paper are as follows:

• We model the HSI inpainting as a reconstruction problem of sparse signals under incomplete observation, with the theory of dictionary learning and sparse representation. To improve the efficiency of the dictionary learning, different from block loading algorithms, the spectral dictionary was trained progressively using the pixel spectral vectors waiting to be trained.
• To enhance the robustness of the dictionary updating process against noise and outliers and improve the inpainting effect of HSIs, we use more robust $l_1$ loss as the data fitting term in the objective function when performing the dictionary learning.

The contents of this paper are divided into five parts. Section 1 briefly introduces the background and current status of the research topic. In Section 2, we present the mathematical inpainting model and the inpainting mechanism. In Section 3, the robust spectral dictionary learning algorithm and the robust sparse coding approach are explained. Section 4 formally outlines INORSDL, an inpainting approach based on online robust dictionary learning and sparse coding in the spectral domain. In Section 5, we present the experimental results of the proposed approach, compared with the state of art HSI inpainting algorithms. The final section concludes this paper.

2. HSI Inpainting Model and Mechanism

2.1. HSI Inpainting Model

The HSI inpainting problem can be modeled as a reconstruction problem of signals acquired under incomplete observation. Assuming that the noise is additive, the inpainting model can be expressed as follows:

$$\mathit{y}=\mathit{M}\mathit{x}+\mathit{n},$$

(1)

where $\mathit{x}\in {\mathbf{R}}^{n_v}$ is the vectorized form of column stacking of clean and defect-free HSI $\mathit{X}\in {\mathbf{R}}^{L\times n_s}$, or $\mathit{x}=\mathrm{vec}\left(\mathit{X}\right)$. Therefore, $n_v=L\times n_s$, $L$ is the number of spectral bands in the HSI and $n_s$ is the number of pixel spectral vectors. Vector $\mathit{M}\in {\mathbf{R}}^{n_m\times n_v}$ is a mask chosen for the observable portion of X that consists of two elements, 0 and 1. There is only one value of 1 in each row, and $n_m\le n_v$. Finally, $\mathit{y}\in {\mathbf{R}}^{n_m}$ is the vector form of the observed incomplete HSI, and the additive noise is $\mathit{n}\in {\mathbf{R}}^{n_m}$.

In this paper, we assume that the mask matrix $\mathit{M}$ is user-provided. In addition, $\mathit{y}$ is also known. Thus, the image inpainting problem is converted into a reconstruction problem for $\mathit{x}$.

2.2. HSI Inpainting Mechanism

In recent years, sparse representations have become a popular research topic for signal recovery and reconstruction [24]. The principle is that because a real image or signal has low rank attributes, it can be assumed that a pure defect-free image or signal is a linear combination of a few atoms in the dictionary. Based on the above assumptions, image inpainting can be considered to be a sparse signal reconstruction task supported by a dictionary.

Because HSIs are sparse, a pure defect-free image can be expressed as follows:

$$\mathit{X}=\mathit{D}\mathit{\alpha },$$

(2)

where $\mathit{D}\in {\mathbf{R}}^{L\times k}$ is the spectral dictionary matrix, $k$ is the number of atoms in the dictionary, $\mathit{\alpha }\in {\mathbf{R}}^{k\times n_s}$ is the corresponding sparse coding matrix, and only a few elements in each column are non-zero.

Substituting Equation (2) into Equation (1) yields the following:

$$\begin{array}{ll}\mathit{y}& =\mathit{M}\cdot \mathrm{vec}\left(\mathit{D}\mathit{\alpha }\right)+\mathit{n}\\ & =\mathit{M}\left(\mathit{I}\otimes \mathit{D}\right)\cdot \mathrm{vec}\left(\mathit{\alpha }\right)+\mathit{n}\end{array}$$

(3)

Here, $\otimes$ is the Kronecker product operation. Therefore, the HSI inpainting problem can be expressed as follows:

$$\underset{\mathrm{vec}\left(\mathit{\alpha }\right)}{\min }\frac{1}{2}{\Vert \mathit{M}\left(\mathit{I}\otimes \mathit{D}\right)\cdot \mathrm{vec}\left(\mathit{\alpha }\right)-\mathit{y}\Vert }_2^2+\lambda \cdot \varphi \left(\mathrm{vec}\left(\mathit{\alpha }\right)\right).$$

(4)

We next address the problem of solving for the pixel spectral vector. Letting ${\mathit{M}}_i$ be a submatrix of $\mathit{M}$ acting on the pixel spectral vector $i$,

$${\mathit{y}}_{o,i}={\mathit{M}}_i{\mathit{x}}_i+{\mathit{n}}_{o,i},$$

(5)

where ${\mathit{y}}_{o,i}\in {\mathbf{R}}^{n_i}$ is the observation vector of the pixel spectral vector $i$, and $n_i$ represents the observable components in the pixel spectral vector, namely the number of spectra. Therefore, $n_i\le L$, ${\mathit{x}}_i$ is the i-th column vector of the pixel spectral matrix X with $i=1,\dots ,n_s$, and ${\mathit{n}}_{o,i}\in {\mathbf{R}}^{n_i}$ is the corresponding noise component.

A pure defect-free pixel spectral vector can be expressed as follows:

$${\mathit{x}}_i=\mathit{D}{\mathit{\alpha }}_i,$$

(6)

where α_i is the i-th column vector of the sparse coding matrix α, and $i=1,\dots ,n_s$.

Substituting Equation (6) into Equation (5) yields the following:

$${\mathit{y}}_{o,i}={\mathit{M}}_i\mathit{D}{\mathit{\alpha }}_i+{\mathit{n}}_{o,i}.$$

(7)

Letting ${\mathit{\Psi }}_i={\mathit{M}}_i\mathit{D}$, Equation (7) can be expressed as follows:

$${\mathit{y}}_{o,i}={\mathit{\Psi }}_i{\mathit{\alpha }}_i+{\mathit{n}}_{o,i}.$$

(8)

Therefore, the optimization problem (4) can be rewritten as an objective function in the form of the least absolute shrinkage and selection operator (LASSO):

$$\underset{{\mathit{\alpha }}_i}{\min }{\displaystyle \sum _i\frac{1}{2}{\Vert {\mathit{\Psi }}_i{\mathit{\alpha }}_i-{\mathit{y}}_{o,i}\Vert }_2^2+\lambda {\Vert {\mathit{\alpha }}_i\Vert }_1}.$$

(9)

In Equation (9), the squared term is the data fitting term, the $l_1$ norm term induces sparse coding, and the regularization parameter $\lambda$ is the weight between the two terms in the objective function.

In real images, the HSI data often contains noise and outliers, and the squared data fitting term used in Equation (9) is sensitive to noise and prone to large deviations. Therefore, we introduce a robust objective function to enhance the robustness of the image inpainting process against noise and outliers, i.e., we convert Equation (4) into an optimization of the following minimization problem:

$$\underset{{\mathit{\alpha }}_i}{\min }{\displaystyle \sum _i{\Vert {\mathit{\Psi }}_i{\mathit{\alpha }}_i-{\mathit{y}}_{o,i}\Vert }_1^1+\lambda {\Vert {\mathit{\alpha }}_i\Vert }_1}.$$

(10)

According to previous discussions [25,26], the robust function error term ${\Vert {\mathit{\Psi }}_i{\mathit{\alpha }}_i-{\mathit{y}}_{o,i}\Vert }_1^1$ makes the data fit less susceptible to the influence of noise or outliers.

The robust sparse coding ${\widehat{\mathit{\alpha }}}_i$ is obtained by solving the optimization problem given by Equation (10) and reconstructing the unobserved parts of each pixel spectral vector to obtain the following:

$${\widehat{\mathit{x}}}_i=\mathit{D}{\widehat{\mathit{\alpha }}}_i,i=1,\dots ,n_s.$$

(11)

Therefore, the inpainted HSI is expressed as $\widehat{\mathit{X}}=\left[{\widehat{\mathit{x}}}_1,{\widehat{\mathit{x}}}_2,\dots ,{\widehat{\mathit{x}}}_{n_s}\right]$.

The process described above uses the sparse representation method to inpaint the hyperspectral remote sensing image. The most critical steps are the acquisition of the spectral dictionary $\mathit{D}$ and the solution of the robust sparse coding ${\widehat{\mathit{\alpha }}}_i$.

3. Robust Spectral Dictionary Learning and Sparse Representation

3.1. Robust Spectral Dictionary Learning

The spectral dictionary $\mathit{D}$ used to solve the sparse coding ${\widehat{\mathit{\alpha }}}_i$ and the final inpainted HSI in this work were obtained by adaptive training, i.e., by a learning method.

Given the large volume of HSI data, dictionary-based learning methods, such as batch processing, result in computational intensiveness, and a low learning efficiency. Therefore, we adopted an online learning method to train the dictionary. This method used only one element or a subset of the data set per iteration. Because dictionary $\mathit{D}$ was regarded as a combination of the statistical parameters X of the observation data during the training process, an update of $\mathit{D}$ did not require complete historical information. Thus, it was not necessary to record each iteration and process all the elements during the online processing. The online dictionary learning method is suitable for large-scale and dynamic data processing, which can greatly improve the computational efficiency and reduce memory overhead.

In addition, to enhance the robustness against noise and outliers, a robust function as a data fitting term of the objective function was introduced for online updating in the dictionary learning process.

Due to the special structure of the HSI data, it was necessary to adjust and improve the existing online dictionary learning algorithm. To make full use of the correlation between the spectral bands of the HSIs, the pixel spectrum vector of the completely observed portion of the HSI was used instead of the image block as the training data for the dictionary learning.

The training process of the dictionary aimed to find the coding base of the information in the HSI to be inpainted. Thus, for a better inpainting effect, it was necessary to ensure that the coding of the information contained in each pixel spectral vector of the image carried a high degree of sparseness. Therefore, the spectral dictionary obtained through the learning was an over-complete dictionary, i.e., $k>>L$.

For a pixel spectral vector with a complete observation in the HSI to be inpainted, the following equation must be applied:

$${\tilde{\mathit{y}}}_{c,q}={\mathit{M}}_{c,q}\mathit{D}{\tilde{\mathit{\alpha }}}_q+{\tilde{\mathit{n}}}_{c,q},$$

(12)

where ${\tilde{\mathit{y}}}_{c,q}\in {\mathbf{R}}^L$ is a pixel spectral vector $q$ that can be observed completely, ${\tilde{\mathit{\alpha }}}_q$ is the corresponding sparse coding, ${\tilde{\mathit{n}}}_{c,q}\in {\mathbf{R}}^L$ is the corresponding noise component, $q=1,\dots ,n_c$ and $n_c$ is the number of pixel spectral vectors with complete observations in the image to be processed. Because ${\tilde{\mathit{y}}}_{c,q}$ was observed completely, ${\mathit{M}}_{c,q}=\mathit{I}$ and Equation (12) was simplified as follows:

$${\tilde{\mathit{y}}}_{c,q}=\mathit{D}{\tilde{\mathit{\alpha }}}_q+{\tilde{\mathit{n}}}_{c,q}.$$

(13)

For a set of completely observed pixel spectral vectors in a given HSI, the robust dictionary learning problem was expressed under the frame of regularization:

$$\underset{\mathit{D}\in C,{\tilde{\mathit{\alpha }}}_1,\dots ,{\tilde{\mathit{\alpha }}}_{n_c}}{\min }{\displaystyle \sum _{q=1}^{n_c}{\Vert {\tilde{\mathit{y}}}_{c,q}-\mathit{D}{\tilde{\mathit{\alpha }}}_q\Vert }_1^1+\lambda {\Vert {\tilde{\mathit{\alpha }}}_q\Vert }_1}.$$

(14)

Equation (14) is an optimization problem. The objective function is constructed from a $l_1$ norm fitting error and a sparsity-inducing term. The weight between the two is determined by the regularization parameter $\lambda \left(>0\right)$, and $n_c$ is the number of pixels in the training set. To prevent the dictionary atom in $\mathit{D}$ from tending toward infinity due to the effect of the $l_1$ norm regularization term, we imposed a constraint $\mathit{D}\in C$, $C=\left\{\mathit{D}\in {\mathbf{R}}^{L\times k}:d_j^T{d}_j\le 1,j=1,\dots ,k\right\}$, which may be viewed as a constraining convex set matrix.

For the joint optimization problem of dictionary $\mathit{D}$ and sparse solution $\tilde{\mathit{\alpha }}=\left[{\tilde{\mathit{\alpha }}}_1,\dots ,{\tilde{\mathit{\alpha }}}_{n_c}\right]$ for Equation (14), a more convenient solution is to alternately fix the values of $\mathit{D}$ and ${\tilde{\mathit{\alpha }}}_q$ and subsequently solve for the minimum value of the other, i.e., to alternately solve and optimize $\mathit{D}$ and ${\tilde{\mathit{\alpha }}}_q$. Because there is a non-smooth term in Equation (14), its solution is somewhat challenging.

Due to the large size of the HSI data, the cost of optimizing $\tilde{\mathit{\alpha }}=\left[{\tilde{\mathit{\alpha }}}_1,\dots ,{\tilde{\mathit{\alpha }}}_{n_c}\right]$ is very high. Therefore, we used the idea of an online method in which only one subset of a randomly selected pixel spectral vector set was trained in each iteration. For each new element in the subset, we first computed the robust sparse coding and subsequently performed a dictionary update.

First, we solved the problem of the robust sparse coding in a given dictionary, i.e., for a given dictionary matrix $\mathit{D}$, we optimized each element ${\tilde{\mathit{\alpha }}}_q$ in $\tilde{\mathit{\alpha }}=\left[{\tilde{\mathit{\alpha }}}_1,\dots ,{\tilde{\mathit{\alpha }}}_{n_c}\right]$ and expressed it as follows:

$${\tilde{\mathit{\alpha }}}_q=\arg \underset{{\tilde{\mathit{\alpha }}}_q}{\min }{\Vert \mathit{D}{\tilde{\mathit{\alpha }}}_q-{\tilde{\mathit{y}}}_{c,q}\Vert }_1^1+\lambda {\Vert {\tilde{\mathit{\alpha }}}_q\Vert }_1,q=1,\dots ,n_c.$$

(15)

Equation (15) is a $l_1$ norm metric and regularization convex optimization problem. We solved the optimization problem expressed by Equation (15) using the equivalent $l_1$ approximation robust sparse coding scheme reported previously [27]. The specific details are discussed in the next subsection.

When the robust sparse coding is assumed to be constant, the optimization solution of the dictionary matrix $\mathit{D}$ is equivalent to minimizing the following problem:

$$\underset{\mathit{D}\ge 0}{\min }\frac{1}{h}{\Vert {\tilde{\mathit{y}}}_c^t-\mathit{D}{\tilde{\mathit{\alpha }}}^t\Vert }_1^1+\lambda {\Vert {\tilde{\mathit{\alpha }}}^t\Vert }_1,$$

(16)

where $t$ is the current number of iterations, $h$ is the number of pixel spectral vectors used in each iteration, ${\tilde{\mathit{y}}}_c^t$ is the current training set consisting of $h$ randomly chosen pixel spectral vectors from the complete observation pixel spectral vector set ${\tilde{\mathit{Y}}}_c=\left[{\tilde{\mathit{y}}}_{c,1},\dots ,{\tilde{\mathit{y}}}_{c,n_c}\right]$, and the current training set robust sparse coding matrix ${\tilde{\mathit{\alpha }}}^t$ is obtained by solving Equation (15). Equation (16) is a standard $l_1$ regression problem. Due to the lack of differentiability, it cannot be solved like an $l_2$ regression problem. Therefore, we resorted to the iteratively re-weighted least squares method (IRLS) [28].

Each $\mathit{D}\left(j,:\right)$ can be estimated independently, and thus, without loss of generality, the j-th optimization function can be expressed as follows:

$$\mathit{D}\left(j,:\right)=\arg \underset{\mathit{D}\left(j,:\right)}{\min }\frac{1}{h}{\displaystyle \sum _{i=1}^h\left|{\tilde{y}}_{c,ij}^t-\mathit{D}\left(j,:\right){\tilde{\mathit{a}}}_i^t\right|},$$

(17)

where $\mathit{D}\left(j,:\right)\in {\mathbf{R}}^{1\times k}$, and ${\tilde{y}}_{c,ij}^t$ is the j-th element of ${\tilde{\mathit{y}}}_{c,i}^t$. According to the IRLS, Equation (17) can be converted into the following two problems:

$$\mathit{D}\left(j,:\right)=\arg \underset{\mathit{D}\left(j,:\right)}{\min }\frac{1}{h}{\displaystyle \sum _{i=1}^h{\omega }_{ij}^t{\left({\tilde{y}}_{c,ij}^t-\mathit{D}\left(j,:\right){\tilde{\mathit{a}}}_i^t\right)}^2}$$

(18)

$${\omega }_{ij}^t=\frac{1}{\sqrt{{\left({\tilde{y}}_{c,ij}^t-\mathit{D}\left(j,:\right){\tilde{\mathit{a}}}_i^t\right)}^2+\delta }},$$

(19)

where $\delta$ is a small positive quantity (the machine precision is chosen to be the experimental value). For robust statistical properties, the weighted square in Equation (5) is a reasonable approximation of the $l_1$ norm. Each iteration of Equation (5) involves minimizing the objective function twice. A global optimum can be achieved by taking the derivatives and setting them to zero.

In the online processing method, each update uses a small batch of new data of a fixed size, and thus, it is necessary to consider the statistical information of the historical data. Therefore, the information in the current iteration can updated as follows:

$${\mathit{M}}_j^t={\mathit{M}}_j^{t-1}+{\displaystyle \sum _{i=1}^h{\omega }_{ij}^t{\tilde{\mathit{a}}}_i^t{\left({\tilde{\mathit{a}}}_i^t\right)}^{\mathrm{T}}}$$

(20)

$${\mathit{C}}_j^t={\mathit{C}}_j^{t-1}+{\displaystyle \sum _{i=1}^h{\omega }_{ij}^t{\tilde{y}}_{c,ij}^t{\left({\tilde{\mathit{a}}}_i^t\right)}^{\mathrm{T}}}$$

(21)

$${\mathit{M}}_j^1={\displaystyle \sum _{i=1}^h{\omega }_{ij}^1{\tilde{\mathit{a}}}_i^1{\left({\tilde{\mathit{a}}}_i^1\right)}^{\mathrm{T}}}$$

(22)

$${\mathit{C}}_j^1={\displaystyle \sum _{i=1}^h{\omega }_{ij}^1{\tilde{y}}_{c,ij}^1{\left({\tilde{\mathit{a}}}_i^1\right)}^{\mathrm{T}}}.$$

(23)

Here, ${\displaystyle \sum _{i=1}^h{\omega }_{ij}^t{\tilde{\mathit{a}}}_i^t{\left({\tilde{\mathit{a}}}_i^t\right)}^{\mathrm{T}}}$ and ${\displaystyle \sum _{i=1}^h{\omega }_{ij}^t{\tilde{y}}_{c,ij}^t{\left({\tilde{\mathit{a}}}_i^t\right)}^{\mathrm{T}}}$ are the information of the observed data for the current iteration.

Therefore, a solution can be sought in the following linear system $\mathit{D}\left(j,:\right)$:

$${\mathit{C}}_j^t=\mathit{D}\left(j,:\right)\cdot {\mathit{M}}_j^t.$$

(24)

To solve Equation (14), a conjugate gradient method was used. The value of ${\mathit{D}}^{t-1}$ in the previous iteration was used as the starting value for the current iteration. Because the matrix ${\mathit{M}}_j^t$ was usually diagonally dominant, a reasonable initialization allowed the conjugate gradient update to converge quickly.

Through the above analysis, the specific steps of the online robust spectral dictionary learning algorithm (ORSDL) can be summarized. Algorithm 1 shows the pseudo code for the ORSDL.

Algorithm 1: Online robust spectral dictionary learning (ORSDL)

Input: complete observation pixel spectral vector ${\tilde{\mathit{y}}}_{c,q}\in {\mathbf{R}}^L,q=1,\dots ,n_c$,    number of iterations $T\in \mathbf{N}$, regularization term coefficient $\lambda >0$    the number of the pixel spectral vectors per iteration $h\in \mathbf{N}$    initial spectral dictionary ${\mathit{D}}_0\in {\mathbf{R}}^{L\times k}$

Output: spectral dictionary $\mathit{D}\in {\mathbf{R}}^{L\times k}$

1   for $t=1$ to $T$ do

2    Randomly choose ${\tilde{\mathit{y}}}_c^t=\left[{\tilde{\mathit{y}}}_{c,i}^t,i=1,\dots ,h\right]$ from ${\tilde{\mathit{Y}}}_c$

/* Solving robust sparse coding problem*/

3    for $i=1$ to $h$ do

4      ${\tilde{\mathit{a}}}_i^t=\arg \underset{{\tilde{\mathit{a}}}_i^t}{\min }{\Vert {\tilde{\mathit{y}}}_{c,i}^t-\mathit{D}{\tilde{\mathit{a}}}_i^t\Vert }_1^1+\lambda {\Vert {\tilde{\mathit{a}}}_i^t\Vert }_1$

5    end for

/* dictionary update */

6    repeat

7      for $j=1$ to $k$ do

8        ${\mathit{M}}_j^t={\mathit{M}}_j^{t-1}+{\displaystyle \sum _{i=1}^h{\omega }_{ij}^t{\tilde{\mathit{a}}}_i^t{\left({\tilde{\mathit{a}}}_i^t\right)}^{\mathrm{T}}}$

9        ${\mathit{C}}_j^t={\mathit{C}}_j^{t-1}+{\displaystyle \sum _{i=1}^h{\omega }_{ij}^t{\tilde{y}}_{c,ij}^t{\left({\tilde{\mathit{a}}}_i^t\right)}^{\mathrm{T}}}$

10        solve linear system ${\mathit{C}}_j^t=\mathit{D}\left(j,:\right)\cdot {\mathit{M}}_j^t$

11        ${\omega }_{ij}^t=\frac{1}{\sqrt{{\left({\tilde{y}}_{c,ij}^t-\mathit{D}\left(j,:\right){\tilde{\mathit{a}}}_i^t\right)}^2+\delta }},$

12     end for

13    until convergence

14   end for

15   return $\mathit{D}$

Because the value of the initialization dictionary has a negligible effect on the learning results of the spectral dictionary, the first $k$ pixel spectral vectors of ${\tilde{\mathit{Y}}}_c$ are usually chosen for the initialization of the spectral dictionary for convenience.

3.2. Robust Sparse Coding

After training the spectral dictionary $\mathit{D}$ by the spectral dictionary learning method proposed in Section 3.1, the quantity ${\mathit{\Psi }}_i$ in the optimization problem (Equation (10)) was known. Thus, solving the corresponding robust sparse coding ${\widehat{\mathit{\alpha }}}_i$ was the same as calculating the sparse coding in the dictionary learning and was regarded as an $l_1$ norm measurement and $l_1$ norm regularization convex optimization problem, expressed as follows:

$${\widehat{\mathit{\alpha }}}_i=\arg \underset{{\mathit{\alpha }}_i}{\min }{\Vert {\mathit{\Psi }}_i{\mathit{\alpha }}_i-{\mathit{y}}_{o,i}\Vert }_1^1+\lambda {\Vert {\mathit{\alpha }}_i\Vert }_1,$$

(25)

where ${\Vert {\mathit{\alpha }}_i\Vert }_1={\displaystyle {\sum }_{ij}\left|{\alpha }_{ij}\right|}$.

Equation (25) can be converted to an equivalent-approximate robust sparse coding problem:

$${\widehat{\mathit{\alpha }}}_i=\arg \underset{{\mathit{\alpha }}_i}{\min }{\Vert \left(\begin{array}{c}{\mathit{y}}_{o,i}\\ 0\end{array}\right)-\left(\begin{array}{c}{\mathit{\Psi }}_i\\ \lambda \mathit{I}\end{array}\right){\mathit{\alpha }}_i\Vert }_1.$$

(26)

Because the set of linear equations,

$$\left[\begin{array}{c}{\mathit{y}}_{o,i}\\ 0\end{array}\right]=\left[\begin{array}{c}{\mathit{\Psi }}_i\\ \lambda \mathit{I}\end{array}\right]{\mathit{\alpha }}_i,$$

(27)

is hyper-determined, Equation (26) satisfies the conditions outlined previously [29], and thus, the existence of a global optimal solution is guaranteed. Finally, the corresponding robust sparse coding ${\widehat{\mathit{\alpha }}}_i$ was obtained by solving Equation (26).

4. HSI Inpainting Algorithm Outline

By analyzing the HSI inpainting model and the HSI inpainting mechanism in Section 2 and studying the online robust spectral dictionary learning approach and the robust sparse coding method in Section 3, an HSI inpainting algorithm based on online robust spectral dictionary learning, INORSDL, is proposed. Algorithm 2 shows the pseudo code of the proposed INORSDL approach.

Algorithm 2 INORSDL.

Input:$\mathit{Y}\in {\mathbf{R}}^{L\times n_s}$ (Observed HSI data), $\mathit{M}$ (inpainting mask matrix)

Output:$\widehat{\mathit{X}}\in {\mathbf{R}}^{L\times n_s}$ (Inpainted HSI data)

1  begin

2   select the complete observed HSI pixel spectral vectors ${\tilde{\mathit{y}}}_{c,q}\in {\mathbf{R}}^L,q=1,\dots ,n_c$

3   $\widehat{\mathit{D}}=\mathrm{ORSDL}\left({\tilde{\mathit{y}}}_{c,q}\in {\mathbf{R}}^L,q=1,\dots ,n_c\right)$ (robust spectral dictionary training, Algorithm 1)

4   solve the robust sparse code matrix $\widehat{\mathit{\alpha }}$

5   $\widehat{\mathit{X}}=\widehat{\mathit{D}}\widehat{\mathit{\alpha }}$

6  end

5. Experimental Results and Analysis

To verify the performance of the HSI inpainting algorithm proposed herein, experiments were performed using simulated and real hyperspectral data. The experimental results were analyzed and compared with the effective HSI inpainting algorithms that are widely used today. The performance of the algorithm was evaluated in terms of subjective effects and objective quality. The experimental environment was the MATLAB R2018a software, the operating system was a 64-bit Windows 10, the hardware platform was an Alienware R15 laptop containing a 2.80 GHz eight-core Intel Core i7-7700HQ CPU and 16 GB RAM.

To validate the effectiveness of the algorithm, we compared the performance of the proposed algorithm with that of the 3D data compatible discrete partial differential equation method (PDE) [30,31], unmixing-based denoising (UBD) [32], a denoising method based on local low rank matrix recovery and global spatial-spectral total variation method (LLRSSTV) [33], the BPFA, and other image inpainting algorithms.

5.1. Simulated Data Experiment

The Pavia Centre HSI was used to construct simulated data (http://www.ehu.eus/ccwintco/index.php?Title=Hyperspectral_Remote_Sensing_Scenes). The Pavia Centre scene was acquired by the reflective optics system imaging spectrometer (ROSIS) sensor during a flight over Pavia, northern Italy, with atmospheric correction [34]. Each pixel in the scene includes 102 spectral bands between 0.43 $\mathsf{\mu }\mathrm{m}$ and 0.86 $\mathsf{\mu }\mathrm{m}$. For the convenience of analysis, the chosen spatial size of the data was 201 × 201 pixels here, as shown in Figure 1. To simulate the clean image, high level noise bands were removed and totally 80 spectral bands were remained. This data set was considered to be a clean image in this section, and each band of data was normalized before adding noise and missing strips.

The effects of four types of noise were considered in the experiment: Gaussian independent and identically distributed (i.i.d) noise, Gaussian non-i.i.d noise, Laplacian noise, and Poisson noise. The Gaussian i.i.d noise was taken to have a zero mean and a variance of $\sigma =0.10$. The Gaussian non-i.i.d noise obeyed the distribution ${\mathit{n}}_i\sim \mathbf{N}\left(\mathbf{0},{\mathit{D}}_d^2\right)$, where ${\mathit{D}}_d$ is a diagonal matrix with the diagonal elements uniformly distributed in the (0, 1) range. The Laplacian noise scale was set to 30. The Poisson noise obeyed the distribution $\mathit{Y}\sim P\left(\phi \mathit{X}\right)$, where $P\left(\mathit{W}\right)$ represents an independent Poisson random variable matrix of size $\mathit{W}$, and the parameter is given by the corresponding elements $\mathit{W}:=\left[w_{ij}\right]$. The signal-to-noise ratio, $\mathrm{SNR}:\phi \left({\displaystyle {\sum }_{i,j}{w}_{ij}^2}\right)/\left({\displaystyle {\sum }_{i,j}{w}_{ij}}\right)$, was set to 15 dB using parameter $\phi$. The setting of the missing strip was done following a previous report [12], and the position of the missing strip was known. In dictionary learning process, the regularization parameter $\lambda$ of the proposed INORSDL is set to 1.

To quantitatively assess the effectiveness of the algorithm, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) indicators were used to quantitatively evaluate and compare the image inpainting performance of each algorithm. The PSNR was used to measure the degree of similarity between the inpainted HSI as a whole and the original real image. SSIM is a quantitative evaluation method based on human vision. It can better reflect the quality of visual perception of an image and gives emphasis to the structural information of the images, making it suitable for the spatial dimensions of HSIs.

Figure 2, Figure 3, Figure 4 and Figure 5 show the results of various hyperspectral inpainting algorithms for Gaussian i.i.d noise, Gaussian non-i.i.d noise, Laplacian noise, and Poisson noise, respectively. For Poisson noise, the algorithm proposed herein first transformed the Poisson noise into an approximate additive Gaussian noise with an approximately equal variance using the variance stability transformation [35] and subsequently processed it.

The inpainting results of different algorithms under various noise conditions show that all the inpainting algorithms could restore the missing strips in the image, but the LLRSSTV algorithm could not completely restore the missing strips, which is evident in Figure 2. Because the PDE algorithm only inpaints missing pixels and does not perform simultaneous denoising on the image, there was still noise in the output image after the algorithm was applied. In sharp contrast, the UBD, LLRSSTV, BPFA, and INORSDL algorithms removed a large portion of the noise while restoring the missing pixels. Because the UBD algorithm reconstructs missing or corrupted pixels using the results of spectral unmixing, its performance depends largely on the estimation accuracy of the endmember and abundance coefficients in the spectral unmixing. In the field of HSI processing, spectral unmixing is still a challenging problem. The BPFA algorithm showed good image inpainting performance under various noise conditions, but its ability to remove Gaussian non-i.i.d noise was poor, which is evident in Figure 3. Figure 4 shows that with the exception of the PDE algorithm, all the other inpainting algorithms could remove Laplacian noise to some extent, but the INORSDL algorithm proposed herein yielded better results. The INORSDL algorithm could restore the missing pixels in the image for all three kinds of noise conditions and could also eliminate most of the noise at the same time, providing a high quality inpainted image directly. Figure 2, Figure 3, Figure 4 and Figure 5 show that the INORSDL algorithm qualitatively achieved the best inpainting results under different noise conditions.

In addition, we also compared the performance of each inpainting algorithm with the mean peak signal-to-noise ratio (Mean-PSNR, MPSNR) and the mean structural similarity (Mean-SSIM, MSSIM), as shown in

Table 1. Performance comparison of different hyperspectral inpainting algorithms applied to Pavia Centre dataset.

	Index	Noisy Image	PDE	UBD	LLRSSTV	BPFA	INORSDL
Gaussian i.i.d. noise	MPSNR (dB)	19.7997	20.0392	34.4112	30.8946	35.6697	36.7168
	MSSIM	0.4331	0.4435	0.9355	0.9120	0.9572	0.9687
	Time (s)	-	2	53	597	1126	903
Gaussian non-i.i.d noise	MPSNR (dB)	28.1013	28.6221	37.5838	34.1243	32.8583	38.9195
	MSSIM	0.7057	0.7206	0.9686	0.9580	0.8240	0.9895
	Time (s)	-	2	60	574	5152	1621
Laplacian noise	MPSNR (dB)	33.1715	33.9045	37.9209	35.3268	38.2032	38.5285
	MSSIM	0.9292	0.9457	0.9873	0.9728	0.9892	0.9914
	Time (s)	-	2	51	586	1525	1124
Poisson noise	MPSNR (dB)	26.5466	26.9952	35.9059	33.8843	37.3994	40.6220
	MSSIM	0.7476	0.7609	0.9647	0.9599	0.9669	0.9870
	Time (s)	-	1	36	287	9779	1531

The highest value in each row uses the bold font.

. The highest value in each row is shown in bold. As shown in the table, the INORSDL algorithm yielded the best results for the four kinds of noise conditions, and its inpainting performance was quantitatively evaluated.

Table 1 also shows the time required for the different algorithms when applied to different noise conditions. Compared with the BPFA, another algorithm based on dictionary learning, the algorithm of this work was faster and more efficient.

5.2. Real Data Experiment

In this section, the algorithm was applied to the real HSI dataset Urban (http: //www.agc.army.mil/). The Urban scene was acquired by the hyperspectral digital collection experiment (HyDICE) sensor during a flight over Copperas Cove, Texas. The dataset had a spatial size of 307 × 307 and included 210 bands ranging from 0.4 to 2.5 $\mathsf{\mu }\mathrm{m}$, with a spectral resolution of 10 nm, as shown in Figure 6. Its spatial resolution was about 2 × 2 m. Due to the influence of water vapor and atmospheric effects, the image had strong noise in many bands, and these bands contained little useful information.

To verify the effectiveness of the algorithm, the proposed algorithm was compared with other hyperspectral inpainting algorithms: PDE, UBD, LLRSSTV, and BPFA. In dictionary learning process, the regularization parameter $\lambda$ of the proposed INORSDL is set to 1. Figure 7, Figure 8 and Figure 9 display the inpainting results of the various hyperspectral repair algorithms in the 105th, 144th, and 208th bands, respectively. It is evident in the figure that the INORSDL algorithm proposed in this paper exhibited better inpainting results than the other inpainting algorithms. However, although the LLRSSTV algorithm removed noise while restoring damaged pixels, the inpainted image showed noticeable patch effects, and some local restorations were unsatisfactory.

6. Conclusions

In this paper, we proposed an HSI inpainting algorithm based on online robust spectral dictionary learning. A spectral dictionary adapted to image inpainting was obtained using a new algorithm proposed in this paper for constructing a dictionary directly from hyperspectral data. The spectral dictionary was subsequently used in the sparse reconstruction of the image, thereby restoring missing or corrupted pixels. Compared to the most advanced HSI inpainting algorithms, the algorithm herein possessed a better inpainting ability for additive Gaussian noise, Laplacian noise, and Poisson noise. Compared to other image inpainting algorithms based on dictionary learning, this algorithm was faster. However, this algorithm requires complex calculations, and thus future research should focus on its rapid implementation. In addition, the research object of this paper is limited to visible near-infrared (VNIR) and short-wave infrared (SWIR) HSIs. However, in the preprocessing of medium-wave infrared (MWIR) HSIs, dead pixels inpainting is a critical problem. The imaging principle of MWIR HSIs is very different from VNIR/SWIR HSIs. In our future research, MWIR HSI inpainting will also be an important direction.