*2.2. Spatial Regularization*

Despite taking into account sparsity, SUnSAL ignores spatial correlation. In SUnSAL-TV, the relationship between each pixel vector and its adjacent pixel vectors is taken into account. The regularizer is defined in [27] as

$$\text{TV}(\mathbf{X}) = \sum\_{\{i,j \in \kappa\}} ||\mathbf{x}\_i - \mathbf{x}\_j||\_1 \tag{5}$$

which is the anisotropic TV with *κ* denoting the set of horizontal and vertical neighbors in **X**.

Adding the TV regularizer to the problem in Equation (4) gives the optimization problem

$$\min\_{\mathbf{X}} \quad \frac{1}{2} \|\mathbf{A}\mathbf{X} - \mathbf{Y}\|\_F^2 + \lambda \|\mathbf{X}\|\_1 + \lambda\_{TV} \text{TV}(\mathbf{X}) \quad \text{s.t.} \quad \mathbf{X} \ge 0. \tag{6}$$

## **3. Proposed Algorithm**

#### *3.1. Local Abundance Correlation*

Hyperspectral data **Y** ∈ R*l*×*n* have linearity in their spectral [38] and spatial [30] domains. Qu et al. [30] provided prior knowledge that the high spatial correlation of the hyperspectral data, implies linearly dependent abundance vectors in the abundance matrix **X** ∈ R*<sup>m</sup>*×*n*. The high correlation also holds among the pixel members of a local region due to the spatial similarity. In a physical sense, the pixels in such regions contain the same materials, either in the same or different fractions. Hence, the abundance matrix of the region can be estimated by the low-rank property [30,34].

However, the success of sparse regression techniques is affected by the low sparsity as well as low correlation between spectral signatures in the library [27]. The former is represented by the number of endmembers existing in the scene, namely, the degree of sparsity [26]. The latter can be defined by an indicator representing the difficulty to accurately solve a linear system equation i.e., mutual coherence. The mutual coherence is defined as the largest cosine among endmembers in the library. In the hyperspectral case, the degree of sparsity is often low, but the mutual coherence is close to one. In fact, higher mutual coherence decreases the quality of the solution [28].

To overcome the high mutual coherence as well as consider the low-rank property of the abundance, we exploit the high correlation of library's spectral signatures by using our LA regularizer. In our experiment with simulated data, we confirmed the idea by observing the linearity of the data distribution in abundance domain by taking the local maximum singular value of the true abundance matrix for each local block (a block refers to the three dimensions (3D), in which the third dimension has a local coverage in the endmember direction). We found that there is one value that dominates others (the ratio is close to one) in each local block. On the other hand, the value will be less dominant as the region becomes the whole matrix (nonlocal). This implies that the linearity in abundance domain is satisfied for the abundance matrix with the local point of view. Thus, we introduce our LA regularizer using the nuclear norm for the local blocks. Instead of the image, our algorithm uses the nuclear norm to the abundance matrix that constitutes the image. Another difference is that our local block slides through all dimensions, i.e., the two spatial dimensions and the endmember direction in the abundance dimension. Figure 2 illustrates the endmember direction. The block moves within the abundance maps of the 3D abundance cube.

In addition, we guarantee high correlation by selecting endmembers from the United States Geological Survey (USGS) library to form the spectral library **A** based on the SA. The USGS library is a collection of the measured spectral signatures of hundreds of materials and used as references for material identification in hyperspectral images. We can find the most similar signatures to each endmember of the simulated data by calculating the SA, besides the mutual coherence. This parameter represents the absolute value of spectral correlation [39]. The value ranges between 0–90 degrees. The lower the SA value, the more similar the compared signature vectors are. In the simulated-data experiment, we adjusted the SA as one of our parameter settings.

**Figure 2.** Illustration of endmember (*m*) direction in abundance dimension. 3D local block moves through pixels (*n*) as well as *m* direction of abundance maps.

#### *3.2. Collaborative Sparsity Regularization*

In practice, the abundance matrix **X** has only a few endmembers (rows) with nonzero entries. Simultaneously, all the column entries of **X** share the same active set of endmembers. In other words, **X** is sparse among the rows while dense among the columns. To implement this prior, *<sup>L</sup>*2,1 norm is used instead of *L*1. It takes the sum of the *L*2 norm of the abundance entries to promote the collaborative sparsity of the abundance matrix.

$$\|\mathbf{X}\|\_{2,1} = \sum\_{i=1}^{m} \|\mathbf{x}\_i\|\_2 \tag{7}$$

where **x***i* represents the *i*-th row of **X**.

#### *3.3. Local Abundance Regularizer*

First, let **X ˆ** ∈ R*nr*×*nc*×*m* be the abundance data in 3D form, where *m* is the number of abundance matrices of the endmembers, *nc* and *nr* are the numbers of columns and rows, respectively, that satisfy *n* = *nc* × *nr*, where *n* is the number of pixels in each abundance matrix. Then, for each abundance matrix **X ˆ** *i* ∈ R*nr*×*nc* (*i* = 1, ... , *<sup>m</sup>*), stacking the column on top of one another gives **xˆ***i* ∈ R*<sup>n</sup>*, the vectorized form of the matrix.

In local regions, let **X ˆ** *b* ∈ R*nb*×*nb*×*mb* denote the *b*-th local block, where *b* = 1, ... , *B*. The *B* is the number of all local blocks in **X ˆ** . Then, for each abundance of each local block **X ˆ** *j*,*b* ∈ R*nb*×*nb* (*j* = 1, ... , *mb*), we vectorize it into **xˆ***j*,*<sup>b</sup>* ∈ <sup>R</sup>*N*, where *N* is the number of pixels in each local abundance matrix that satisfies *N* = *nb* × *nb* , and *j* is the index of local abundance matrices. Figure 3 illustrates the procedure. With this in mind, we introduce the local abundance matrix w. r. t the *b*-th block

$$\mathbf{H}\_{\mathbb{K}\_b} = (\mathfrak{x}\_{1,b}, \dots, \mathfrak{x}\_{m\_b,b}) \in \mathbb{R}^{N \times m\_b}. \tag{8}$$

Finally, the function of our proposed LA regularization is defined as follows

$$\|\mathbf{X}\|\_{LA\*} = \sum\_{b=1}^{B} \|\mathbf{H}\_{\mathbf{k}\_b}\|\_{\*} \tag{9}$$

where ·∗ denotes the nuclear norm, **X** → *rank*(**X**) ∑ *i*=1 *<sup>σ</sup>i*(**X**), with *σi* denotes the *i*-th singular value.

**Figure 3.** Process of vectorizing and arranging local abundance matrix of hyperspectral image.
