**1. Introduction**

In the past decade, numerous methods have been introduced for unmixing hyperspectral imagery [1,2]. Spectral mixture analysis (SMA) is one of most commonly-used methods, and is used in different applications. Basically, the spectra of mixed pixels are modeled using linear or non-linear mixture models. The spectral signature of each pixel is converted to a set of fractional abundances of its constituent spectra (endmembers) by these models [3]. The answer to this question of which one (linear or non-linear models) is superior for unmixing the hyperspectral data is not clear, and depends on the type of the mixture of objects and their applications. However, the acceptable accuracy and the simplicity of linear mixture models entice more researchers to employ them [4]. If the multiple scattering among the endmembers is negligible and the mixture could be supposed macroscopic, a linear mixture model (LMM) can be written as Equation (1):

$$\mathbf{y}(n) = \sum\_{i=1}^{p} a\_i(n)\mathbf{m}\_i + \mathbf{v}(n) = \mathbf{M}\mathbf{a}(n) + \mathbf{v}(n),\tag{1}$$

where **y**(*n*) = [*y*1(*n*), *y*2(*n*),..., *yB*(*n*)]*<sup>T</sup>* ∈ R*<sup>B</sup>* is the vector of observations; B is the number of bands; *n* = 1, ··· , *N* is the index of pixels in the image; **m***i* ∈ <sup>R</sup>*B*, *i* = 1, ... , *p* is the spectral signature of endmembers; p is the number of endmembers; *<sup>α</sup>i*(*n*) is the abundance of the *i*th endmember in the *n*th pixel; **M** = \* **m**1, ··· , **<sup>m</sup>***p* + ∈ R*B*×*p* is the coefficient matrix of endmembers; α(*n*) = %*<sup>α</sup>*1(*n*), ··· , *<sup>α</sup>p*(*n*)&*<sup>T</sup>* ∈ R*p* is the vector of abundance values in the *n*th pixel; and ν(*n*) ∈ R*<sup>B</sup>* represents noise.

The accuracy of the unmixing process highly depends on the completeness and goodness of the selected endmembers. Therefore, many endmember extraction algorithms have been developed in recent years [3,5]. The accuracy of the fractional abundances obtained from SMA is affected by the residual spectral error caused by inaccurate atmospheric correction, an insufficient signal-to-noise ratio (SNR), and the noise caused by neglecting the non-linear effect of inputs. However, the most important source of error in SMA is due to ignoring the spectral variability (SV) of endmembers caused by variable illumination and environmental, atmospheric, and temporal conditions [4]. These algorithms generally model the entire image using a constant spectral feature for each endmember. In fact, this is a simplification, because in many cases the spectrum of endmembers could change in different spatial and temporal conditions.

Generally, two types of SV can be distinguished among the samples from different classes: (1) the variability within the endmembers of a specific class (intra-class variability); and (2) the spectral similarity between the endmembers of different classes (inter-class variability) [4]. By increasing the intra-class variability, the accuracy of sub-pixel fraction estimation decreases linearly [6]. On the other hand, in some applications where the separation of similar phenomena is of interest, the spectral similarity among the different endmembers (e.g., crops and weeds in agricultural fields or spectral similarity among minerals) makes it difficult to separate these classes. The estimation of the fractional abundances using the linear mixture model could be achieved by different methods, such as least squares and sparse regression with different constraints [3,7]. In the least squares-based spectral unmixing problem, the spectral similarity among the endmembers results in a high correlation between the columns of the coefficients matrix (M) in Equation (1). Consequently, the rank deficiency of the coefficients matrix leads to an unstable solution for the least squares problem and decreasing the accuracy of the estimation of the fractional abundances. Despite the serious effects in the LMMs and the destruction of the reliability of the results of spectral unmixing (SU), this issue is typically ignored [8].

According to [4], the efforts to decrease the effect of the SV can be classified into five general categories: (1) the use of multiple endmembers for each component in an iterative mixture analysis procedure; (2) the spectral weighting of bands; (3) the spectral transformations; (4) the use of radiative transfer models in a mixture analysis; and (5) the selection of a subset of stable spectral features. In addition, to significantly improve the accuracy of the estimation of fractional abundances, the last strategy effectively reduces the computational cost.

The non-orthogonality of the endmembers appears when a linear correlation exists between two endmembers or a multi-collinearity exists among some endmembers. By increasing the correlation among the endmembers, the LMM tends to be instable and extremely sensitive to the small variations of the input spectrum and noises. According to [8], the approaches to deal with the problem could be categorized as: (1) excluding the correlated endmember; (2) de-correlating the endmembers using the spectral transformations; (3) using iterative approaches to select the independent endmembers; and (4) the regularization of the SU equations. Regarding the redundancy of bands in the hyperspectral

images, it is not unexpected to identify a subset of bands that decreases the correlation of endmembers. To deal with the problem, the correlation of endmembers can be evaluated using singular value decomposition (SVD) and the condition number of the coefficient matrix of the endmembers in the unmixing procedure.

This paper presents a novel and effective approach for managing the SV and decreasing spectral correlation among the endmembers based on the selection of the optimal bands in the Prototype Space (PS) [9]. The proposed method consists of two main steps. Based on the spectral behavior of the endmembers' set, the image bands are firstly prioritized in such a way that they have the least sensitivity to the SV of the endmembers. Then, the optimal band selection is done based on this prioritization. Since the spectral correlation among the image bands is not considered in this process, in the second step the independent bands are selected using their angles in the PS. In this way, the spectral correlation among the endmembers is reduced as well. Besides, collecting a spectral library from the SV of endmembers is an expensive and time-consuming process. Therefore, these sets were directly extracted from the image in this paper.

The remaining parts of the paper are organized as follows: the theoretical background and the previous algorithms are introduced in Section 2; the proposed method is explained in Section 3; the experimental results and further discussion are provided in Section 4; and concluding remarks are found in Section 5.
