SAOCNN: Self-Attention and One-Class Neural Networks for Hyperspectral Anomaly Detection

Abstract

Hyperspectral anomaly detection is a popular research direction for hyperspectral images; however, it is problematic because it separates the background and anomaly without prior target information. Currently, deep neural networks are used as an extractor to mine intrinsic features in hyperspectral images, which can be fed into separate anomaly detection methods to improve their performances. However, this hybrid approach is suboptimal because the subsequent detector is unable to drive the data representation in hidden layers, which makes it a challenge to maximize the capabilities of deep neural networks when extracting the underlying features customized for anomaly detection. To address this issue, a novel unsupervised, self-attention-based, one-class neural network (SAOCNN) is proposed in this paper. SAOCNN consists of two components: a novel feature extraction network and a one-class SVM (OC-SVM) anomaly detection method, which are interconnected and jointly trained by the OC-SVM-like loss function. The adoption of co-training updates the feature extraction network together with the anomaly detector, thus improving the whole network’s detection performance. Considering that the prominent feature of an anomaly lies in its difference from the background, we designed a deep neural extraction network to learn more comprehensive hyperspectral image features, including spectral, global correlation, and local spatial features. To accomplish this goal, we adopted an adversarial autoencoder to produce the residual image with highlighted anomaly targets and a suppressed background, which is input into an improved non-local module to adaptively select the useful global information in the whole deep feature space. In addition, we incorporated a two-layer convolutional network to obtain local features. SAOCNN maps the original hyperspectral data to a learned feature space with better anomaly separation from the background, making it possible for the hyperplane to separate them. Our experiments on six public hyperspectral datasets demonstrate the state-of-the-art performance and superiority of our proposed SAOCNN when extracting deep potential features, which are more conducive to anomaly detection.

1. Introduction

Hyperspectral images are a set of three-dimensional data and involve two-dimensional spatial information and abundant spectral information with hundreds of narrow contiguous bands [1]. Hyperspectral images have greater advantages in the areas of ground object classification [2,3,4], change detection [5,6], and target detection [7,8,9], depending on their powerful space–spectral information expression capabilities compared with other remote sensing images [10], such as visible and infrared light. According to whether the spectrum information of targets is known, target detection can be divided into either supervised target detection or anomaly detection [11,12]. Anomaly detection divides the pixels into anomaly and normal (background) without prior spectral information of the targets [13]. Generally, anomalies in hyperspectral images refer to the spectral irregularities caused by the presence of atypical ground objects [14]. The anomalies have not been authoritatively defined by researchers and are widely agreed upon as targets whose spectral information clearly deviates from the background distribution, such as airplanes, vehicles, and other man-made objects in natural scenes [15,16].

Traditional methods of hyperspectral anomaly detection are generally divided into three categories: background modeling-based methods, distance-based methods, and representation-based methods [17,18].

The background modeling method is based on statistical theory. Assuming that the background obeys a certain distribution, statistical methods can be used to measure the likelihood of test pixels to be anomalous after estimating its parameters.

The well-known Reed–Xiaoli (RX) detector is a classic background modeling method [19]. It builds a statistical model of the background through the mean and covariance matrix of the whole image with the assumption that the background obeys a multivariate Gaussian distribution. Then, it detects anomalies by measuring the Mahalanobis distance between the test pixel spectrum and background spectral distribution. However, a whole-image-based background modeling method may be inaccurate with a complex background consisting of a variety of different distributions. Therefore, the Local RX (LRX) method uses samples from the local area around the test pixel to perform statistical background modeling [20]. Another issue with the RX method is that the background disobeys the multivariate Gaussian distribution in most cases. Therefore, Kwon and Nas [21] proposed a detection method based on the kernel function (Kernel RX, KRX), which maps hyperspectral image data into a higher dimensional feature space. Their results show that hyperspectral data tend to obey the multivariate Gaussian distribution in such a kernel space. Many improved RX detectors have appeared in follow-up studies, such as Segmented-RX [22], Weighted-RX, and linear filter-based RX detectors [23], etc.

In hyperspectral anomaly methods based on distance, pixels are grouped according to distance, and the pixels that deviate from the center of the cluster are considered anomalies. Distance-based approaches can be further classified into: support vector machine (SVM)-based, clustering-based, and graph-based methods. The SVM-based method obtains a hyperplane with the smallest distance in background samples by a support vector data description (SVDD) [24], and the distance between the test pixel and the center of the hyperplane in the higher feature space is considered the anomalous score [25]. In the clustering-based method, the original hyperspectral data are clustered by k-means [26], fuzzy clustering [27] or other approaches, and the Mahalanobis distance is calculated for each cluster to estimate the anomalous scores of test pixels. Graph theory [28], which reflects the internal structure of the data, is introduced in the graph-based method. Pixels in hyperspectral images are considered as the set of vertices of a graph and are connected with an edge, becoming inseparable when the similarity exceeds a given threshold. The large-component pixels of the graph form the background set; therefore, the test pixel’s anomalous score is regarded as the distance from the nearest point in the background graph [29,30].

To avoid estimating the background with an inappropriate data distribution or evaluating the anomalies just by cluster distance, another method based on representation uses the linear combination of elements in the background dictionary to express each pixel in a hyperspectral image. Because the background and anomalies belong to different clusters, the representation of the background pixels will perform better than the anomalies through the constructed background-based dictionary, so that the residual image of the represented and original images is considered a measure of anomalies. Sparse (SR) [31], collaborative (CR) [32], and low-rank representation (LRR) [33] are all representation methods.

SR theory considers that signals are sparse [34], i.e., the features of a background sample are far fewer than its number. Therefore, the background pixels can be expressed by as few linear combinations of elements as possible in a given over-complete dictionary, which is constructed by a large amount of background sample data. In the SR-based anomaly detection method, the reconstruction of anomalies through a learned dictionary is worse than that of the background [13], and various anomaly detection models based on sparse representation theory have been proposed [35,36,37]. Furthermore, a CR-based anomaly detector was proposed based on the assumption that the collaboration between dictionary atoms is more considerable than the competition between them [38]. The number of dictionary atoms used to represent test pixels is required to be as small as possible in SR theory, while all dictionary atoms are used in CR theory. CR-based methods have rapidly developed many extensions [39,40,41]. The over-complete background dictionary in SR and CR theory is usually obtained by a dual-window strategy [42], which indicates that only local attributes are used to constrain the coding vector; however, global structure information is not considered. In addition, setting the dual window to a reasonable size without polluting anomalies is difficult when constructing the dictionary in the absence of prior information.

Different from the two theories above, LRR assumes that each pixel in hyperspectral data can be expressed as a linear representation of the basis vectors formed by the dictionary [43]. In real hyperspectral images, the number of categories is far fewer than the number of pixels; therefore, the matrix’s rows or columns are strongly correlated, creating a low-rank hyperspectral image. Based on LRR theory, a hyperspectral image is decomposed into a low-rank background matrix and a sparse anomaly matrix with the assumption that the background is in a low-dimensional subspace and considered to be highly correlated; furthermore, anomalies only occupy a small part of the image scene and have a low occurrence probability [44]. Other algorithms were presented to better represent and decompose the background and anomaly matrices [45,46,47] and to find the lowest rank representation of all pixels and use the sparse constraint anomalies under the background dictionary.

In recent years, deep learning has gradually become popular in the field of computer vision and has achieved many outstanding study results. Due to its excellent performance in the modeling and feature extraction of complex data, deep learning has been introduced into hyperspectral anomaly detection. A deep neural network’s main role in anomaly detection is to extract the essential features of the original hyperspectral image as a feature extractor while reducing the dimensionality.

Most researchers proposed generative models, including autoencoders (AEs) [48], deep belief networks (DBNs), [49], and generative adversarial networks (GANs) [50], to minimize the error between the original and reconstructed spectra to extract features in the hidden layer or reconstruction spaces. Subsequently, traditional anomaly detectors (such as RX, CRD, LRR, etc.) have been applied to the features extracted by deep neural networks to perform the final detection. Arisoy et al. [51] utilized a GAN model to generate a synthetic background image similar to the original and an RX detector to detect anomalies on the residual image of synthetic and original images. Jiang et al. [52] used an adversarial autoencoder (AAE) to reconstruct the hyperspectral image and combined the morphological filter and RX detector on the residual image of original and reconstructed images to detect anomalies. AAE was also adopted in the spectral adversarial feature learning (SAFL) architecture proposed by Xie et al. [53]. Self-attention mechanisms [54] have been widely employed for anomaly detection in texts [55], videos [56], and industrial images [57] because of the effective learning capabilities. Jiang [58] proposed a manifold-constrained, multi-head, self-attention, variational autoencoder (MMS-VAE) method for hyperspectral anomaly detection, which introduced a self-attention mechanism to focus on abnormal areas by learning context-related information.

The main problem for methods based on deep learning is that the objective functions do not jointly optimize the feature extractor and anomaly detector, which makes the deep neural network unable to exert its advantages.

To address the above problems, we propose a new hyperspectral anomaly detection model based on deep learning to jointly optimize the feature extractor and hyperplane layer (i.e., the anomaly detector) in this paper. Accordingly, we present a new model with an extractor to obtain both global and local features in hyperspectral images, in addition to an anomaly weight map that roughly evaluates the probability of each pixel to be anomalous and enhances possible anomalous regions. Specifically, we designed our proposed network structure and loss function as an extension of the basic one-class support vector machine (OC-SVM) algorithm [24,59]. OC-SVM is a variant of the traditional SVM algorithm in which all training samples are considered as positive and the original data are mapped to a new high-dimensional space through the kernel function, solving the problem of unbalanced samples in two-class classification. Our proposed model replaces the kernel function in the original OC-SVM with a feature extractor based on a deep neural network with optimizable parameters, making it possible to jointly train the feature extraction network with the subsequent hyperplane layer for a one-class objective.

In the feature extractor, both global and local features are obtained, and the original data are mapped into a new feature space. We introduced an attention mechanism to mine the feature correlation of each pixel with its neighbor in the image as the global information. In addition, we utilized an anomaly weight map to enhance the possible anomalous regions, considering that low-SNR targets may decrease anomaly detection performance. Moreover, we also designed a local feature extraction block because a local spatial structure helps to improve the efficiency of feature utilization [60,61]. In the hyperplane layer, most of the pixels are considered as background, while the origin of the high-dimensional space is representative of the anomaly points. Finally, we trained the best hyperplane to separate the background from the anomaly by maximizing the distance between the hyperplane and the representative point for anomaly.

Compared with other existing state-of-the-art hyperspectral anomaly detection methods, the main contributions of our new proposed method are summarized as follows:

• We propose a new hyperspectral anomaly detection model and an unsupervised training framework. Whereas other methods separated them, we combine and co-optimize the feature extraction network and hyperplane layer through the one-class objective function, which maximizes the advantages of a deep neural network to extract features customized for the anomaly detector.
• Our model simultaneously extracts global and local image features to achieve higher feature utilization levels. We calculate the relevant information between pixels based on self-attention mechanisms to obtain the image’s global features; however, we also use a local feature extraction block to mine the local spatial information of each pixel’s neighboring regions.
• We apply the anomaly weight map in the self-attention mechanism to enhance the possible anomalous regions in the original image. The differences between targets and the background are highlighted through the anomaly weight map, which helps the final anomaly detection results, especially when the target is weak and confounded by the background.

The remainder of our article is organized as follows. Section 2 presents the proposed method in detail. Section 3 introduces experimental settings and provides the detection results. Section 4 discusses the efficiency of the adversarial learning and joint training methods. Finally, Section 5 presents our conclusions.

2. Proposed Method

This section will provide a brief introduction of our proposed hyperspectral anomaly detection method based on self-attention and one-class neural networks (SAOCNNs). The SAOCNN contains two components: an attention-oriented feature extraction network and a hyperplane layer based on the OCSVM network (OCSVM-Net). The loss function constrains the extracted features to make the new high-dimensional feature space more separable by maximizing the distance between the hyperplane layer and anomaly representative point, achieving the objective of joint training feature extraction and hyperplane optimization.

2.1. Overview

Figure 1 shows the complete flow chart of our proposed method. Suppose that the original hyperspectral data are reshaped to $\mathit{X}=\left[x_1,x_2,x_3,\cdots ,x_N\right]\in {\mathbb{R}}^{k\times N}$, the image contains N pixels (the image size is $H\times W$, the total number of pixels is $N=H\times W$), and each pixel has k bands.

The original hyperspectral image $\mathit{X}$ is first input into an adversarial autoencoder (AAE) [62] to obtain the reconstructed image $\widehat{\mathit{X}}$. Due to the majority of background pixels in training samples and the constraint of the feature discriminator Dz in the hidden layer, the reconstruction performance of the background is better than that of anomalies in $\widehat{\mathit{X}}$, which will be explained in detail in Section 2.2.1. The residual image $\mathit{R}$ is obtained by the residual between the original $\mathit{X}$ and reconstruction $\widehat{\mathit{X}}$, in which the residual value of possible anomalous pixels is greater than that of background pixels.

The residual $\mathit{R}$ is considered an anomaly weight map and is input into a residual non-local network (rNon-local) together with the original $\mathit{X}$. In rNon-local, the anomaly weight map is added to enhance the possible anomaly areas in the original $\mathit{X}$, which makes the anomaly area features more prominent.

The global features of each pixel are extracted in rNon-local, while the local features are extracted in the subsequent local feature learning network, including spectral features, spectral correlations of adjacent spectral segments, and spatial correlations of adjacent pixels. The whole feature extraction network maps the hyperspectral data to a new feature space through the local feature learning network, making the anomalies and background easier to separate with hyperplanes.

The feature extraction network is followed by OCSVM-Net as the hyperplane layer to train a hyperplane that separates the anomaly from the background. The OCSVM-Net evaluates the possibility of an anomaly according to the distance between the feature vector of each pixel mapped in the feature space and the hyperplane. During the whole training process, the hyperplane layer and feature extraction network are repeatedly optimized and iterated according to the loss function, obtaining the final hyperspectral anomaly detection model.

2.2. Attention-Oriented Feature Extraction Network

As shown in Figure 1, this module consists of three parts: AAE, rNon-local, and Local feature learning Net. In most hyperspectral anomaly detection methods based on deep learning, the networks are usually constructed from stacked fully connected layers because a single pixel is just a multi-dimensional vector considered as a separate sample, leading to only spectral features being extracted in each pixel. However, the similarity between background pixels is greater than between anomaly and background pixels, which means the global correlation of background pixels is better than that of anomalous pixels. Therefore, we extracted global correlation features based on self-attention mechanisms to achieve better detection results in the rNon-local network. We added the residual map of the original and reconstructed images generated by the AAE as the anomaly weight map to enhance the possible anomaly regions, which further improves the performance of global feature extractor. In addition, local structure information is extracted through Local feature learning Net in the whole feature extraction block to improve feature utilization, including local-region spectral and spatial features extraction with two- and three-dimensional convolutional layers, respectively.

2.2.1. Anomaly Weight Map Generated Block

AAE combines the generative adversarial network’s (GAN) generalization ability [50] with the autoencoder’s (AE) reconstruction ability [48], utilizing the potential adversarial learning method to train the autoencoder. The anomaly is distinguished in the reconstructed map due to the higher reconstruction error of the anomaly compared with the background.

(1) 

Autoencoder:

AE is a two-layer neural network model for unsupervised learning features to accurately reconstruct the original input by minimizing the reconstructed error between the output $\widehat{\mathit{x}}$ and input vectors $\mathit{x}$. The AE is divided into two parts, namely, encoder E and decoder De: the encoder is a structure in which $\mathit{x}$ is projected from the input to hidden layer through a mapping function $f(·)$, and the decoder is a structure in which $\mathit{z}$ is projected from hidden to output layer through a mapping function $\widehat{f}(·)$:

$$\begin{array}{c}\hfill \mathit{z}=f(\mathit{W}\mathit{x}+\mathit{b})\\ \hfill \widehat{\mathit{x}}=\widehat{f}(\widehat{\mathit{W}}\mathit{z}+\widehat{\mathit{b}})\end{array}$$

(1)

where $\mathit{W}$ and $\widehat{\mathit{W}}$ are weight matrices; $\mathit{b}$ and $\widehat{\mathit{b}}$ are deviation vectors.

(2) 

Generative Adversarial Networks:

GAN is a generative model that learns specific data distributions, which consists of two networks: generator G and discriminator D. Let the real data $\mathit{x}$ fit the $p_{\mathrm{data}}$ distribution, the generator map the fake data $\mathit{z}$ conforming to the $p_z$ distribution to the expected distribution, and make it as close to $p_{\mathrm{data}}$ distribution as possible; subsequently, the discriminator attempts to identify the real samples in these two distributions:

$$\underset{\mathrm{G}}{min}\underset{\mathrm{D}}{max}V(\mathrm{G},\mathrm{D})=\underset{\mathrm{G}}{min}\underset{D}{max}{\mathbb{E}}_{\mathit{x}\sim p_{\mathrm{data}}\left(\mathit{x}\right)}[log\mathrm{D}\left(\mathit{x}\right)]+{\mathbb{E}}_{\mathit{z}\sim p_{\mathrm{z}}\left(z\right)}[log(1-\mathrm{D}\left(\mathrm{G}\left(\mathit{z}\right)\right))]$$

(2)

where $V(\mathrm{G},\mathrm{D})$ is maximized when optimizing the discriminator and minimized when optimizing the generator. Through continuous repetitive training, the model can reach convergence and minimize the Jensen–Shannon (JS) difference between the two distributions [52].

In hyperspectral anomaly detection, the anomaly and background belong to anomaly and background clusters in original feature space, respectively, which are supposed to belong to hidden-layer cluster A and hidden-layer cluster B, respectively, in the hidden feature space. The reconstruction process of AE for anomalous pixels can be explained as the mapping stage in Figure 2a, while the background pixels’ reconstruction process is demonstrated in Figure 2b. We expected that the background-cluster learning is better than the anomaly cluster in AE, which means the residual value between reconstruction $\widehat{\mathit{X}}$ and original $\mathit{X}$ anomalous pixels will be greater than that of background pixels, therefore highlighting the target and suppressing the background in the residual image $\mathit{R}$.

However, the reconstruction ability of AE is too strong in real situations, meaning both mapping processes are learned efficiently during training; consequently, the residual map cannot effectively highlight the target. Therefore, the feature discriminator Dz is introduced, and encoder E is equivalent to the corresponding generator. The Gaussian distribution is selected as the prior distribution in discriminator Dz because we assumed that the hyperspectral image background pixels obey Gaussian probability [19]. Adversarial learning method forces constrain hidden features $\mathit{z}$ and make all hidden-layer samples belong to the same hidden-layer cluster G. Because the number of background samples is much higher than the number of anomalies, the decoder De only learns the mapping from hidden-layer cluster G to the background cluster, as demonstrated in Figure 3. The reconstruction error of anomalous pixels will be larger, effectively highlighting the anomalies in residual images $\mathit{R}$.

The specific structure of AAE in our proposed model is shown in Figure 4, and it is composed of an encoder E, a decoder De, and a feature discriminator Dz. The encoding–decoding process is considered the reconstruction of the original data $\mathit{X}$. The reconstruction loss function is the Huber loss [63]:

$$L_{\delta }(\mathit{x},\widehat{\mathit{x}})=\left\{\begin{array}{cc}\frac{1}{2}(\mathit{x}-\widehat{\mathit{x}})& \mathrm{for}|\mathit{x}-\widehat{\mathit{x}}|\le \delta \\ \delta |\mathit{x}-\widehat{\mathit{x}}|-\frac{1}{2}{\delta }^2& \mathrm{otherwise}\end{array}\right.$$

(3)

where $\delta$ represents the hyperparameters when training.

The discriminator’s loss function is Wgan-gp [64]:

$$L_{\mathrm{Gan}}=E_{\tilde{\mathit{z}}\sim P_{\mathrm{gauss}}}\left[\mathrm{Dz}\left(\tilde{\mathit{z}}\right)\right]-E_{\mathit{z}\sim P_E}\left[\mathrm{Dz}\left(\mathit{z}\right)\right]-\lambda E_{\mathit{t}\sim P_{\mathit{t}}}\left[{\left({\parallel \mathrm{Dz}\left(\mathit{t}\right)\parallel }_2-1\right)}^2\right]$$

(4)

where $\tilde{\mathit{z}}$ is the Gaussian sampling data; $P_{\mathrm{gauss}}$ is the Gaussian distribution to which the Gaussian sampling data belongs; $P_E$ is the data distribution of the hidden-layer space; $\mathrm{Dz}(·)$ represents the output in Dz; $\mathit{t}$ is a uniform sampling of the linear space $P_{\mathit{t}}$ along the sampling points between $P_{\mathrm{gauss}}$ and $P_E$; and $E[·]$ represents the mathematical expectation function.

In our proposed model, we used a two-dimensional convolutional layer with a kernel size of 1 × 1 instead of a fully connected layer. The group normalization [65] and the LeakyReLU [66] activation function are applied between each layer.

LeakyReLU is a variant of ReLU (rectified line unit):

$$LeakyReLU\left(x\right)=\left\{\begin{array}{cc}x& \mathrm{for}x\ge 0\hfill \\ \alpha x& \mathrm{otherwise}\hfill \end{array}\right.$$

(5)

where the hyperparameter $\alpha$ is set to 0.2 in the method.

Because the 1 × 1 convolution kernel is implemented instead of a fully connected layer, the training batch is very small, and group normalization is selected in the network:

$$\mathrm{GN}\left(\mathit{x}\right)=\frac{\mathit{x}-E\left(\mathit{x}\right)}{\sqrt{Var\left(\mathit{x}\right)+\epsilon }}\ast \gamma +\beta$$

(6)

where $E\left(\mathit{x}\right)$ is the average value; $Var\left(\mathit{x}\right)$ is the standard deviation; and $\epsilon$, $\gamma$, and $\beta$ are the learned parameters.

In summary, the normalization and activation between adjacent convolutional layers can be expressed as:

$$\mathit{y}=LeakyReLU\left(\mathrm{GN}\left(\mathit{x}\right)\right)$$

(7)

where $\mathit{x}$ represents the output of the previous convolutional layer and $\mathit{y}$ represents the input of the next convolutional layer.

2.2.2. Residual Non-Local Network

The original non-local network [67] flow chart is shown as Figure 5a, and assuming $\mathit{X}=\left[x_1,x_2,x_3,\cdots ,x_N\right]$ as the input and $\mathit{Y}=\left[y_1,y_2,y_3,\cdots ,y_N\right]$ as the output, the original non-local network extracting global features is

$${\mathit{y}}_i=\frac{1}{\mathcal{C}\left(\mathit{x}\right)}\sum _{\forall j}f\left({\mathit{x}}_i,{\mathit{x}}_j\right)g\left({\mathit{x}}_j\right)$$

(8)

where i is the index of the output position; j is the index of enumerating all possible positions; $\mathit{x}$ is the input signal; and $\mathit{y}$ is the output signal. The function $g(·)$ is used to calculate the feature representation of ${\mathit{x}}_j$, and the function $f(·)$ is used to calculate the similarity between ${\mathit{x}}_i$ and ${\mathit{x}}_j$. $\mathcal{C}\left(\mathit{x}\right)$ is a function used for normalization. In the original non-local network, the similarity between the current pixel and all other pixels in the image is calculated through the function $f(·)$ to obtain the global correlation features of the current pixel by connecting local and non-local information.

However, the global features extracted by the original non-local network cannot effectively highlight the anomaly when the target is weak, which we demonstrated in an ablation experiment (Section 3.5). The non-local network is improved to a residual non-local network in order to solve the above problems (Figure 5b). The input of rNon-local is set to the original hyperspectral image $\mathit{X}$ and the anomaly weight map as the residual $\mathit{R}$, which roughly evaluates the anomaly probability of each pixel in the original image. The size of the original image and anomaly weight map is reshaped into $k\times N$, i.e., $\mathit{X},\mathit{R}\in {\mathbb{R}}^{k\times N}$. The Hadamard product of the two is presented to obtain the anomaly enhancement map $\mathit{V}=\mathit{R}\circ \mathit{X}\in {\mathbb{R}}^{k\times N}$. The possible anomalous area in $\mathit{V}$ has been enhanced compared with $\mathit{X}$.

The similarity between vector ${\mathit{v}}_i$ and ${\mathit{x}}_j$ is calculated and the Gaussian function is chosen:

$$f\left({\mathit{v}}_i,{\mathit{x}}_j\right)=e^{{\mathit{v}}_i^T{\mathit{x}}_j}$$

(9)

Softmax is used to perform normalization; therefore, the rNon-local is equivalent to the self-attention mechanism. The global feature response of each position in the output of the rNon-local is

$$\begin{array}{cc}\hfill {\mathit{y}}_i& =\frac{1}{{\mathcal{C}}_i}\sum _{\forall j}f\left({\mathit{r}}_i\circ {\mathit{x}}_i,{\mathit{x}}_j\right)g\left({\mathit{x}}_j\right)\hfill \\ \hfill {\mathcal{C}}_i& =\sum _{\forall j}f\left({\mathit{r}}_i\circ {\mathit{x}}_i,{\mathit{x}}_j\right)\hfill \end{array}$$

(10)

The integral global correlation feature $\mathit{Y}\in {\mathbb{R}}^{N\times k}$ is

$$\mathit{Y}=softmax\left({(\mathit{R}\circ \mathit{X})}^T\mathit{X}\right)g\left(\mathit{X}\right)$$

(11)

Finally, the output $\mathit{Z}\in {\mathbb{R}}^{N\times k}$ of rNon-local is expressed with the residual connection:

$$\mathit{Z}={\mathit{W}}_z\mathit{Y}+{\mathit{X}}^T$$

(12)

where ${\mathit{W}}_z$ is the weight matrix of the residual connection, $\mathit{Y}$ is given in Equation (11), and “$+{\mathit{X}}^T$” represents the residual connection, which allows the insertion of a new non-local block and maintains the initial network when it is a zero matrix.

2.2.3. Local Feature Extraction Block

Because the rNon-local network with the anomaly weight-map-generated block only learns the global correlation features of hyperspectral data, the Local feature learning Net, which is composed of a two- and three-dimensional convolution layer with kernel sizes of 1 × 1 and 3 × 3 × 3, respectively, is applied to further extract local features. The two-dimensional convolution layer is used to extract the spectral features of each pixel, while the three-dimensional convolution layer simultaneously extracts the spatial features of adjacent pixels and the spectral correlation of adjacent bands.

The two convolutional layers are followed by group normalization layers and the activation function before being input to the next layers. The convolutional layers’ outputs in the local feature extraction block are normalized after self-multiplication:

$$\mathit{y}=LeakyReLU\left(\mathrm{GN}\right(\mathit{x}\circ \mathit{x}\left)\right)$$

(13)

where $\mathit{x}\circ \mathit{x}$ stands for the self-multiplication of $\mathit{x}$, which makes the distribution of samples much denser in the feature space; therefore, it is more conducive to the division of the hyperplane in the next step.

The three-dimensional convolutional layer is implemented on the expanded data with one extra dimension. Compared with the 3 × 3 two-dimensional convolution that is directly performed on the original data, the three-dimensional convolution can not only fuse the spatial adjacent pixels, similar to two-dimensional convolution, but it also performs the convolution on the spectral dimension to extract the spectral correlation features of adjacent bands, which cannot be achieved by the simple two-dimensional convolution.

Because the three-dimensional convolutional layer extracts the spatial features of targets by a convolution kernel with a set size, the performance of this convolution layer will be affected by the anomaly size and background complexity. The detection results of the model with a three-dimensional convolution layer, and the results of the model without it, are compared in Section 3.5.

2.3. OCSVM-Net-Based Hyperplane Layer

The final hyperplane layer and the objective function are proposed based on the OC-SVM algorithm [59], in which all data points are considered as positive instances, while the origin is the only negative instance. When the hyperspectral image $\mathit{X}$ is directly used as the input of the hyperplane layer, suppose the output is the final anomaly score $A=\left[a_1,a_2,a_3\cdots ,a_N\right]\in {\mathbb{R}}^N$, in which the value of a is the anomaly score of each pixel. The linear decision function of this layer is defined as:

$$a_n={\omega }^T{\mathit{x}}_n+b$$

(14)

where $\omega$ and b are the weight and bias parameters, respectively, and ${\omega }^T{\mathit{x}}_n+b=0$ is the hyperplane represented by this layer.

We expected that the hyperplane would be far away from the origin, which represents the anomaly, and that as many as possible background vectors would be on the same side of the hyperplane. That is, the pixel may be a background pixel when a is negative or close to 0, while the larger the value of a is, the more likely the pixel is anomalous. The optimization function can be expressed as:

$$\underset{\omega ,b}{min}\frac{1}{2}{\parallel \omega \parallel }_2^2+\frac{1}{v}·\frac{1}{N}\sum _{n=1}^{N}max\left(0,{\omega }^T{\mathit{x}}_n+b\right)-b$$

(15)

where $v\in (0,1)$ is a parameter that weighs the two problems of maximizing the distance between the hyperplane, origin, and the data allowed to pass through the hyperplane.

Equation (15) is the optimization function when the hyperplane layer is trained separately, and the optimization function of the traditional OC-SVM is

$$\underset{\omega ,b}{min}\frac{1}{2}{\parallel \omega \parallel }_2^2+\frac{1}{v}·\frac{1}{N}\sum _{n=1}^{N}max\left(0,〈\omega ,\Phi \left({\mathit{x}}_n\right)〉+b\right)-b$$

(16)

where $\Phi (·)$ represents the kernel function to map the input data into a high-dimensional space, in which the background and anomalies are linearly separable. The kernel function is a fixed function, and its selection has a considerable influence on the performance of a traditional OC-SVM in anomaly detection.

Inspired by the one-class neural network (OC-NN) model [68], which designs a neural architecture based on an OC-SVM-equivalent loss function, the feature mapping method in our proposed model is a feature extraction network in place of a kernel function, which is the main difference compared with the traditional OC-SVM. Although they play the same role, the parameters in the proposed feature extraction network can be optimized because of the deep neutral network’s learning ability; however, the parameters in a manually selected kernel function are fixed. Therefore, the feature extraction network is added into the optimization function of the proposed model:

$$\underset{\omega ,b,\Psi }{min}\frac{1}{2}{\parallel \omega \parallel }_2^2+\frac{1}{v}·\frac{1}{N}\sum _{n=1}^{N}max\left(0,〈\omega ,\Psi \left({\mathit{x}}_n\right)〉+b\right)-b$$

(17)

where $\Psi (·)$ represents the feature extraction network, which replaces $\Phi (·)$ in Equation (16) and makes it possible to leverage and refine the features learned by the extraction network.

The hyperplane layer is a two-dimensional convolutional layer with a 1 × 1 convolution kernel. The loss function is calculated through Equation (17):

$$L_{\mathrm{OCSVM}}=\frac{1}{2}{\parallel \omega \parallel }_2^2+\frac{1}{v}·\frac{1}{N}\sum _{n=1}^{N}max\left(0,〈\omega ,\Psi \left({\mathit{x}}_n\right)〉+b\right)-b$$

(18)

Because $\omega$, b, and $\Psi$ are all in the form of network parameters, the error back propagation algorithm is used to minimize the loss function during training. The loss function in Equation (18) drives the optimization of the hyperplane parameters and feature extraction network parameters simultaneously, making the features obtained by the extraction network more conducive to the anomaly detector. The OCSVM loss function makes it possible for the feature extraction network to update together with the anomaly detector, i.e., OCSVM-Net.

2.4. Training Steps

The pseudocode of the SAOCNN is summarized as Algorithm 1, in which LFL-Net stands for the local feature learning network. The hyperspectral images with no or few anomalous pixels are used as the training sample. The AAE part is trained separately in the pre-training period to make the network training more stable. Then, it is trained together with the rNon-local, LFL-Net, and OCSVM-Net. The number of times that the AAE is trained alone is AAE_epoch_num. When the number of training epochs is larger than the AAE_epoch_num, a joint training approach is carried out, which updates and iterates the OCSVM-Net with the feature extraction network. The AAE is first trained under the constraint of $L_{\delta }$ and $L_{\mathrm{GAN}}$ and then updated in the back propagation of $L_{\mathrm{OCSVM}}$.

Algorithm 1: Framework of SAOCNN

3. Experiments Section

We used six real hyperspectral images and five methods for comparison, including traditional anomaly detection methods, such as GRX [19], CRD [39], and LSAD [69], and two recent anomaly detection methods based on deep learning, namely, HADGAN [52] and LREN [47], to verify the performance of our proposed hyperspectral anomaly detection algorithm. Because three-dimensional convolution can play different roles according to the anomaly size in images, we also introduced another version without the use of three-dimensional convolution for spatial feature extraction named SAOCNN_NS, where “NS” stands for “no spatial features”.

3.1. Datasets Description

We evaluated our proposed method by six public hyperspectral datasets, including Coast [15], Pavia [70], Washington DC Mall, HYDICE [71], Salinas, and San Diego [72].

Coast: Coast dataset was acquired by Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor, which contains 100 × 100 spectral vectors, each with 207 channels, covering a range of 450 to 1350 nm. There are 141 bands left after removing some noise interference bands in the data. Buildings of different sizes in the image were marked as anomalies, and the total number of anomalous pixels was 155, accounting for 1.55%; furthermore, the number of pixels occupied by one single target was less than 20.

Pavia: Pavia dataset was acquired by Reflective Optics System Imaging Spectrometer (ROSIS) sensor over the Pavia center area. A sub-image with 100 × 100 px, 102 spectral bands in a wavelength ranging from 430 to 860 nm, and with a 1.3 m spatial resolution was selected from the hyperspectral data. The background was composed of rivers and bridges, and the anomalous targets were vehicles driving on the bridge, as well as bare soil near the bridge piers. The anomaly distribution map was obtained in the ENVI software by manual annotation, and the total number of anomalous pixels was 61, accounting for 0.61%; furthermore, the pixel number of each anomalous target ranged from 4 to 15.

Washington DC Mall: This dataset is composed of airborne hyperspectral data for Washington DC Mall. Its size is 1208 × 307 and contained 191 bands after discarding some useless bands. We selected a sub-image with 100 × 100 px where the main background was vegetation, and the anomalous targets were the two man-made buildings in the image, occupying 7 and 12 pixels, respectively, accounting for 0.19% in total.

HYDICE urban: This dataset was collected by the Hyperspectral Digital Imagery Collection Experiment (HYDICE) airborne sensor for an urban area. A sub-image of this dataset was selected, which had 80 × 100 pixels and 162 spectral bands in a wavelength ranging from 400 to 2500 nm. The background contained vegetation, highways, and parking lots, and the image’s anomalous targets were vehicles. The total number of anomalous pixels was 19, accounting for 0.24%; furthermore, the pixel number of each anomalous target ranged from 1 to 4.

Salinas Valley: This dataset was collected by the high-spatial-resolution, 224-band AVIRIS sensor over Salinas Valley, California. It had 204 bands after discarding the 20 water absorption bands. The original data size was 512 × 217, and we selected the 200 × 200 sub-image where the background was mainly various crops and roads between farmland; furthermore, the anomalous targets were man-made houses of different sizes. We obtained the anomaly distribution map in ENVI software by manual annotation, and the total number of anomalous pixels was 102, accounting for 0.22%; furthermore, the size of one single target ranged from 10 to 50 pixels.

San Diego: This dataset was collected by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor from San Diego, CA, USA. The spatial resolution was 3.5 m per pixel. We selected the 90 × 90 sub-image, which contained 189 bands and a spectral range from 400 to 2500 nm. The background included aprons, grass, roofs, and shadows. The three aircraft were considered in the scene as anomalous targets, and they occupied 134 pixels, accounting for 1.34%; furthermore, the size of one single target ranges from 30 to 50 pixels.

In the hyperspectral image datasets above, we considered anomalous targets in San Diego and Salinas as planar targets, while smaller targets in others are considered small targets.

3.2. Evaluation Metric

We used receiver operating characteristic (ROC) curves and the area under the ROC curve (AUC) to evaluate detection results [73] and quantitatively analyze and compare the detection performance of our proposed method with other methods.Two types of ROC curves are applied in our experiment through the three parameters of $P_d$, $P_f$, and $\tau$, which represent the true positive rate, the false positive rate, and the detection threshold, respectively. The first type is the ROC curve of $(P_d,P_f)$, which indicates the trade-off between $P_d$ and $P_f$ and the closer curve $(P_d,P_f)$ is to the upper-left corner; furthermore, the closer the area is to 1, the better the detection performance. The other type is the ROC curve of $(P_f,\tau )$, which reports $P_f$ at each threshold, and the closer curve $(P_f,\tau )$ is to the lower-left corner, the closer the area is to 0, which means the false alarm rate is lower.

In addition, we also used box plots [74] to show the difference in the separation between the background and anomalies of each method in a more intuitive way. In the box plot, the red and green boxes, respectively, represent the anomaly data distribution and background in the detection map, where the bottom and top of each box represent 25% and 75% of the sample. Thus, a greater distance between the red and green boxes means a better separation between background and anomalies, and a narrower green box indicates a better background suppression.

3.3. Experimental Setting

As mentioned in Section 2, our proposed method consists of two parts: a feature extraction network and an OC-SVM layer. In the feature extraction network, both the E and De of the AAE are composed of four convolutional layers with kernel sizes of 1 × 1 and the number of hidden-layer spatial features set to 25; meanwhile, the Dz of the AAE contains three convolutional layers with kernel sizes of 1 × 1 and one average pooling layer. We applied a two-dimensional convolutional layer with a convolution kernel size of 1 × 1 and a three-dimensional convolution layer with a convolution kernel size of 3 × 3 × 3 in the local feature extraction block, reducing the number of original hyperspectral image features to 100 dimensions. In the OC-SVM layer, we employed a two-dimensional convolutional layer with a convolution kernel size of 1 × 1 instead of the single fully connected network.

Adam is used to optimize the network parameters during the training process. The network learning rate is set to ${10}^{-5}$, which decays exponentially during training. In the optimization functions, we set v in Equation (18) to 0.1 and $\lambda$ in Wgan-GP (Equation (4)) to 10. We used Pytorch to implement our approach, and all the experiments were performed on a GeForce GTX 1080Ti graphics card and an Intel(R) Core (TM) i9-7920U CPU machine with 64 GB of RAM.

3.4. Experimental Results

Anomaly detection on six hyperspectral image datasets using the two proposed algorithms, namely, the SAOCNN and SAOCNN_NS, and the five contrast algorithms were performed. Figure 6 shows the anomaly detection maps of each algorithm. Figure 7 is the ROC curve $(P_d,P_f)$ and

Table 1. ${\mathrm{AUC}}_f$ of SAOCNN, SAOCNN_NS, HADGAN, LREN, LSAD, CRD, and GRX. (The maximum is in bold and the second largest is underlined).

	SAOCNN	SAOCNN_NS	HADGAN	LREN	LSAD	CRD	GRX
Coast	0.9962	0.9993	0.9925	0.9966	0.9261	0.9213	0.9969
Pavia	0.9977	0.9962	0.9922	0.9948	0.9541	0.9135	0.9885
DC	0.9991	0.9997	0.9999	0.9479	0.9766	0.9298	0.9989
HYDICE	0.9908	0.9997	0.9918	0.9779	0.9929	0.9868	0.9763
Salinas	0.9990	0.9812	0.9944	0.9702	0.9465	0.9474	0.9949
SanDiego	0.9962	0.9884	0.9773	0.9359	0.7645	0.7870	0.9106
Average	0.9965	0.9941	0.9913	0.9705	0.9268	0.9143	0.9777

lists the ${\mathrm{AUC}}_f$ (AUC of curve $(P_d,P_f)$) for each algorithm, in which the largest ${\mathrm{AUC}}_f$ values are indicated in bold and the second largest are underlined. Figure 8 is the ROC curve $(P_f,\tau )$, and

Table 2. ${\mathrm{AUC}}_{\tau }$ of SAOCNN, SAOCNN_NS, HADGAN, LREN, LSAD, CRD, and GRX. (The minimum is in bold and the second smallest is underlined).

	SAOCNN	SAOCNN_NS	HADGAN	LREN	LSAD	CRD	GRX
Coast	0.0015	0.0254	0.0133	0.0091	0.0042	0.0041	0.0088
Pavia	0.0011	0.0026	0.0185	0.0423	0.1485	0.1171	0.0259
DC	0.0027	0.0240	0.0395	0.1759	0.0697	0.0552	0.0452
HYDICE	0.0138	0.0492	0.0749	0.0088	0.0693	0.0351	0.0380
Salinas	0.0020	0.0170	0.0016	0.0213	0.0082	0.0102	0.0022
SanDiego	0.0389	0.1464	0.1122	0.1937	0.2177	0.1459	0.0297
Average	0.0100	0.0441	0.0433	0.0752	0.0863	0.0613	0.0250

lists the ${\mathrm{AUC}}_{\tau }$ (AUC of curve $(P_f,\tau )$) for each algorithm, in which the minimums are indicated in bold and the second smallest are underlined. In addition, the box plot in Figure 9 can be used to analyze the separation of the background and anomalies of each algorithm on the six datasets.

3.4.1. Performance for Small Target

The three hyperspectral image datasets of Coast, Pavia, and Washington DC Mall have small targets and simple backgrounds; therefore, the anomaly detection in these images is the simplest to perform. Figure 6a–c, shows that the targets in detection maps of our proposed methods are prominent; moreover, the suppression on the background is better. The ${\mathrm{AUC}}_f$ values of the proposed methods are approximate to the ideal value of 1 in Table 1; furthermore, they are usually the maximum or the second largest of all of the methods. In addition, the ${\mathrm{AUC}}_{\tau }$ values of SAOCNN are about 0.001 which is smaller than other methods in Table 2 and indicates that the SAOCNN has a lower false alarm rate. Compared with CRD and LSAD, the detection results of our methods have a higher contrast between anomalies and background and have a better suppression of the background according to our visual perception of the detection maps in Figure 6 and the box plot shown in Figure 9, especially in the Pavia dataset. Compared with GRX, although the ROC curves $(P_d,P_f)$ of SAOCNN are very close to GRX in Figure 7, the SAOCNN is faster than approach 1, and the ${\mathrm{AUC}}_f$ values of our proposed methods are slightly higher than those of GRX. Compared with LREN and HADGAN, which were recently proposed as deep learning methods, the detection maps of our proposed methods consist of fewer backgrounds and more anomalies. Moreover, the ROC curves of $(P_d,P_f)$ for SAOCNN_NS are presented at the top-left corner, indicating that our proposed network is more effective for extracting the deep features of anomalies while suppressing the background compared with LREN and HADGAN do. Although, the ROC curves of $(P_d,P_f)$ for HADGAN and SAOCNN_NS in the DC dataset are very close, and the former reaches 1 the fastest; it can be seen from the box plot that the green boxes of the proposed method are at a lower level, which indicates a better background suppression.

In the HYDICE dataset, the targets are smaller at a pixel level and the background is more complex compared with the Coast, Pavia, and Washington DC Mall datasets. Compared with the traditional LSAD, CRD, GRX, and the deep-learning-based HADGAN, our proposed methods produce visual results that are closer to the reference images. Moreover, the green boxes of our proposed methods are narrower in the box plot shown in Figure 9, indicating a better efficiency in background suppression. Compared with the LREN, our proposed method’s background suppression is slightly inferior in detection maps. The LREN and SAOCNN are very close and lower than others—${\mathrm{AUC}}_{\tau }$ of 0.0088 and 0.0138 shown in Table 2. However, the ROC curves of $(P_d,P_f)$ shown in Figure 7 demonstrate that the true positive rate $P_d$ of SAOCNN_NS is the fastest to reach 1, while $P_d$ of the LREN is the last to reach 1 because some pixels are not detected. We found that the LREN may have a lower false alarm rate due to its better background suppression; however, it causes missed detection for some targets and makes the ${\mathrm{AUC}}_f$ value smaller. Our proposed methods present a superior balance between detection and false alarm rates.

For small targets, our methods perform well and have better background suppression rates with proper target prominence in both simple and complex backgrounds compared with other contrast methods. We compared the four ROC curves and found that curves of the proposed SAOCNN_NS are basically the closest to the top-left corner, and the proposed SAOCNN can quickly reach 1 in most of the datasets, except for HYDICE. The detection maps in Figure 6 show that targets detected by the SAOCNN are larger than the real ones annotated in reference maps, and the main reason is that a three-dimensional convolution kernel is added to the local feature extraction network. Especially in the HYDICE dataset, most of the anomalies are point targets, while the sizes of detected targets in the detection map appear to be much larger. In addition, the role of the three-dimensional convolution kernel in the network is to extract spatial features (such as texture and shape features), which the point target is lacking in. However, the background in this dataset is more complicated, with obvious spatial background features, and is therefore more easily confused with targets, which means that the SAOCNN presents a lower detection probability compared with others when the false alarm rate is low in the detection results. Without three-dimensional convolution, targets detected by the SAOCNN_NS seem to be at a proper size in Figure 6, and it presents a better balance between detection probability and false alarm rate in Figure 7. However, the SAOCNN_NS does not perform as well as the SAOCNN in background suppression, with the latter presenting a lower false alarm rate in Figure 8.

3.4.2. Performance for Planar Target

The Salinas dataset has larger targets and a simpler background. The detection results of each algorithm on the dataset are shown in Figure 6e. Targets in the detection maps of SAOCNN are the most prominent, and the SAOCNN_NS and LREN background suppression levels are not as high as SAOCNN. Targets in the detection maps of HADGAN, LSAD, CRD, and GRX are almost invisible, and there are obvious false alarm pixels. Figure 7 shows the SAOCNN ROC curve of $(P_d,P_f)$ is at the top-left corner and is the first one to reach 1, with the largest AUC value of 0.9990. From the box plot shown in Figure 9, the HADGAN, LSAD, CRD, and GRX red boxes and green boxes are very close, with very low positions. However, the red boxes of our proposed method are relatively high, indicating that the four contrast methods are inferior when separating anomalies from the background. Compared with the LREN, the SAOCNN_NS has a similar performance level in all aspects. However, upon adding a three-dimensional convolution kernel, the ${\mathrm{AUC}}_f$ value of SAOCNN improves by 2.968% compared with the LREN. Meanwhile, its background suppression is also better, with a narrower green box in shown in the box plot.

In the San Diego dataset, the targets are three airplanes that are not only larger but also have certain shape features, and the background is more complicated, with some roof areas that may be easily detected as anomalies compared with other datasets. From the detection results shown in Figure 6f, the targets detected by the SAOCNN are very prominent and completely retain their basic shape structures with superior background suppression. Although targets are detected in the SAOCNN_NS, the suppression of the roof areas in the background is imperfect. The HADGAN and LREN detection maps only highlight part of the anomalous pixels and preserve some background details, while targets in the LSAD, CRD, and GRX methods are almost invisible. In the ROC curves of $(P_d,P_f)$ in Figure 7, the SAOCNN is at the top-left corner and is the fastest to approach 1, and the probability of its detection is always higher than in other methods when the false alarm rate is about 0–0.01. For ${\mathrm{AUC}}_f$ in Table 1, the SAOCNN has the largest value (0.9962), followed by the SAOCNN_NS (0.9884), which is a considerable increase compared with other methods. Figure 8 shows that the ROC curve of $(P_f,\tau )$ of GRX is slightly lower than the SAOCNN, with Table 2 showing the lowest ${\mathrm{AUC}}_{\tau }$ value as 0.0297. This may be because the GRX green box in the box plot of Figure 9 is smaller, resulting in a lower false alarm rate at a lower threshold. However, the red box of the targets in GRX is lower and extremely close to the green box, while our proposed methods showed red and green boxes farther apart, which better distinguished the anomaly from the background.

For planar targets, the two versions of our proposed method have better target prominence and recognize the approximate shape of anomalies in the detection maps. However, the anomalous targets in other contrast methods are usually submerged in the background, which is not conducive to detection. Comparing the two proposed methods, the SAOCNN_NS does not present as well on background suppression as the SAOCNN. Especially when the background is complex, the SAOCNN has high levels of superiority because of assistance from the three-dimensional convolution kernel, which extracts the favorable spatial features of targets and inhibits the unfavorable features.

3.4.3. Detection Accuracy and Efficiency

Table 1 lists the average ${\mathrm{AUC}}_f$ for all algorithms of the six datasets. The average SAOCNN and SAOCNN_NS ${\mathrm{AUC}}_f$ values are 0.9965 and 0.9941, respectively, while the HADGAN, LREN, LSAD, CRD, and GRX values are 0.9913, 0.9705, 0.9268, 0.9143, and 0.9777, respectively, which indicates that the two versions of our method are superior. The ${\mathrm{AUC}}_f$ values of our proposed methods are close to the ideal value 1 for all datasets, even if the lowest detection accuracy is 0.9812 for the SAOCNN_NS in the Salinas dataset. Table 2 lists the average ${\mathrm{AUC}}_{\tau }$ for all algorithms. The average minimum ${\mathrm{AUC}}_{\tau }$ value of the SAOCNN is 0.0100, followed by GRX at 0.0250. The SAOCNN_NS is 0.0441, which is larger than the HADGAN and GRX but less than the LREN, LSAD, and CRD. In summary, the SAOCNN has superior performance for accuracy in anomaly detection, with higher detection and lower false alarm rates compared with the other methods, which demonstrates that the anomalies are sufficiently recognized and match the detection requirements.

Both the HADGAN and SAOCNN introduce the AAE to reconstruct hyperspectral data and highlight anomalies. Their difference is that the traditional anomaly detector RX is combined in the subsequent step to obtain the final anomaly detection map in the HADGAN, while we proposed OCSVM-Net to train together with a feature extraction network to obtain the results in the SAOCNN. Through the above experiments, it can be seen that although they both effectively highlighted anomalies, the box plot showed that the HADGAN method is far less effective in suppressing the background than the SAOCNN, with obvious background texture features displayed in the detection maps. This proves that the idea of jointly training in the SAOCNN may extract deep features that are more conducive to detecting anomalies, trading a balance between abnormal prominence and background suppression.

Due to the random neural network training, we repeated our network training experiments 10 times to verify the credibility of our experimental results. The obtained results, including average and standard deviation (std), are shown in

Table 3. ${\mathrm{AUC}}_f$ of SAOCNN and SAOCNN_NS in repeated experiments.

	SAOCNN		SAOCNN_NS
	Average	Std	Average	Std
Coast	0.987853	0.004966	0.988417	0.017487
Pavia	0.996868	0.000658	0.991104	0.004563
DC	0.998792	0.000318	0.995962	0.003893
HYDICE	0.975760	0.026773	0.983838	0.025194
Salinas	0.997116	0.001316	0.969657	0.008905
SanDiego	0.983805	0.006330	0.969812	0.020098

, and they prove that our proposed methods are also stable.

In addition to accuracy, the calculation efficiency is also a considerable aspect of detection performance.

Table 4. Operation time in seconds of SAOCNN, SAOCNN_NS, HADGAN, LREN, LSAD, CRD, and GRX. (The minimum is in bold and the second is underlined).

	SAOCNN	SAOCNN_NS	HADGAN	LREN	LSAD	CRD	GRX
Coast	0.0229	0.0129	1.8826	99.9463	12.4239	1.5438	0.0548
Pavia	0.0189	0.0099	1.4586	99.4378	7.0819	0.8629	0.0532
DC	0.1415	0.0569	2.2735	101.0966	12.9656	1.3515	0.1023
HYDICE	0.0169	0.0079	1.6805	73.0596	10.3797	1.1974	0.0596
Salinas	0.2263	0.1117	7.5416	384.9072	57.3966	6.5627	0.2836
SanDiego	0.1132	0.0149	1.9432	73.5684	47.0938	5.5229	0.0666

records the operation time of each algorithm for six datasets. The shortest time value in the table is displayed in bold, and the second is underlined. Our proposed methods have the shortest operation times, mainly due to the fewer network layers and the acceleration of the GPU. Compared with the SAOCNN, the SAOCNN_NS has no three-dimensional convolution layer, leading to a faster calculation. Although the LREN also utilizes a deep neural network, it has the longest operation time. Networks in the LREN are only used for dimension reduction, feature extraction, and dictionary generation. However, an anomaly detector is still the traditional method based on low-rank representation and solves weight coefficients through iterative optimization, which greatly reduces the operation efficiency.

3.5. Ablation Study

To verify the effectiveness of the major components in SAOCNN, we designed a set of ablation experiments based on the benchmark OC-NN [68]. In this experiment, the following combinations are used for comparison:

• SAOCNN: our complete proposed method based on OC-NN, in which the non-local method and three-dimensional convolutional layer using anomalous weight map is added;
• SAOCNN_NS: rNon-local mechanism with an anomaly weight map is added in OC-NN, but without a three-dimensional convolutional layer to extract spatial features. “NS” stands for “No spatial feature”;
• OCNN: the original OCNN is used as a benchmark for comparing the detection results of ablation experiments. However, the difference is that the loss function is Equation (18) and error back propagation is presented to optimize the parameters;
• OCNN_S: a three-dimensional convolutional layer is added to extract spatial features based on OCNN. “S” stands for “Spatial feature”;
• OCNN_NL: the original non-local network is added to extract global features based on OCNN. “NL” stands for “non-local network”;
• SAOCNN_NA: the original non-local network and a three-dimensional convolutional layer are added based on OCNN. Compared with the complete algorithm, the residual weight map of AAE is not used in the non-local network. “NA” stands for “No AAE”.

The detection results of each module combination in the ablation experiment are shown in Figure 10. The ROC curves of $(P_d,P_f)$ are shown in Figure 11, and the corresponding AUC values are listed in

Table 5. ${\mathrm{AUC}}_f$ of ablation study. (The maximum is in bold and the second largest is underlined).

	SAOCNN	SAOCNN_NS	OCNN	OCNN_S	OCNN_NL	SAOCNN_NL
Coast	0.9962	0.9993	0.9993	0.9934	0.9992	0.9959
Pavia	0.9977	0.9962	0.9967	0.9985	0.9946	0.9978
DC	0.9991	0.9997	0.9997	0.9993	0.9999	0.9991
HYDICE	0.9908	0.9997	0.9936	0.9877	0.9977	0.9693
Salinas	0.9990	0.9812	0.9778	0.9980	0.9588	0.9981
SanDiego	0.9962	0.9884	0.9844	0.9862	0.9842	0.9927
Average	0.9965	0.9941	0.9919	0.9939	0.9891	0.9922

. Finally, the box plot is used in Figure 12 to show the separation between the background and anomalies on different datasets in each module combination. In addition, the repeated experiment is likewise presented in

Table 6. ${\mathrm{AUC}}_f$ of OCNN, OCNN_S, OCNN_NL, and SAOCNN_NA in repeated experiments.

	OCNN		OCNN_S		OCNN_NL		SAOCNN_NA
	Average	Std	Average	Std	Average	Std	Average	Std
Coast	0.978943	0.031274	0.979385	0.019868	0.990201	0.018966	0.957941	0.028588
Pavia	0.964453	0.036719	0.983363	0.017115	0.970988	0.021899	0.953301	0.020565
DC	0.995598	0.005574	0.965748	0.068885	0.994500	0.008199	0.996005	0.003954
HYDICE	0.966670	0.033513	0.936631	0.019346	0.992880	0.007649	0.920196	0.013655
Salinas	0.975203	0.006805	0.995252	0.007107	0.962751	0.021042	0.994950	0.002550
SanDiego	0.953151	0.011778	0.953332	0.017886	0.968287	0.016956	0.970973	0.012985

to verify the reliability of results.

(1) 

AAE

The AAE is the first module in the whole network and is applied to generate the anomaly weight map to highlight the possible anomalous areas. By comparing the SAOCNN and SAOCNN_NA, we can see that the ROC curves of SAOCNN are higher than that of SAOCNN_NA and reach the ideal 1 faster (Figure 11). The average ${\mathrm{AUC}}_f$ value increased by 0.433%, which indicated that AAE’s assistance can improve the detection probability of targets with a low false alarm rate.

(2) 

rNon-local Network

The main function of non-local network is to obtain global features by calculating the correlation between the test pixel and all other pixels in the image. In our ablation study, the OCNN_NL adds the ordinary non-local network based on the benchmark OCNN. The average ${\mathrm{AUC}}_f$ value of the OCNN is 0.029 larger than the OCNN_NL in Table 5, which indicates that the simple non-local network cannot improve the detection performance of the OCNN. Furthermore, the background suppression in the OCNN_NL detection maps is worse than that of the OCNN in Figure 10. The main reason for this may be that the non-local network enhances both the targets and background by extracting the global features of each pixel in a way that is not directed at anomalies.

We added the residual image reconstructed by the AAE to improve the non-local structure as rNon-local, which refers to the SAOCNN_NS in the experiment, enhancing the possible anomalous areas in the feature map produced by the non-local network. Comparing the ${\mathrm{AUC}}_f$ of the SAOCNN_NS and OCNN in Table 5, the SAOCNN_NS detection results are better compared with the OCNN on each dataset, with the average value increased by 0.222%. From the box plot in Figure 12, we observed that the distances between the red and the green boxes of the SAOCNN_NS are longer, which suggests that it is necessary to combine the AAE and rNon-local because the use of the anomaly weight map in rNon-local effectively enhances the features of anomalous areas.

(3) 

The 3D convolutional layer

The OCNN_S adds a three-dimensional convolutional layer to extract spatial features based on the OCNN. The average ${\mathrm{AUC}}_f$ value of OCNN_S is 0.202% higher than that of the OCNN in Table 5. The ROC curves of OCNN_S reach 1 faster compared with the OCNN in the four datasets shown in Figure 11, except for Coast and HYDICE. The green boxes of OCNN_S are narrower than that of the OCNN in the box plot comparison in Figure 12, which means that OCNN_S demonstrates a better background suppression than the OCNN.

In the Coast dataset, the target sizes in the detection map of OCNN_S are much larger than the real ones, causing lower ${\mathrm{AUC}}_f$ values because of the three-dimensional convolution kernel (Section 3.4.1). In the HYDICE dataset, the background contains more spatial features compared with the anomaly. The three-dimensional convolution layer used in the OCNN_S cannot extract enough spatial information from the target’s very small pixels, resulting in a few false alarm targets in the detection map of OCNN_S (Figure 10d). For the datasets Salinas and San Diego with larger targets and certain spatial features, the performance of OCNN_S is considerably better than that of the OCNN. Figure 10e,f shows that targets are more prominent in the detection maps of OCNN_S; furthermore, its background suppression is also better.

It can be concluded that a three-dimensional convolutional layer is more advantageous for targets with shape structures and can suppress backgrounds that lack spatial features, thus increasing the contrast between the background and targets.

(4) 

Non-local network combined with 3D convolutional layer

A non-local network and a three-dimensional convolutional layer are used in the SAOCNN_NA. Table 5 shows that the optimal method in the average ${\mathrm{AUC}}_f$ is the OCNN_S, followed by the SAOCNN_NA and OCNN. The three methods’ ${\mathrm{AUC}}_f$ values are very close in the Coast, Pavia, and Washington DC Mall datasets; however, the OCNN performs best in the HYDICE dataset, and the SAOCNN_NA is the best in the Salinas and San Diego datasets. The reason may be that the ordinary non-local network amplifies both the advantages and disadvantages of the three-dimensional convolutional layer.

In summary, the non-local network and the three-dimensional convolution layer we added to the OC-NN can effectively improve anomaly detection performance; furthermore, the anomaly weight map of AAE is added to enhance the possible anomalous regions in the feature map extracted by the improved rNon-local network. Therefore, the detection performance of SAOCNN reached the optimal level, with average ${\mathrm{AUC}}_f$ values exceeding 0.464% of the benchmark OCNN, and the ${\mathrm{AUC}}_f$ is significantly increased, especially in the Salinas and San Diego datasets.

4. Discussion

We carried out a training analysis with four scenarios to verify the effectiveness of the training tricks in our proposed methods, including adversarial learning and joint training approaches. The ${\mathrm{AUC}}_f$ values for each dataset are shown in

Table 7. ${\mathrm{AUC}}_f$ of training study. (The maximum is in bold and the second largest is underlined).

	AAE+OCSVM	AAE+OCSVM(se)	AE+OCSVM	AE+OCSVM(se)
Coast	0.9962	0.9946	0.9932	0.9931
Pavia	0.9977	0.9982	0.9989	0.9985
DC	0.9991	0.9993	0.9990	0.9994
HYDICE	0.9908	0.9914	0.9852	0.9861
Salinas	0.9990	0.9986	0.9983	0.9977
SanDiego	0.9962	0.9928	0.9856	0.9839
Average	0.9965	0.9959	0.9933	0.9931

. Figure 13 displays the ROC curves $(P_d,P_f)$, and Figure 14 shows the box plots for each scenario. In addition, the repeated experiment is also presented in

Table 8. ${\mathrm{AUC}}_f$ of OCNN, OCNN_S, OCNN_NL, and SAOCNN_NA in repeated experiments.

	AAE+OCSVM(se)		AE+OCSVM		AE+OCSVM(se)
	Average	Std	Average	Std	Average	Std
Coast	0.991742	0.005183	0.989259	0.003943	0.992701	0.000304
Pavia	0.996883	0.003059	0.997840	0.000697	0.997279	0.002112
DC	0.999121	0.000133	0.998961	0.000422	0.999260	0.000136
HYDICE	0.985342	0.007138	0.975644	0.005904	0.981095	0.005834
Salinas	0.997680	0.001136	0.996643	0.002436	0.996969	0.000538
SanDiego	0.987765	0.003566	0.982193	0.001865	0.982184	0.001213

.

• AAE+OCSVM: our proposed method with adversarial learning (i.e., an additional discriminator Dz) and joint training for the whole network. “+OCSVM” stands for the rest part, except for the anomaly weight-map-generated block;
• AAE+OCSVM(se): a two-step training version of our proposed network. The AAE is first trained with $L_{\delta }$ and $L_{\mathrm{GAN}}$ separately, and then the trained AAE is fixed and used to help train the rest part with $L_{\mathrm{OCSVM}}$. “se” stands for separate;
• AE+OCSVM: the AE is used to generate the anomaly weight map with the basic encoder–decoder structure and loss function $L_{\delta }$;
• AE+OCSVM(se): a two-step training version of the third scenario.

The four schemes are trained in identical epochs to assure the rationality of our experiment. In the two-step approaches, the AAE or AE are trained in 500 epochs separately, and the subsequent parts are trained in additional 500 epochs with the fixed AAE or AE. The co-training approaches train in 500 epochs for the whole network.

(1) 

Adversarial learning method

As discussed in Section 2.2.1, the adversarial training approach is presented in an anomaly weight-map-generated block to highlight the anomalous area in a residual image, with an additional discriminator Dz that helps constrain the hidden feature $\mathit{z}$ of AE to follow the background distribution, which is concerned to obey Gaussian probability. We compared scenario (A) and (C) to validate the effectiveness of the adversarial learning method. Table 7 shows that the average value of ${\mathrm{AUC}}_f$ is improved by 0.321% when adding the adversarial training. The AAE obtains better detection results of up 1.064%, especially for the San Diego dataset with planer targets and complex backgrounds.

Although the ${\mathrm{AUC}}_f$ values do not significantly increase, we noticed by comparing ROC curves in Figure 13 that the red curves are at the upper left of the yellow curves in most datasets, and the detection probability is much higher when the false alarm rate is about 0–${10}^{-2}$. For the Pavia dataset, the AE performed better than AAE in ROC curves; however, we found that the red box of AAE is farther from the green box than that of AE from the box plot comparison in Figure 14. This suggests that the AAE highlighted anomalies better, although it is not obvious from the ${\mathrm{AUC}}_f$ perspective.

In conclusion, our comparison results indicate that additional adversarial training helps yield a better anomaly weight map, which highlights the possible anomalous areas and improves the final detection performance.

(2) 

Joint training method

To better demonstrate the superiority of joint optimizing the feature extractor and subsequent anomaly detector in our proposed method, we present a two-step training version of our proposed network for comparison. As described in Section 2.4, when training the whole SAOCNN, the AAE should be trained separately for the AAE_pre_num epochs at first, and then the subsequent network, including the OCSVM layers, can be updated together with the feature extraction network through $L_{\mathrm{OCSVM}}$. Such joint training drives the feature extractor to learn additional deep details for specifically anomaly detection tasks, which cannot be achieved with the two-step training method.

In our experiment, we used the two-step training version for both scenario (A) and (C). Figure 7 shows that the joint training methods demonstrate superior performance than the related separate training methods. The average ${\mathrm{AUC}}_f$ values are increased by 0.060% and 0.020% for the AAE and AE versions, respectively, when implementing joint training. For our proposed method, the ${\mathrm{AUC}}_f$ values increased by 0.342% and 0.343% in the Coast and San Diego datasets, respectively. The ROC curves in Figure 13 show that the red curves, which represent our proposed method, are located in the upper-left corner in these two datasets. This shows that the joint training approaches present a prime balance between high detection and low false alarm rates. For other datasets, although the ROC curves of the two types are close, the comparison of the box plot in Figure 14 shows that the red boxes of the co-training approaches are farther from the green boxes than that of the two-step training approach, which means the joint training is capable of better highlighting anomalies. In the HYDICE dataset, co-training approaches can also suppress the background better because the green box of the two joint training methods is much closer to 0 in the box plot.

The overall results show that the $L_{\mathrm{OCSVM}}$ joint training approach in our proposed methods help the whole SAOCNN to extract the relevant anomaly features compared with the two-step training approach. In the HADGAN [52] and SAFL [53] methods, the feature extractor and anomaly detector are separated, and the latter makes no contribution to constraining the feature extraction network. This two-step training approach is necessary for these methods because the anomaly detector still adopts the traditional method. However, in our method, a one-class network is applied for anomaly detection and updated together with the whole network through $L_{\mathrm{OCSVM}}$. The joint training method is not only more concise but also enables the anomaly detector to better guide the feature extraction network and make the anomalies more prominent while suppressing the background in the detection maps.

5. Conclusions

In our study, we proposed an unsupervised deep learning model (SAOCNN) for hyperspectral anomaly detection. The advantage of an SAOCNN is that a one-class objective function trains a neural network that maps hyperspectral data to a feature space specially constructed for an anomaly detection task. To capture rich features in hyperspectral images, the improved residual non-local network with an added adversarial autoencoder extracts global information and enhances possible anomalous regions, while the local feature extraction network is sufficient to obtain local spatial information. SAOCNN adopts a training mode to jointly optimize a feature extractor and anomaly detector and accurately separates anomalies from the background according to the hyperplane learned in the feature space. After conducting experiments on six public hyperspectral datasets. It can be concluded that SAOCNN outperforms state-of-the-art methods with higher accuracy and lower false alarm rates, in addition to having faster calculation speeds, whether from a qualitative or quantitative perspective. In the future, we may try new deep learning techniques, such as Transformer, in our proposed unsupervised model to improve its performance.