Feature Learning for SAR Target Recognition with Unknown Classes by Using CVAE-GAN

Abstract

Even though deep learning (DL) has achieved excellent results on some public data sets for synthetic aperture radar (SAR) automatic target recognition(ATR), several problems exist at present. One is the lack of transparency and interpretability for most of the existing DL networks. Another is the neglect of unknown target classes which are often present in practice. To solve the above problems, a deep generation as well as recognition model is derived based on Conditional Variational Auto-encoder (CVAE) and Generative Adversarial Network (GAN). A feature space for SAR-ATR is built based on the proposed CVAE-GAN model. By using the feature space, clear SAR images can be generated with given class labels and observation angles. Besides, the feature of the SAR image is continuous in the feature space and can represent some attributes of the target. Furthermore, it is possible to classify the known classes and reject the unknown target classes by using the feature space. Experiments on the MSTAR data set validate the advantages of the proposed method.

1. Introduction

Automatic target recognition (ATR) is a challenging task for synthetic aperture radar (SAR) [1,2]. In general, SAR-ATR may consist of two stages: first extract the representative features of SAR image and then assign the image to a predetermined set of classes using a classifier. The discriminative features are crucial for SAR-ATR and can significantly influence the success of the latter classifier. In the last few decades, lots of feature-extracted technologies have been proposed in SAR-ATR, including but not limited to the principal component analysis (PCA)-based method [3,4], the scattering center model (SCM)-based method [5,6], sparse representation method [7,8], Krawtchouk moments method [9], and multi-features fusion method [10]. Different from the traditional technologies which extract the image features manually, the deep learning (DL) technology can automatically extract the image features by combining the feature extractor and the classifier, so the remarkable performance on target recognition can be achieved. Several results on some public data sets for SAR-ATR by using DL (deep learning) have been reported and are far beyond the results by using traditional technologies [11,12,13]. However, there are some obvious limitations on the DL-based SAR-ATR studies at present. On one hand, many of the existing DL models are black boxes that do not explain their predictions in a way that humans can understand, so it is hard to ensure the rationality and reliability of the predictive models. On the other hand, most of the existing SAR target recognition methods suppose that the target classes in the test set have appeared in the training set. However, in a realistic scenario, there probably exist some unknown target classes. The traditional SAR-ATR methods will not work or deteriorate seriously for the unknown classes.

In order to cope with the above challenges, some researches have been done. In [14,15], the transfer learning is studied to transfer the knowledge from some SAR data sets with abundant labeled samples to the target task with limited labeled samples. In [16], the recognition model is firstly trained on the simulated SAR data, and then is transferred to the transferred target task. In [17,18], the simulated SAR images are refined to reduce the difference between the simulated image and the real image. Besides the transfer learning methods, the generative networks [19,20] are also used to produce the SAR images that are not contained in the training data. All the above studies are helpful to improve the classification results for the case that only few samples of the SAR target classes are available. However, for the target classes with no one sample, the few-shot learning (FSL) methods are valid. To address this issue, several zero-shot learning (ZSL) methods have been studied in SAR-ATR. In [21,22], the optical images or the simulated SAR images of the unknown class targets are utilized to provide the aided information for zero-shot target recognition. Without adding extra information, a feature space for SAR-ATR is built based on the known class targets in [23]. By mapping the SAR images of the unknown class targets into the feature space, the attribute of the target can be explained in some extent. Moreover, the feature space is continuous and is able to generate the target images that are not in the training data, although the generated images are usually obscure. In [24], a more stable feature space is built which has the better interpretability. The problem in [23,24] is the feature space can only explain the unknown class qualitatively, but can’t distinguish the unknown class from the known classes effectively. Open set recognition (OSR) is a similar task as zero-shot recognition. Besides several known classes, OSR takes all the unknown classes as an extra class. In test, it can classify targets of the known classes and reject targets of the unknown classes. Several studies have been done on OSR for SAR targets, such as the support vector machine (SVM)-based methods [25] and the edge exemplar selection methods [26,27]. These methods don’t exploit the strong ability of feature extraction in DL. Recently, a OSR method for SAR-ATR is proposed based on Generative Adversarial Network (GAN) [28]. A classification branch is added to the discriminator, so the discriminator can provide the target class as well as the score of probability that the target belongs to the known class. However, the recognition accuracy decreases obviously as the number of know classes increases. What is more, the proposed model also lacks transparency and interpretability, like most of existing DL models.

In this paper, we focus on the problem of OSR in SAR-ATR where there is no information about the unknown classes, including the training samples, class number, semantic information, etc. The feature learning method is studied for SAR-ATR, which should classify the known classes, reject the unknown classes and is also explainable in some degree. Inspired by [23,24], a feature space which can well represent the attributes of the SAR target and also has the ability of image generation is expected in the paper. Variational Auto-encoder (VAE) [29,30] and GAN [31,32] are the two most popular deep generative models, which can both generate images by inputting random noise. The noise in VAE is supposed to have some certain distribution and can be seen as the representation of images. VAE is rigorous in theory, but the generated images are often blurry. GAN has no explicit assumption on the noise distribution and can usually produce clear images. However, the input noise in GAN can’t represent the images and the interpretability is not enough. What is more, the traditional VAE and GAN can’t conduct the conditional generation. It is possible to improve the generated image by combining VAE and GAN [33,34]. In the paper, a conditional VAE (CVAE) together with GAN (CVAE-GAN) is proposed for feature learning in SAR-ATR. The latent variable space in VAE, where the noise is produced, is selected as the feature space. Both the target representation and classification are conducted in this feature space. The attribute and class label of the target are used as the conditions to control the image generation. An extra discriminator is used to improve the quality of the generated image. Compared with the existing approaches, the proposed CVAE-GAN model can obtain a feature space for SAR-ATR, which has three main advantages.

• Condition generation for SAR images. By using the encoder network and the generation network in CVAE-GAN jointly, clear SAR images, which are absent in the training data set, can be generated with the given target class label and target attributes, while the network in [23] only produces the obscure images. It can be used in data augmentation and benefit for SAR-ATR with limited labeled data.
• Feature interpretability in some degree. Different from that of auto-encoder, the latent variable of VAE is continuous in the variable space. Therefore, the latent variable obtained via the encoder network can act as the representation of SAR images and reflects some attributes of the target. The network in [28] is incapable of the interpretability.
• Recognition of unknown classes. Because of the additive angular margin in the feature space, the intra-class distance of the known classes is decreased and the inter-class distance is increased. As a result, it is possible to classify the known classes and recognize the unknown target classes whose features are outside the angular range of the known classes. The networks in [23,24] can’t recognize the unknown classes.

The rest of this paper is organized as follows. In Section 2, the proposed CVAE-GAN is introduced in detail, including the model derivation, loss function and network structure. In Section 3, experimental results are shown to demonstrate the effectiveness of the proposal. In Section 4, the results are further discussed compared with the previous studies. The work is concluded in Section 5.

2. CVAE-GAN for SAR-ATR with Unknown Target Classes

The main objective in this paper is to find an explicable feature space, in which the attributes of the SAR images can be well represented and the unknown classes can be distinguished. The generative model is a good choice to achieve the interpretability. We select VAE as the generative model, because it not only has the generation ability, but also can obtain the feature representation of images. Considering only random images can be generated in the original VAE, in this section, we will firstly derive a conditional VAE model by embedding the target attributes. And then integrate the classification loss and GAN loss into the CVAE model and achieve the CVAE-GAN model. The resultant model encourages the abilities of conditional image generation, feature interpretability and target classification simultaneously. A deep network will be built based on the proposed model in the last.

2.1. CVAE-GAN Model

We expect to generate the corresponding target SAR image with a given class label and some attribute parameters such as the observation angle, resolution, frequency and polarization mode. Let $x,y,\theta$ denote the target image, the class label and the related attribute parameters, respectively. The generation model can be realized by maximizing the conditional log-likelihood function

$$\max {{\rm E}}_{x~p\left(x\right)}\log p\left(x|y,\theta \right)$$

(1)

where $p\left(x\right),p\left(x|y,\theta \right)$ are the prior distribution and posterior distribution of $x$. According to the idea of VAE, an auxiliary latent variable $z$ is added to represent $\log p\left(x|y,\theta \right)$. Let $q\left(z|x\right)$ be the posterior distribution of $z$, and $\log p\left(x|y,\theta \right)$ can be expressed as

$$\begin{array}{cc}\hfill \log p\left(x|y,\theta \right)& {\displaystyle ={\int }_{z}q\left(z|x\right)}dz\times \log p\left(x|y,\theta \right)\hfill \\ & ={\displaystyle {\int }_{z}q\left(z|x\right)}\log p\left(x|y,\theta \right)dz\hfill \\ & ={\displaystyle {\int }_{z}q\left(z|x\right)}\log \frac{p\left(x,z|y,\theta \right)}{q\left(z|x,y,\theta \right)}dz\hfill \\ & ={\displaystyle {\int }_{z}q\left(z|x\right)}\log \frac{p\left(x,z|y,\theta \right)}{q\left(z|x\right)}\frac{q\left(z|x\right)}{q\left(z|x,y,\theta \right)}dz\hfill \\ & ={\displaystyle {\int }_{z}q\left(z|x\right)}\log \frac{p\left(x,z|y,\theta \right)}{q\left(z|x\right)}dz+{\displaystyle {\int }_{z}q\left(z|x\right)}\log \frac{q\left(z|x\right)}{q\left(z|x,y,\theta \right)}dz\hfill \\ & ={\displaystyle {\int }_{z}q\left(z|x\right)}\log \frac{p\left(x,z|y,\theta \right)}{q\left(z|x\right)}dz+KL\left[q\left(z|x\right)\Vert q\left(z|x,y,\theta \right)\right]\hfill \end{array}$$

(2)

where $KL\left[q\left(z|x\right)q\left(z|y,\theta \right)\right]$ refers to the (Kullback–Leibler, KL)-divergence between the distributions $q\left(z|x\right)$ and $q\left(z|y,\theta \right)$. Because of the non-negativity of KL-divergence, we can get the variational lower bound of $\log p\left(x|y,\theta \right)$ which is written as

$$\log p\left(x|y,\theta \right)\ge {{\displaystyle \int }}_{z}q\left(z|x\right)\log \frac{p\left(x,z|y,\theta \right)}{q\left(z|x\right)}dz$$

(3)

Therefore, the maximization of $\log p\left(x|y,\theta \right)$ can be converted to maximize its variational lower bound, which is further derived as

$$\begin{array}{cc}\hfill {\int }_{z}q\left(z|x\right)\log \frac{p\left(x,z|y,\theta \right)}{q\left(z|x\right)}dz& ={\int }_{z}q\left(z|x\right)\log \frac{p\left(x|y,\theta ,z\right)p\left(z|y,\theta \right)}{q\left(z|x\right)}dz\hfill \\ & ={\int }_{z}q\left(z|x\right)\log \frac{p\left(z|y,\theta \right)}{q\left(z|x\right)}dz+{\int }_{z}q\left(z|x\right)\log p\left(x|y,\theta ,z\right)dz\hfill \\ & =-KL\left[q\left(z|x\right)\Vert p\left(z|y,\theta \right)\right]+{\int }_{z}q\left(z|x\right)\log p\left(x|y,\theta ,z\right)dz\hfill \end{array}$$

(4)

Consider the latent variable $z$ can be generated with $y,\theta$ by $p\left(z|y,\theta \right)$, so the term $\log p\left(x|y,\theta ,z\right)$ in (4) is simplified as $\log p\left(x|z\right)$. That means the latent variable $z$ contains the information of target class label and target attribute, so it can be used to generate the corresponding target image. As a result, the generation problem in (1) can be re-expressed as

$$\underset{\begin{array}{c}\begin{array}{c}p\left(z|y,\theta \right),\\ q\left(z|x\right),\\ p\left(x|z\right)\end{array}\end{array}}{\max } -KL\left[q\left(z|x\right)\Vert p\left(z|y,\theta \right)\right]+{{\rm E}}_{z~q\left(z|x\right)}\log p\left(x|z\right)$$

(5)

In Equation (5), the expectation sign ${{\rm E}}_{x~p\left(x\right)}$ is omitted for simplicity. The first term in Equation (5) aims to minimize the difference between the distribution $q\left(z|x\right)$ and $p\left(z|y,\theta \right)$. As a result, the latent variable $z$ produced from $x$ will have the information of $y,\theta$. This makes it possible for conditional image generation and the latter target classification compared with the original VAE. The purpose of the second term in Equation (5) is to maximize the probability of generating the image $x$ from the corresponding $z$ which is produced by $q\left(z|x\right)$. The two losses above are called as the distribution loss and the reconstruction loss, respectively, in the following.

Since the latent variable $z$ contains the target class information, it can be used as the recognition feature for target classification. Therefore, a classification loss ${{\rm E}}_{z~q\left(z|x\right)}\log p\left(y|z\right)$ is added in Equation (5) and the problem is further expressed as

$$\underset{\begin{array}{c}\begin{array}{c}p\left(z|y,\theta \right),\\ q\left(z|x\right),\\ p\left(x|z\right),\\ p\left(y|z\right)\end{array}\end{array}}{\max } -KL\left[q\left(z|x\right)\Vert p\left(z|y,\theta \right)\right]+{{\rm E}}_{z~q\left(z|x\right)}\log p\left(x|z\right)+{{\rm E}}_{z~q\left(z|x\right)}\log p\left(y|z\right)$$

(6)

Considering the generated images via VAE are often blurry, GAN is combined with CVAE in this paper to improve the image quality. Let $D(\cdot )$ be an extra discriminator, and then the problem of the generation model is expressed as

$$\underset{\begin{array}{c}\begin{array}{c}p\left(z|y,\theta \right),\\ q\left(z|x\right),\\ p\left(x|z\right),\\ p\left(y|z\right)\end{array}\end{array}}{\max } -KL\left[q\left(z|x\right)\Vert p\left(z|y,\theta \right)\right]+{{\rm E}}_{z~q\left(z|x\right)}\left[\log p\left(x|z\right)\right]+{{\rm E}}_{z~q\left(z|x\right)}\left[\log p\left(y|z\right)\right]+{{\rm E}}_{x~p\left(x|z\right)}\left[\log D\left(x\right)\right]$$

(7)

The adversarial loss for the discriminator is expressed as

$$\underset{D}{\max } L_D={{\rm E}}_{x~p\left(x\right)}\left[\log D\left(x\right)\right]+{{\rm E}}_{x~p\left(x|z\right)}\left[\log \left(1-D\left(x\right)\right)\right]$$

(8)

Equations (7) and (8) are the final objective functions for the CVAE-GAN model, which will be solved alternately in training.

2.2. Realization of Loss Functions

The concrete form of the loss function in Equation (7) is derived in the following.

2.2.1. Distribution Loss

We suppose $q\left(z|x\right)$ and $p\left(z|y,\theta \right)$ in the first term of Equation (7) have the normal distribution $q\left(z|x\right)=N\left({\mu }_1\left(x\right),{\sigma }_1\left(x\right)\right)$ and $p\left(z|y,\theta \right)=N\left({\mu }_2\left(y,\theta \right),{\sigma }_2\left(y,\theta \right)\right)$, respectively. The mean value ${\mu }_1\left(x\right)$ and standard deviation ${\sigma }_1\left(x\right)$ are the functions of $x$, while ${\mu }_2\left(y,\theta \right),{\sigma }_2\left(y,\theta \right)$ are the functions of $y,\theta$. Different from the standard normal distribution used for all samples in the original VAE, the distributions for different samples can be various based on the target attributes. Therefore, it is possible to distinguish the samples with different target attributes in the feature space. Mark ${\mu }_1\left(x\right),{\sigma }_1\left(x\right),{\mu }_2\left(y,\theta \right),{\sigma }_2\left(y,\theta \right)$ as ${\mu }_1,{\sigma }_1,{\mu }_2,{\sigma }_2$, respectively, for simply, and the distribution loss in (7) can be expressed as

$$\begin{array}{cc}\hfill L_1& =-KL\left[N\left({\mu }_1,{\sigma }_1\right)\Vert N\left({\mu }_2,{\sigma }_2\right)\right]\hfill \\ & =-{\displaystyle {\int }_x\frac{1}{\sqrt{2\pi {\sigma }_1^2}}{e}^{-\frac{{\left(x-{\mu }_1\right)}^2}{2{\sigma }_1^2}}}\log \frac{e^{-{\left(x-{\mu }_1\right)}^2/2{\sigma }_1^2}/\sqrt{2\pi {\sigma }_1^2}}{e^{-{\left(x-{\mu }_2\right)}^2/2{\sigma }_2^2}/\sqrt{2\pi {\sigma }_2^2}}dx\hfill \\ & =-{\displaystyle {\int }_x\frac{1}{\sqrt{2\pi {\sigma }_1^2}}{e}^{-\frac{{\left(x-{\mu }_1\right)}^2}{2{\sigma }_1^2}}}\left[\log \frac{{\sigma }_2}{{\sigma }_1}+{\left(x-{\mu }_2\right)}^2/2{\sigma }_2^2-{\left(x-{\mu }_1\right)}^2/2{\sigma }_1^2\right]dx\hfill \\ & =-\log \frac{{\sigma }_2}{{\sigma }_1}-\frac{{\sigma }_1^2+{\left({\mu }_1-{\mu }_2\right)}^2}{2{\sigma }_2^2}+0.5\hfill \end{array}$$

(9)

If both ${\sigma }_1$ and ${\sigma }_2$ are alterable, the possible results of ${\sigma }_1$ and ${\sigma }_2$ may be both close to 0 in order to maximize $\begin{array}{c}{L}_1\end{array}$. In this case, the randomness which is expected in VAE will have disappeared. To avoid this problem, we fix ${\sigma }_2$ and let ${\sigma }_2=1$ in this paper. And then (9) is revised as

$$\begin{array}{c}{L}_1=\end{array}\log {\sigma }_1-\frac{{\sigma }_1^2+{\left({\mu }_1-{\mu }_2\right)}^2}{2}+0.5$$

(10)

As the latent variable is multidimensional, the final loss is the sum of $\begin{array}{c}{L}_1\end{array}$ in all the dimensions.

2.2.2. Reconstruction Loss

Suppose $q\left(x|z\right)$ in the second loss of Equation (7) has the normal distribution $N\left({\mu }_3\left(z\right),{\sigma }_z\right)$, where ${\mu }_3\left(z\right)$ is a function of $z$ and ${\sigma }_z$ is a constant. Thus, the reconstruction loss is expressed as

$$\begin{array}{cc}\hfill L_2& =\log p\left(x|z\right)\hfill \\ & =\log \frac{1}{\sqrt{2\pi {\sigma }_z^2}}{e}^{-{\left(x-{\mu }_3\left(z\right)\right)}^2/{\sigma }_z^2}\hfill \\ & =-{\left(x-{\mu }_3\left(z\right)\right)}^2/2{\sigma }_z^2-\log \sqrt{2\pi {\sigma }_z^2}\hfill \end{array}$$

(11)

It can be seen from Equation (11) that, the maximization of $L_2$ is equal to the minimization of $x-{\mu }_3{\left(z\right)}^2$.

2.2.3. Classification Loss

The cross-entropy loss (CEL) is the commonly used classification loss. For the $i$th target sample and the corresponding feature $z_i$, whose class label is $y_i$, the CEL (maximization) is expressed as

$$CEL=\log \frac{e^{f_{y_i}}}{{{\displaystyle \sum }}_{j=1}^C{e}^{f_j}}$$

(12)

where $C$ is the number of the known target classes, $f_j=W_j^T{z}_i+b_j$ is the score that $z_i$ belongs to the $j$th class, $W_j,b_j$ are the weight and bias in the classifier corresponding to the $j$th class. However, just as stated in the introduction, there may exist some unknown classes in practice and the traditional CEL can’t distinguish them from the known classes. In order to recognize the unknown classes, an idea is to incorporate the margin in the softmax loss functions, which can push the classification boundary closer to the weight vectors and add the inter-class separability. It is a popular line of research for face recognition in recent years [35,36,37,38,39,40]. In this paper, we apply the ArcFace [40] to add the angular margin for the known classes in the feature space. Let $\left|W_j\right|=1,b_j=0$ and $\left|z_i\right|=s$, $s$ is a predetermined constant. And then the classification loss is expressed as

$$L_3=\log \frac{e^{f_{y_i}}}{e^{f_{y_i}}+{{\displaystyle \sum }}_{j=1,j\ne y_i}^C{e}^{f_j}}$$

(13)

where $f_j=s\cos \left({\varphi }_j\right)$, $f_{y_i}=s\cos \left({\varphi }_{y_i}+m\right)$, ${\varphi }_j$ and ${\varphi }_{y_i}$ are the angles between $z_i$ and $W_j$,$W_{y_i}$, respectively, $m$ is the additive angular margin. In Equation (13), the different classes can be separated in angle. For the simplest binary classification, the classification boundaries for Class 1 and Class 2 are $s\cos \left({\varphi }_1+m\right)=s\cos \left({\varphi }_2\right)$ and $s\cos \left({\varphi }_1\right)=s\cos \left({\varphi }_2+m\right)$, respectively. The schematic diagram is shown in Figure 1. For $m=0$, the two classification boundaries coincide with each other and the unknown classes can’t be identified, while $m>0$ the two classification boundaries are separated and the unknown classes are possible to be distinguished from the known.

The fourth term in Equation (7) is realized by using the traditional GAN loss and more details can be found in [31].

2.3. Network Architecture

Based on the above model, the overall network of the proposed CVAE-GAN is shown in Figure 2a. It is composed of 5 sub-networks, i.e., the encoder networks $E_1$ and $E_2$, the generation network $G$, the classification network $C$ and the discrimination network $D$. $E_1$,$E_2$,$G$,$C$ are expected to fit the distributions $q\left(z|x\right),p\left(z|y,\theta \right),p\left(x|z\right)$ and $p\left(y|z\right)$ in Equation (7), respectively. More specifically, $E_1$ is used to realize the functions of ${\mu }_1\left(x\right),{\sigma }_1\left(x\right)$ in $\begin{array}{c}{L}_1\end{array}$, $E_2$ is used to realize the function ${\mu }_2\left(y,\theta \right)$ in $\begin{array}{c}{L}_1\end{array}$, and $G$ is used to realize the function ${\mu }_3\left(z\right)$ in $\begin{array}{c}{L}_2\end{array}$. The function of condition generation is realized by jointly utilizing the encoder network $E_2$ and the generation network $G$. By projecting the images into the feature space with $E_1$ and generating the images from the features with $G$, the attributes of the target can be explained in some extent. Besides, by using $E_1$ and $C$, we can classify the known classes and recognize the unknown classes.

The particular architectures of the sub-networks used in the experiment section are shown in Figure 2b. $E_1$,$G$ and $D$ are all modified from the popular U-Net [41] which is widely used in natural image processing. $E_2$ is constructed by using the multi-layer perceptron. In the figure, Conv denotes a convolution layer, while ConvT means a transposed convolution layer. Conv(c_in, c_out, k, s, p) indicates the layer who has c_in input channels, c_out output channels and a k × k sized convolution filter with stride = s and padding = p. Linear(a_in, a_out) indicates a full-connection layer with input size a_in and output size a_out. BN and DP are the abbreviations of BatchNorm and Dropout. $C$ is not shown in Figure 2b, which is simply constructed according to (13). The total number of parameters in CVAE-GAN is about 33.46 million.

In the network training, a reparameterization trick [29] is used to sample $z$. It aims to make the sampling step derivable, which is expressed as

$$z={\mu }_1+{\sigma }_1\times \epsilon$$

(14)

where $\epsilon$ is the noise with the standard normal distribution $\epsilon \sim N\left(0,1\right)$, ${\mu }_1=\mu \left(x\right),{\sigma }_1=\sigma \left(x\right)$. Thus, the sampled $z$ has the normal distribution $q\left(z|x\right)=N\left(\mu \left(x\right),\sigma \left(x\right)\right)$. The second term in Equation (7) can be expressed in the summation form with the samples of $z$ as

$${{\rm E}}_{z~q\left(z|x\right)}\left[\log p\left(x|z\right)\right]={\displaystyle \sum _{k=1}^K\log p\left(x|z_k\right)}, z_k~q\left(z|x\right)$$

(15)

Since the network is usually trained with mini-batch, the sample number for $z$ in Equation (15) can be set as $K=1$ for each iteration. Similarly, one sample of $z$ is also used for ${{\rm E}}_{z~q\left(z|x\right)}\left[\log p\left(y|z\right)\right]$ and ${{\rm E}}_{x~p\left(x|z\right)}\left[\log D\left(x\right)\right]$ in Equation (7).

3. Experiments

In this section, the commonly used public MSTAR data set is applied to test the proposed feature learning method. There are 10 types of vehicle targets in the data set, including 2S1 (Self-Propelled Howitzer), BDRM2 (Armored Reconnaissance), BTR60 (Armored Personnel Carrier), D7 (Bulldozer), T62 (Tank), ZIL131 (Military Truck), ZSU234 (Air Defense Unit), T72 (Main Battle Tank), BMP2(Infantry Fighting Vehicle) and BTR70(Armored Personnel Carrier), which are numbered from Class 0 to Class 9 in order. The former 7 classes are selected as the known target classes and the latter 3 classes are the unknown classes. The target images of the 7 known classes with depression angle of ${15}^{\circ }$ are used for training, while the images of all the 10 classes with depression angle of ${17}^{\circ }$ are used for testing. In the training, inputs of the network contain the SAR image, the azimuth angle and the corresponding class label. All the SAR images are cropped to the size of 128 × 128. The azimuth angle information is represented as a 2-D vector ${\left[\cos \varphi ,\sin \varphi \right]}^T$, where $\varphi \in 0^{\circ },{360}^{\circ })$ denotes the azimuth angle. The class labels are encoded into the 7-D one-hot vectors. The dimension of the latent variable is set as 512. The constants in the classifier are set as $s=64,m=1$ in the experiment.

In order to make the training of CVAE-GAN easier, a stepwise training strategy is applied. In the first, the loss $\left(L_1+L_2\right)$ is utilized to train the CVAE networks $E_1$, $E_2$ and $G$. Next, the classification network $C$ together with the CVAE networks will be trained by using the loss $\left(L_1+L_2+L_3\right)$. In the last, the loss $L_1+L_2+L_3+L_4$ and $L_D$ will be trained adversarially to improve the clarity of generated images. The network is initialized with the default parameters in Pytorch and is trained by using the Adam optimization with a learning rate of 0.0001. All the experiments are performed on a workstation with Intel Core i9-9900 CPU and NVIDIA GeForce RTX 2080 Ti GPU.

3.1. Visualization of the Learned Feature Space

The latent variables, i.e., the extracted features, are high-dimensional of 512, which are difficult to visualize. Therefore, the t-distributed stochastic neighbor embedding (t-SNE) [42], a widely used dimension reduction technique, is utilized to embed the high-dimensional features into a 2-D feature space. Since only the angle of the feature is used for classification in Equation (13), we normalize all the features before visualization. Figure 3 visualizes the extracted features of the 7 known classes with depression angle of ${15}^{\circ }$. Colors for Class 0 ~ Class 6 are ‘blue’, ‘orange’, ‘green’, ‘red’, ‘purple’, ‘brown’ and ‘pink’, respectively. It can be seen that different classes can be separated from each other easily in this feature space. The azimuth angles of the features in Figure 3 are marked in Figure 4. It shows that the feature of SAR image moves continuously in the space along with the gradual change of azimuth angle. Thus, the extracted feature is useful to explain the angle attribute of the SAR image.

Furthermore, the features of the 7 known classes in the testing data with depression angle of ${17}^{\circ }$ are also examined. All these extracted features are marked in gray and shown in Figure 5. It can be seen the features of the testing data have the similar distribution as that of the training data on the whole. The features of the 3 unknown classes together with the known classes are shown in Figure 6, where the unknown Class 7, 8 and 9 are marked in ‘olive’, ‘cyan’ and ‘gray’, respectively. We can see most samples of the unknown classes are separated from that of the known classes in the feature space, which makes it possible to distinguish the unknown from the known. We can also see that Class 7 (T72) is close to Class 4 (T62) in the feature space. It indicates that there may be high similarity between the two classes. It is consistent with the physical truth that T72 tank approximates to T62 tank in shape and structure. The analogous conclusion can be made for Class 9 (BTR70) and Class 2 (BTR60). Therefore, the feature space is useful to explain the target attribute of SAR image for the unknown classes.

3.2. Results of Image Generation

In this part, the image generation ability of the proposed model is tested. Figure 7 shows the generated images with given conditions. Figure 7a is the result of one random sample. $x$ is a real SAR image randomly selected from the training data set. ${\mu }_3\left({\mu }_1\left(x\right)\right)$ denotes the generated image of $x$ after going through encoding $z={\mu }_1\left(x\right)$ and decoding ${\mu }_3\left(z\right)$ successively. Take the class label $y$ and the azimuth angle $\theta$ of the real SAR image $x$, and conduct encoding $z={\mu }_2\left(y,\theta \right)$ and decoding ${\mu }_3\left(z\right)$, we can obtain the conditional generation image ${\mu }_3\left({\mu }_2\left(y,\theta \right)\right)$. It can be seen both the encoding-decoding image and the conditional generation image are very similar to the original SAR image. The additive noise is also considered for the conditional generation. The feature now is $z={\mu }_2\left(y,\theta \right)+\epsilon$, where $\epsilon$ is the uniformly distributed noise within the range of $\left[-U_{\epsilon },U_{\epsilon }\right]$. The noisy images with $U_{\epsilon }=1,3,5$ are denoted as ${\mu }_3\left({\mu }_2\left(y,\theta \right)+{\epsilon }_1\right)$, ${\mu }_3\left({\mu }_2\left(y,\theta \right)+{\epsilon }_2\right)$ and ${\mu }_3\left({\mu }_2\left(y,\theta \right)+{\epsilon }_3\right)$ in Figure 7a. We can see the randomly generated images are still similar to the original images although some slight differences exist. Some other random samples are shown in Figure 7b, which are consistent with the above result. The normalized features of the noisy images (marked in gray) are visualized in Figure 8. It demonstrates the generation ability of the continuous feature space.

Images with the azimuth angles that are absent in the training data are generated in the following. The obvious angle gaps for Class 0~6 in the training data are (83.3, 93.3), (275.5, 289.5), (200.3, 210.3), (84.5, 94.5), (83.1, 93.1) and (83.0, 93.0), respectively. Sample the angle with equal interval in the gaps and generate the SAR images with the sampled angles and class labels. The results are shown in Figure 9. Figure 9a shows the result of Class 0. $x\left({\theta }_1\right)$ and $x\left({\theta }_2\right)$ denote the real images with the angle ${\theta }_1,{\theta }_2$, ${\mu }_3\left({\mu }_2\left(y,{\theta }_1\right)\right)$ and ${\mu }_3\left({\mu }_2\left(y,{\theta }_2\right)\right)$ denote the generated images with the present angle ${\theta }_1,{\theta }_2$, ${\mu }_3\left({\mu }_2\left(y,{\theta }^{\prime }\right)\right)$ denotes the generated image with the absent angle ${\theta }^{\prime }$ and ${\theta }^{\prime }\in \left\{{\theta }_1^{\prime },{\theta }_2^{\prime },{\theta }_3^{\prime },{\theta }_4^{\prime }\right\}$ where ${\theta }_1^{\prime },{\theta }_2^{\prime },{\theta }_3^{\prime },{\theta }_4^{\prime }$ are the interpolated absent angles between ${\theta }_1$ and ${\theta }_2$. The results of Class 1~6 are shown Figure 9b and the visualized features of the generated images (marked in gray) are shown in Figure 10. It can be seen the generated images are changed smoothly and can well fill the angle gaps in the feature space.

Further, the weight vectors of the classifier for each class are also visualized in the feature space. Figure 11 shows the result, where the weight vectors are marked as gray ‘9’. We can see the weight vectors are close to features of the samples in the corresponding classes. In the classifier, the images whose features are identical to the weight vectors will obtain the highest classification score for the corresponding class. Next, we will show the generated images by using the weight vectors as features. The feature amplitude may influence the result of the generated image. The mean amplitude of the original weight vectors is about 198 and the mean amplitude of features in the training data is about 70. Herein, we set the weight vectors with different amplitudes of (198, 127, 81, 52). The generated images for Class 0, as an example, are shown individually in Figure 12a for clarity. The others for Class 1~6 are shown in Figure 12b. The results may reflect the most recognizable image feature for each class. For example, the generated images for Class 2 all have strong clutter background. It reflects the fact that, in some SAR images for Class 2, the clutter is strong, while for the other classes this characteristic is not obvious. Therefore, for a SAR image who has the strong clutter background, the classifier will classify it into Class 2 with high probability. According to the generated images sampled in the feature space, the actions of the classifier can be understood in some degree.

3.3. Results of SAR Target Recognition with Unknown Classes

In this section, we will test the classification ability of the proposed model on unknown classes. The SAR images with depression angle of ${17}^{\circ }$ are used for testing. There are 7 known classes with 2049 samples and 3 unknown classes with 698 samples in the test data set. As illustrated in Figure 1, the target which is located inside the classification boundary will be classified to the corresponding known class (Class 0 ~ Class 6), while the target outside all the classification boundaries of the known classes will be classified to the unknown class. The rate of the samples correctly classified in the known classes is denoted as the true positive ratio (TPR), and the rate of the samples that are falsely classified in the unknown classes is denoted as the false positive ratio (FPR). The classification boundary is adjustable in the test by using a classification threshold $m_{\mathrm{th}}$. The recognition results with different $m_{\mathrm{th}}$ are shown in

Table 1. Recognition results with different classification thresholds. TPR denotes the true positive ratio and FPR denotes the false positive ratio.

$\mathbf{Threshold}{\mathit{m}}_{\mathbf{th}}$	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1
TPR	0.9956	0.9893	0.9810	0.9702	0.9492	0.9097	0.8414	0.7252
FPR	0.7063	0.4542	0.3281	0.2264	0.1519	0.0888	0.0501	0.0143
accuracy	0.8173	0.8766	0.9024	0.9203	0.9236	0.9101	0.8689	0.7914

. It can be found, with the increase of $m_{\mathrm{th}}$, both TPR and FPR are reduced. The overall accuracy achieves the best result 0.9236 when $m_{\mathrm{th}}=0.7$.

To further illustrate the result, the confusion matrix with $m_{\mathrm{th}}=0.7$ is shown in Figure 13. It can be seen there are no falsely classified samples among the 7 know classes, but only some falsely classified samples exist between the known and unknown classes. Based on the confusion matrix, the average of precision, recall and F1_score on the 8 classes can be calculated by Equation (16), where $\stackrel{-}{C}$ is the total number of classes. The precision, recall and F1_score with $m_{\mathrm{th}}=0.7$ are 0.9410, 0.9359 and 0.9369, respectively.

$$\begin{array}{l}\mathrm{precision}=\frac{1}{\overline{C}}{\displaystyle {\sum }_{i=0}^{\overline{C}-1}{\mathrm{precision}}_i}, {\mathrm{precision}}_i=\frac{{\mathrm{TP}}_i}{{\mathrm{TP}}_i{+\mathrm{FP}}_i}\\ \mathrm{recall}=\frac{1}{\overline{C}}{\displaystyle {\sum }_{i=0}^{\overline{C}-1}{\mathrm{recall}}_i}, {\mathrm{recall}}_i=\frac{{\mathrm{TP}}_i}{{\mathrm{TP}}_i{+\mathrm{FN}}_i}\\ \mathrm{F}1\_\mathrm{score}=\frac{1}{\overline{C}}{\displaystyle {\sum }_{i=0}^{\overline{C}-1}{\mathrm{F}1\_\mathrm{score}}_i}, {\mathrm{F}1\_\mathrm{score}}_i=\frac{2\times {\mathrm{precision}}_i\times {\mathrm{recall}}_i}{{\mathrm{precision}}_i+{\mathrm{recall}}_i}\end{array}$$

(16)

In the above, the additive angular margin is set as $m=1$. The recognition results with different $m$ are also tested, which are shown in

Table 2. Recognition results with different additive angular margins.

Angular Margin $m$	0.5 ($m_{\mathrm{t}\mathrm{h}}$ = 0.4)	0.8 ($m_{\mathrm{t}\mathrm{h}}$ = 0.6)	1 ($m_{\mathrm{t}\mathrm{h}}$ = 0.7)	1.2 ($m_{\mathrm{t}\mathrm{h}}$ = 0.8)	1.4 ($m_{\mathrm{t}\mathrm{h}}$ = 0.9)
TPR	0.9370	0.9390	0.9492	0.9531	0.9590
FPR	0.1991	0.1447	0.1519	0.1461	0.1304
accuracy	0.9024	0.9177	0.9236	0.9279	0.9363
precision	0.9238	0.9382	0.9410	0.9444	0.9505
recall	0.9194	0.9278	0.9359	0.9400	0.9469
F1_score	0.9194	0.9316	0.9369	0.9401	0.9473

. We can see all of the accuracy, precision, recall and F1_score are improved with the increase of $m$. For $m=1.4$ and $m_{\mathrm{th}}=0.9$, the overall accuracy is up to 0.9363. It should be also noted that the network will be hard to convergence as $m$ is too large.

4. Discussion

In this section, we will discuss the above results compared with the existing studies.

With regard to the explainable feature space for unknown SAR target recognition, the related studies have been done in [23,24]. The feature spaces built in these papers only represent the categories of the target, but can’t show the other target attributes, such as the azimuth angle, effectively. It can be seen clearly from the Figures 9 and 11 in [23] and Figures 6 and 7 in [24]. In this paper, the target attributes including the azimuth angle can be explained in the continuous feature space, which is useful to understand the result of deep network in some extent and benefits for the conditional image generation.

About the image generation, the network proposed in [23] is able to generate the SAR image with the given class and angle. However, the generated images are blurry as shown in Figures 5 and 7 of [23]. Besides, some variations of GAN have been proposed to generate the images with given class labels [43,44]. Herein, we also test this kind of methods on SAR image generation. Figure 14 shows the generated SAR images with class labels by using the conditional GAN in [44]. It can be seen the generated images are clear. However, as well as we know, the conditional GANs only deal with the concrete conditions well, such as the class label, but don’t work for the continuous conditions at present, such as the azimuth angle. Moreover, the conditional VAE, which is without the discriminator compared with the proposed CVAE-GAN, is also tested. The result is shown in Figure 15. We can see it is able to deal with the continuous angle, but the generated images are not as clear as the conditional GANs. The proposed CVAE-GAN performs well on both the image quality and the conditional generation, as shown in Figure 7, Figure 9 and Figure 12.

With regard to the SAR target recognition with unknown classes, the open set recognition method proposed in [28] is also tested. The test condition is the same as that in Section 3.3. The network is built according the Figure 1 in [28] and the input images are cropped into the size of 64 × 64 as [28] did. The network is trained in 300 epochs and all the networks with different epochs are used for the OSR test. The best result is obtained by using the network in epoch 275 and with the threshold 0.1. The confusion matrix is shown in Figure 16 and the recognition metrics are shown in

Table 3. Recognition results with the method in [28].

TPR	FPR	Accuracy	Precision	Recall	F1_score
0.9546	0.2479	0.9031	0.9157	0.9292	0.9190

. By comparing with the results in Section 3.3, we can see the proposed method has the better performance for OSR.

5. Conclusions

In this paper, a deep generation and recognition network is proposed for SAR-ATR based on CVAE-GAN model. The proposed network can form a feature space, in which both the conditional generation task and the open-set recognition task can be realized. More importantly, some actions of the deep network can be explained in the feature space. In the next work, we will pay attention to the explainable feature space which can show the confidence coefficient for classification. Moreover, to classify the unknown classes with the help of some aided target information is also a meaningful job in the further. The unbalanced data is a critical issue for the classification of unknown classes.