Research on the Enhancement Method of Specific Emitter Open Set Recognition

Abstract

Open set recognition (OSR) aims at dealing with unknown classes that are not included in the train set. However, existing OSR methods rely on deep learning networks that perform supervised learning on known classes in the train set, resulting in poor performance when the unknown class is very similar to the known class. Considering the subtle individual differences under the same type in specific emitter identification (SEI) applications, it is difficult to distinguish between known classes and unknown classes in open set scenarios. This paper proposes a pseudo signal generation and recognition neural network (PSGRNN) to address relevant problems in this situation. PSGRNN applies complex-value convolution operations to accommodate IQ signal inputs. Its key idea is to utilize samples of known classes to generate pseudo samples of unknown classes. Then, the samples of known classes and the generated pseudo samples of unknown classes are jointly input into the neural network to construct a new classification task for training. Moreover, the center loss is improved by adding inter-class penalties to maximize the inter-class difference. This helps to learn useful information for separating known and unknown classes, resulting in clearer decision boundaries between the known and the unknown. Extensive experiments on various benchmark signal datasets indicate that the proposed method achieves more accurate and robust open set classification results, with an average accuracy improvement of 4.62%.

1. Introduction

Specific emitter identification (SEI) can obtain subtle features of different emitters in radio frequency (RF) signals based on the inherent differences in hardware. It achieves individual identification under the same type of target cluster conditions. These subtle features are called the RF fingerprint, which plays an important role in fields such as electronic reconnaissance and physical-layer security [1]. With the development of deep learning (DL), neural networks have achieved better performance in signal recognition problems compared to traditional methods [2,3,4,5]. However, most existing DL methods for SEI follow the closed set assumption, which means that only known classes will appear in the testing environment. Due to the rapid development of wireless technology, emitters like internet of things (IoT) devices are experiencing explosive access demand [6]. The environment is filled with a large number of unknown wireless signals. It is impractical to obtain signal samples from all types of emitters. Test samples may come from not only those known classes observed during the training but also unseen classes. When an unknown class appears in the test set, a DL method that follows the closed set assumption can only forcibly classify it into a known class, which goes against the real scenario. Open set assumes that the test set contains samples from unknown classes which have not appeared in the train set. The open set recognition (OSR) task requires identifying known classes and rejecting unknown classes. Therefore, OSR is a more realistic and challenging task.

An OSR model is a way to solve the above problems. Most of the existing OSR models, in essence, learn effective feature representation (or activation vector) by training known classes, and the trained models are used to extract features of unknown classes. Then, the models seek decision boundaries through feature differences to distinguish unknown classes from known classes [7]. However, since the unknown class has never appeared during the training phase, it is extremely difficult to learn the distinguishing features between known and unknown classes. If the unknown class is very similar to the known class, its features extracted by the model will be extremely difficult to distinguish. This leads to the inability of obtaining a clear boundary between the known and unknown classes. In SEI, the recognition object belongs to different individuals from the same type of emitters, thus the samples of unknown class may have a high similarity with the known class. This makes the OSR task of SEI much more difficult compared to other OSR tasks.

Based on the above analysis, we propose an OSR model for SEI called pseudo signal generation and recognition neural network (PSGRNN). Inspired by generative adversarial networks (GANs) [8], the PSGRNN utilizes the known class information to generate pseudo samples of unknown class. These samples are similar to the known classes but not belonging to them. Aiming to learn the features for predicting the unknown class’s distribution, the samples from the known class and pseudo unknown class are jointly input into the complex-value convolutional neural network (CvCNN) to construct a new classification task. In addition, we improve the center loss [9] and use it in training. In this way, the features of known and unknown classes can be better separated, providing a clearer distinction. The classifier will model the extracted features based on extreme value theory (EVT) [10], then identify known classes and reject unknown classes according to the distribution probability. In summary, the main contributions of this paper are as follows:

• We propose a novel OSR model for the SEI task called PSGRNN. It utilizes generated pseudo samples of unknown classes to assist in training neural networks for feature learning. We modified the generator loss in traditional GANs based on the characteristics of the pseudo unknown classes. By inverting the part that deceives the discriminator and applying the mean square error (MSE) loss, the generated pseudo samples of unknown classes are similar to known classes but can easily be recognized as new classes.
• In CvCNN, the complex-value convolution operation is realized through linear transformation to adapt to IQ signal inputs. In addition, we add inter-class penalties for center loss to optimize the distribution of known classes and pseudo unknown classes in the feature space, providing a clearer decision boundary.
• Extensive experiments on the ADS-B system IQ signal dataset are presented to evaluate the reliability and effectiveness of PSGRNN. We first evaluate our proposed method through an ablation study in terms of convergence and classification performance. Then, the open-set recognition performance is measured along several factors including signal-to-noise ratio (SNR) and the number of unknown classes. This shows that our proposed method outperforms existing solutions. Moreover, experiments on the WiSig datasets have shown that it also has good adaptability to different receivers and time-scale gaps.

2. Related Works

SEI researches extracting subtle features that can uniquely identify the emitter from RF signals. Traditional SEI algorithms can be divided into two categories based on the state of the emitter: transient-based signal features [11,12] and steady-state-based signal features [13,14,15]. The former is independent from the influence of the signal data, but needs to be located due to the short signal, making it difficult to capture. Also, it is sensitive to multipath effects, Doppler frequency shifts, and changes in the internal circuit environment of the emitter. The latter is relatively easy to capture and has better stability, but it requires a high sampling rate for the receiving device and the process is complex. In addition, the above methods need to focus on specific data, rely on expert experience, and have a strong dependence on prior information.

Compared to traditional algorithms, DL methods solve the dependence on prior knowledge and have stronger feature extraction capabilities, demonstrating advantages in the field of signal recognition. Kulin et al. [16] and Sankhe et al. [17] both use time-domain physical-layer I/Q samples to train convolutional neural network (CNN) classifiers, aiming to learn the fine-grained hardware damage caused by circuits, and further identify devices under the internet of things (IoT). Rajendran et al. [18] use time-domain amplitude and phase information as inputs to the long-short term memory (LSTM) network for recognition on a distributed wireless spectrum sensing network. Ref. [19] proposes a hybrid CNN structure that uses raw I/Q samples and frequency domain representations as inputs to identify LTE and WiFi transmissions. In [20], the author utilized a GAN to identify rogue wireless signals through adversarial learning. The above methods can learn inherent features from different specific emitters, and the results show that the DL models are superior to traditional algorithms.

However, these methods are based on the assumption that the class of test set and train set are exactly the same, and use the typical softmax cross-entropy loss to classify. This will inevitably lead to normalization problems, making it inherently a closed set and unable to effectively identify unknown classes. Recent work has shifted towards OSR methods based on DL. Bendale and Boult [21] proposes the first openmax approach to solve the OSR problem of deep neural networks (DNNs). Replacing the softmax layer in DNN with an openmax layer, the activation vector values are reassigned based on Weibull distribution fitting scores, and then used to calculate the probability of unknown classes. Prakhya et al. [22] continued to follow openmax’s technology roadmap and explore open set text classification. Shu et al. [23] replaced the softmax layer with a pair of remaining s-type final layers and proposed a deep open classifier (DOC) model. Ref. [24] proposed a classification reconstruction learning algorithm for open set recognition (CROSR), which utilizes latent representations for reconstruction and achieves robust unknown class detection without compromising the accuracy of known class classification. Oza and Patel [25] proposed a C2AE model for OSR using class conditional autoencoders with new training and testing methods. The success of these OSR algorithms mentioned above is only on image or text data, and they have not been successfully applied in the signal field.

There is relatively little research on OSR algorithms in the field of signals. Wu et al. [26] proposed an OSR method based on the Siamese network for feature extraction and recognition of mobile Bluetooth signals. In Ref. [27], the author proposes a DL-based OSR method for wireless communication signal modulation, which improves the generalized end-to-end loss for training neural networks, increases the similarity of feature vectors of the same modulation type, and reduces the similarity of feature vector quantities of different types. Bassey et al. [28] proposed a new intrusion detection method to detect unauthorized IoT devices. A CNN is used to extract features from radio frequency (RF) trajectories, and perform dimensionality reduction and decorrelation processing on the features. Then, the simplified features are clustered to identify IoT devices. In Ref. [29], the author proposed an extension of resampling in the frequency and time network (RiftNet) model, allowing for simultaneous new device detection and RF fingerprint recognition. Dong et al. [30] proposed a zero-shot learning framework, signal recognition and reconstruction convolutional neural network (SR2CNN), to solve the OSR of signal modulation patterns, identifying unknown signals by learning the signal semantic feature space and gradually improving themselves.

Although existing signal OSR methods have achieved some results, they are limited to optimization in the training of known classes and cannot preprocess unknown classes. Based on the similarity between signals of the same type of emitter, our method aims to use pseudo unknown classes to make the model focus on distinguishing features not only between known classes during training, but also between known and unknown classes in advance, in order to better recognize unknown classes.

3. Proposed Approach

This section formally presents the proposed OSR model PSGRNN. Overall, the proposed framework is mainly composed of the following three modules: a pseudo unknown signal generation module (SG), a feature extraction module (FE), and an open set classification module (OC). Figure 1a shows the structure of PSGRNN. According to the description in [31], the objects to be identified are divided into known known classes (KKCs) and unknown unknown classes (UUCs). In addition, the SG module generates pseudo unknown classes (PUCs) for auxiliary training. Hereafter, the above abbreviations will be used to represent these categories for simplicity.

Our method consists of three main steps. The first step is to input the train set containing only KKCs to the SG module, which generates PUCs by training KKCs. The second step is to input the samples of KKCs and PUCs into the FE module for feature extraction training. The third step is to input the test set containing KKCs and UUCs to the trained FE module to extract features, and then send it to the OC module for modeling and recognition. The goal of our proposed method is to identify the type of KKCs (predict their category label $y\in \{1,\dots ,C\}$), while rejecting any other UUCs by identifying them as outliers (predict their category label $y=C+1$). Note that C is the number of categories of KKCs.

3.1. SG Module

We propose using an SG module to generate the required signal. The SG module is inspired by the architecture of GANs and consists of two parts: a generator G and discriminator D. The details of the network architecture are shown in

Table 1. Network structure of our proposed method.

SG Module	FE Module
Generator G	CVCNN
Input $x_i$	Input $x_i$ and $\overline{x_i}$
Conv 32, (1 × 2), stride 2, ReLU, BN	CvConv 64, (2 × 3), stride 1,
Conv 64, (1 × 2), stride 2, ReLU, BN	BN, ReLU, Dropout, MaxP (1 × 2)
Deconv 32, (1 × 2), stride 2, ReLU	CvConv 128, (2 × 3), stride 1
Deconv 1, (1 × 2), stride 2	BN, ReLU, Dropout, MaxP (1 × 2)
Output $\overline{x_i}$	CvConv 256, (2 × 3), stride 1
Discriminator D	BN, ReLU, Dropout, MaxP (1 × 2)
Input $x_i$ or $\overline{x_i}$	CvConv 512, (2 × 3), stride 1
Conv 32, (1 × 2), stride 2, Leaky ReLU, IN2d	BN, ReLU, Dropout, AvgP (2 × 2)
Conv 64, (1 × 2), stride 2, Leaky ReLU, IN2d	FC 1024, ReLU, Dropout
FC 1024, Leaky ReLU, IN1d	FC 512, ReLU, Dropout
FC 512, Leaky ReLU, IN1d	FC 128, ReLU, Dropout
FC 1, Sigmoid	FC 2C, softmax
Output [0, 1]	Output (1 × 128)

. G can transform the input samples $x_i$ into hidden vectors and reconstruct them into samples $\overline{x_i}$, hoping that the network can learn the mapping: $f_G:x_i\to \overline{x_i}$. G consists of two parts: an encoder and a decoder. The encoder attempts to learn mapping relationships $f_e:x_i\to z_i$, which transforms high-dimensional inputs $x_i$ into low-dimensional hidden variables $z_i$. The decoder attempts to learn mapping relationships $f_d:z_i\to \overline{x_i}$, which reconstitute the encoded $z_i$ into the high-dimensional $\overline{x_i}$. The mathematical model of G is as follows:

$$\overline{x_i}=f_d\left(z_i\right)=f_d\left(f_e\left(x_i\right)\right)=f_G\left(x_i\right)$$

(1)

The function of D is to determine whether $\overline{x_i}$ is the fake data generated by G. The task of the SG module is to generate PUCs. We hope that PUCs are similar to KKCs but can be recognized as UUCs by D. In summary, the whole process can be described as follows: G generates a set of samples $\overline{x_i}$ that are similar to $x_i$ but can be judged as fake by D. The discriminator D uses the minimum cross-entropy method to distinguish $x_i$ and $\overline{x_i}$. The loss function of D is expressed as:

$$\begin{array}{c}{L}_D=-\frac{1}{n}{\displaystyle \sum _{i=1}^n}[log\left(f_D\left(x_i\right)\right)+log(1-f_D\left(\overline{x_i}\right)]\hfill \\ =-\frac{1}{n}{\displaystyle \sum _{i=1}^n}[log\left(f_D\left(x_i\right)\right)+log(1-f_D\left(f_G\left(x_i\right)\right))]\hfill \end{array}$$

(2)

where n is the number of samples, and $f_D\left(\right)$ represents the probability of the real sample. D hopes that $f_D\left(x_i\right)$ is as close as possible to 1. And conversely, that $f_D\left(\overline{x_i}\right)$ is as close as possible to 0.

For G, it uses MSE to learn the distribution of real samples and generates fake data. We hope that they can be as similar to real samples as possible. Therefore, G needs to minimize the MSE loss in the first term of $L_G$. In addition, due to the particularity of PUCs, we still want the generated samples to be recognized as fake by D, so that the differences between PUCs and KKCs can be retained. Thus, the output of D is made to be close to 0 in the second term of $L_G$. The loss function of G is expressed as:

$$L_G=\frac{1}{n}\sum _{i=1}^n\left|\right|x_i-f_G\left(x_i\right)|{|}^2-log(1-f_D\left(f_G\left(x_i\right)\right)$$

(3)

G and D alternate steps in training to optimize $L_D$ and $L_G$. The flow is shown in Figure 1b. Therefore, the SG module differs from a traditional GAN in that it has both a game process and a consistent goal. The generated samples $\overline{x_i}$ of G need to be as consistent as possible with the real samples $x_i$, while D needs to find the fake data created by G as much as possible. This is an adversarial process. Both G and D want the generated data to be identified as fake data, which is a consistency process. In this way, the generated PUCs can participate in FE module training as a new class that is very similar to KKCs.

3.2. FE Module

CNNs are generally used for image feature extraction, while signal data are a special data type which are different from images. For IQ signal samples, there is a specific structure in the complex field. Therefore, we implement the complex-value convolution (CvConv) operation through linear transformation. The CvCNN is constructed as the basis of the FE module, which is suitable for feature extraction of signal samples.

Assuming that the complex representation of the input I/Q signal $x_i$ and convolutional kernel w is as follows:

$$\left\{\begin{array}{c}{x}_i=I+iQ\hfill \\ w=R+iV\hfill \end{array}\right.$$

(4)

where I and Q are the in-phase and orthogonal components of $x_i$, and R and V are the real and imaginary parts of w.

In the complex field, the convolution operation between $x_i$ and w is as follows:

$$x_i\ast w=(I\ast R-Q\ast V)+i(I\ast V+Q\ast R)$$

(5)

The convolution operation in CNNs will change the dimension by zero padding the input data and convolution with the convolution kernel as follows:

$$\left[\begin{array}{ccccc}0& 0& 0& \dots & 0\\ I_0& I_1& I_2& \dots & I_L\\ Q_0& Q_1& Q_2& \dots & Q_L\\ 0& 0& 0& \dots & 0\end{array}\right]\ast \left[\begin{array}{c}V\\ R\end{array}\right]=\left[\begin{array}{c}I\ast R\\ I\ast V+Q\ast R\\ Q\ast V\end{array}\right]$$

(6)

where L is the length of the IQ signal sequence.

As shown in Figure 1c, by using the linear transformation, the first row of the Equation (6) matrix is subtracted from the third row as the real part, and the second row is directly used as the imaginary part to achieve the result of complex-value convolution. Adding a linear transformation block after the convolutional layer can achieve a CvConv layer.

In CvCNN, the main structure consists of 4 CvConv layers, 3 fully connected (FC) layers, 1 classification layer, and several pooling layers. The data are processed through CvConv layers to output a feature matrix through convolution operations, achieving automatic feature extraction. The pooling layer is used to filter out unimportant parts of the feature matrix and reduce the number of parameters. Three FC layers integrate these features and reduce the dimensionality layer by layer to obtain an appropriate dimensionality feature vector. The final classification layer will make a closed set decision. It remains to observe the convergence of the model during the training of KKCs and PUCs. The output of the FE module is the feature vector in the third FC layer. The details of CvCNN are shown in Table 1.

In order to maximize the intra-class similarity and inter-class differences of the extracted features, the FE module is trained using the combination of cross-entropy loss and an improved center loss. In this way, the outputs of different classes are separated in the feature space, and the OC module can better formulate the decision boundary.

First, cross-entropy loss is used to enhance the discrimination between different classes, which is expressed as the difference between the real probability distribution and the predicted probability distribution. Its mathematical expression is as follows:

$$L_{ce}=-\frac{1}{n}\sum _{i=1}^n{y}_{i}log\left(C\left(x_i\right)\right)$$

(7)

where $C\left(x_i\right)$ is the predicted label from classification layer, and $y_i$ is the true label.

Then, center loss is used to drive features of the same class closer to a center. This maximizes the intra-class similarity. The center loss is defined as follows:

$$L_{ct}=\frac{1}{n}\sum _{i=1}^n\left|\right|F\left(x_i\right)-C_{y_i}{\left|\right|}_2^2$$

(8)

where $F\left(\right)$ denotes the output of the third FC layer. $C_{y_i}$ denotes the feature center vector of class $y_i$. The $C_{y_i}$ is updated as the features of class $y_i$ change. The application is updated in each batch according to $C_{y_i}\leftarrow C_{y_i}-\alpha \Delta C_{y_i}$, where $\alpha$ is the learning rate and $\Delta C_{y_i}$ is calculated as follows:

$$\Delta C_{y_i}=\frac{{\sum }_{j=1}^n\delta (y_j=y_i)(C_{y_i}-F\left(x_j\right))}{1+{\sum }_{j=1}^n\delta (y_j=y_i)}$$

(9)

where $\delta (y_i=y_j)=1$ if the condition in the parentheses is true, otherwise $\delta (y_i=y_j)=0$.

In view of the center loss being incapable of separating different classes effectively, we added an inter-class penalty item to improve it. For samples of class $y_i$, it is necessary to keep them away from the nearest other class centers in each iteration so the boundaries between the different classes can be clearer. The inter-class penalty item is defined as follows:

$$p=\underset{j=1}{\overset{n}{argmin}}[(1-\delta (y_j=y_i))\left|\right|F\left(x_i\right)-C_{y_j}{\left|\right|}_2^2{]}_{+}$$

(10)

where $argmin$ denotes the value taken when the minimum value is reached in the parentheses, ${\left[\begin{array}{c}\end{array}\right]}_{+}$ is the cut-off at zero operation.

The center loss of the increased penalty item (hereinafter referred to as class loss) is expressed as:

$$\begin{array}{c}{L_{ct}}^{+}=\frac{1}{n}{\displaystyle \sum _{i=1}^n}\left|\right|F\left(x_i\right)-C_{y_i}{\left|\right|}_2^2-p\hfill \\ =\frac{1}{n}{\displaystyle \sum _{i=1}^n}\left\{\right||F\left(x_i\right)-C_{y_i}{\left|\right|}_2^2-\underset{j=1}{\overset{n}{argmin}}[(1-\delta (y_j=y_i))\left|\right|F\left(x_i\right)-C_{y_j}{\left|\right|}_2^2{]}_{+}\}\hfill \end{array}$$

(11)

Therefore, the total loss function of the FE module is $L_{FE}=L_{ce}+{L_{ct}}^{+}$. During the training of the FE module, the expected distribution of features is controlled by reducing $L_{FE}$.

3.3. OC Module

The OC module receives the feature vectors (hereinafter referred to as $FVs$) output by the FE module for recognition. EVT is commonly used in many visual recognition systems and is an effective tool for modeling the distribution boundary. In OSR, the first step is to determine when an input is likely not from KKCs. In the OC module, we follow the Weibull distribution of EVT to model the decision boundary of KKCs and UUCs.

In order to use FVs to identify KKCs and reject UUCs, the concept of centroids is adopted. Taking the class-i KKCs as an example, all training samples of class-i KKCs are input into the FE module to obtain their $FVs$, and the $FVs$ of correctly classified samples are retained. The set of $FVs$ retained is $F{V}_i=\{F{V}_1,F{V}_2,\dots ,F{V}_k\}$, where k refers to k samples in class-i KKCs training samples recognized by CvCNN as class-i. The centroid of the class-i samples is defined as the mean feature vector of $F{V}_i$, which is represented as $MF{V}_i$.

The set of distances between each $F{V}_i$ and the centroid $MF{V}_i$ is denoted as $D_i=\{D_1,D_2,\dots ,D_k\}$. The boundary of a class is defined as the point farthest from the centroid $MF{V}_i$ in the class samples, i.e., the maximum distribution of the random variable sequence $D_i$. The maximum distribution in $D_i$ is normalized by fitting the Weibull distribution of EVT, and the result is the cumulative distribution function (CDF) of the Weibull distribution in Equation (9), representing the probability of the maximum appearing in.

$$CDF\left(x\right)=\left\{\begin{array}{c}1-e^{-\left(\frac{x}{\eta }\right)\beta },x\ge 0\\ 0,x<0\end{array}\right.$$

(12)

where x is a random variable, $\beta$ is a shape parameter, and $\eta$ is a scaling factor.

The larger the maximum value of $D_i$ is, the farther the sample is from the centroid of a certain class. This means that the sample is less likely to belong to that class. That is, $CDF\left(x\right)$ is used to estimate the probability of an input being an outlier with respect to class-i. Based on this probability distribution, a simple rejection model will have a threshold defined by the user to determine whether an input should be rejected (for example, the definition should have at least 90% of the samples identified correctly).

4. Experimental Results

In this section, we evaluate the performance of the proposed method and illustrate its benefits and improvements by conducting various experiments on the ADS-B system IQ signal dataset. We first introduce the data used, then present an analysis of the proposed method, and finally, compare the performance of our method with that of recent state-of-the-art open set recognition algorithms.

4.1. Dataset

Given that in the past few years RF fingerprint recognition methods have been successfully used in the automatic dependent surveillance–broadcast (ADS-B) system of air traffic control to identify aircraft [32], the experiment uses ADS-B system IQ signals from multiple aircraft as data support. There are a total of 25 classes of ADS-B signals. In the experiment, 15 classes are selected as KKCs, labeled as 0–14. The remaining 10 classes are selected as UUCs. The train set and test set are randomly divided in an 8:2 ratio. For samples of UUCs, they will not be divided into the train set, and only 20% of them are selected to join the test set. The data description is shown in

Table 2. Collection settings of ADS-B signals dataset.

Classes	Samples of Each Class	Total Samples	Feature Dimension
25	100	2500	2 × 4096
Train set	15 classes (KKCs)		1200 samples
Test set	25 classes (KKCs + UUCs)		500 samples

.

In order to better illustrate the experimental results, the OSR classification problem definition in

Table 3. Open set classification definition.

True Label		Predict Label
		K		U
		P	N
K	P	TP	FN	FU
	N	FP	TN
U		FK		TU

is obtained based on the definition extension in the binary classification problem. K: KKCs, U: UUCs, P/N: different classes of KKCs, TP/TN: correctly classified KKCs, FP/FN: internally misclassified KKCs, TU: correctly classified UUCs, FK: UUCs are classified as KKCs, FU: KKCs are classified as UUCs. Thus, the following indicators can be obtained:

$$\mathrm{KKCs}\mathrm{accuracy}=\frac{TP+TN}{TP+FP+TN+FN+FU}$$

(13)

Accuracy is the most common evaluation metric, which is the number of samples correctly classified divided by all samples. Generally speaking, the higher the accuracy, the better the classifier performance. KKCs accuracy represents the rate of correctly classified KKCs samples to all KKCs samples. It reflects the classifier’s recognition performance for KKCs.

$$\mathrm{UUCs}\mathrm{accuracy}=\frac{TU}{FK+TU}$$

(14)

Similarly, UUCs accuracy represents the rate of correctly classified UUCs samples to all UUCs samples. It reflects the classifier’s recognition performance for UUCs.

$$\mathrm{Average}\mathrm{accuracy}=\frac{TP+TN+TU}{TP+FP+TN+FN+FK+TU+FU}$$

(15)

Average accuracy represents all correctly classified samples divided by the total number of samples. It is an evaluation of the overall classification performance.

$$\mathrm{Flase}\mathrm{UUCs}\mathrm{rate}\left(\mathrm{FUR}\right)=\frac{FU}{TP+FP+TN+FN+FU}$$

(16)

FUR represents the rate of KKCs samples that are misclassified as UUCs to all KKCs samples. This reflects the recognition error in the classifier.

4.2. In-Training Views

The network of the SG module and FE module is implemented using the Python framework under the Windows 11 platform and trained on NVIDIA RTX A6000. The parameter settings during the training are shown in

Table 4. Training parameter configuration.

Train set size of SG module	1200
Train set size of FE module	2400
Batch size	128
Epoch (SG/FE)	500/1000
Learning rate	1.00 × 10${}^{-3}$
Optimizer	Adam

. Train set size represents the total number of train samples. Batch size represents the number of samples fed into the neural network for training in each batch. Epoch represents the number of network training iterations. Learning rate is a hyperparameter that controls how quickly neural network parameters are updated. Optimizer is a tool that guides the neural network to update the parameters.

The SG module alternately optimized and updated G and D during training with 500 epochs, with the discriminator loss $L_D$ and generator loss $L_G$ shown in Figure 2a. During the training process, the two undergo severe oscillations, which is reasonable. Although we make G and D have consistent goals, which weakens the adversarial process of GANs, as the generated samples of G become closer to the original sample, it will pose a challenge to D, causing the loss value of D to increase. Additionally, because there is a common part of the loss between the two, G oscillates.

Finally, the generator loss converges to about 0.05, which means there is only a small error between the generated PUCs and KKCs. Figure 2b shows the I-channel waveform of an input KKCs sample and the corresponding generated PUCs sample. It can be seen that the two are very similar, with only slight differences. The discriminator loss almost converges to 0, indicating that the PUCs generated by G can be well distinguished from KKCs, ensuring that the combination of PUCs and KKCs does not cause any inconvenience during training in the FE module.

The FE module receives both KKCs and PUCs inputs, requiring training to double the number of samples, thus increasing the epoch compared to the SG module. In addition to observing the loss during training, the softmax function of the classification layer is also used for closed set recognition to calculate the accuracy of the KKCs train set, which can to some extent help observe the FE module’s discrimination of features extracted from different classes and the convergence of the network. As shown in Figure 2c, the train accuracy converges to approximately 97% and the loss converges to 0 on the train set. It can be seen that PUCs have no impact on the convergence of the FE module.

4.3. Analysis of Our Approach (Ablation Study)

To understand the rationality of the proposed method, we evaluated each module through ablation experiments, alternately deleting a certain part of the model to measure the performance gain it contributes. The explanation of the ablation method is as follows:

• Without SG: delete SG module without generating PUCs, FE module only trains KKCs samples.
• Without class loss: the class loss of the FE module loss function is deleted, and only cross-entropy loss is used for training.
• Without LT block: the linear transformation block in the FE module network structure is deleted, so that it is reduced to a common CNN.
• Our method: this is our complete method.

In addition, we have provided the softmax accuracy and OC module’s accuracy of KKCs for all four methods mentioned above, in order to evaluate the performance of the OC module.

Firstly, evaluate the convergence performance of the four methods. Figure 3 shows the (FE) loss changes in the four methods in the training iteration and the softmax accuracy changes in KKCs in the test set. Method 3 achieved the fastest convergence in both the loss and accuracy changes, which is attributed to the removal of linear transformation blocks making the network structure simpler. Method 2 and our complete method are almost the same in the two figures, but the role of class loss is more likely to provide a better decision boundary for the OC module, and it seems that it makes little contribution to the softmax method. Method 1, which lacks the SG module, performs poorly. The results shown in Figure 3 indicate that our complete method can achieve the best final convergence performance.

The final convergence models of the four methods were used to extract FVs of KKCs and send them to the OC module for recognition.

Table 5. KKCs recognition performance of all methods under the OC module and softmax.

Method	KKCs Accuracy
	OC Module	Softmax
Without SG	60.67%/9.67%	71.48%
Without class loss	85%/9.67%	92.38%
Without LT block	77.19%/9.67%	81.63%
Our method	89%/9.67%	93.16%

shows the relevant results. Due to the decision threshold of the OC module, the rejection threshold of the OC module was set with FUR < 10% in the experiment to ensure that at least 90% of KKCs can be accepted. The results of method 1 under the OC module were 60.67%/9.67%, with 60.67% being the KKCs accuracy and 9.67% being FUR. Overall, there is a 4–10% decrease in the KKCs accuracy of the OC module compared to softmax, as some KKCs will be rejected by the OC module and judged as UUCs. In the experiment, the FUR of all four methods is 9.67%, so an error within 10% is reasonable. Method 1 and method 2 have significant losses in this comparison, while method 3 and our complete method only differ by about 4%, which proves the contribution of the SG module and class loss in optimizing feature space distribution and improving the performance of the OC module.

Then, we turned to the experiment on UUCs, inputting the test set containing KKCs and UUCs to the FE model and the OC module corresponding to the four methods. The number of UUCs has gradually increased from 1 class to 10 classes. Figure 3a shows the UUCs accuracy of the OC module under changes in the number of UUCs classes. It can be seen that the accuracy fluctuates as UUCs increases. This phenomenon is caused by some UUCs being easier to identify, while the overall downward trend is inevitable: more untrained samples appear, posing risks to the OC’s decision making. In addition, we found that the contribution of the SG module to recognizing UUCs is enormous, and the UUCs accuracy without the SG module method is only about 30%. According to the results of Figure 4a and Table 5, the overall average accuracy in Figure 4b can be obtained. It can be clearly seen that our complete method has the best OSR performance, with an average accuracy of 90% for 15 classes of KKCs and 10 classes of UUCs.

Figure 5 visualizes the feature spaces of method 1 and our complete method through t-sne dimensionality reduction, where 0–14 represent 15 classes of KKCs and −1 represents UUCs. The UUCs feature distribution of method 1 is more confusing than that of our complete method, resulting in poor recognition performance. This verifies that using PUCs-assisted training can predict the distribution of UUCs to a certain extent, separating them from KKCs.

4.4. Open Set Recognition Performance

In order to prove the progressiveness of PSGRNN, we tested the OSR performance under different SNRs, and compared it with two new OSR methods: CROSR [25] and SR2CNN [31]. CROSR utilizes latent representations for reconstruction to enhance the useful separation of KKCs and UUCs in learning. SR2CNN proposes a ZSL framework for signal recognition and reconstruction of convolutional neural networks. Both of them use reconstruction to enhance performance, while the core idea of PSGRNN is to generate PUCs based on KKCs for auxiliary training.

Firstly, the adaptability of several OSR methods to the number of UUCs classes in the absence of SNRs is compared, and the average accuracy is used as an indicator. As shown in Figure 6a, compared with other methods, our method has the best adaptability to UUCs of more classes. When there were 10 UUCs classes, the average accuracy of our method was better than those of the CROSR and SRCNN methods by 8.79% and 4.62%, respectively.

Then, with the number of UUCs remaining at 10 classes, we compared our method with CROSR and SR2CNN under different SNRs. As shown in Figure 6b, it is clear that our method provides the best performance at different SNRs compared to the others. Moreover, at lower SNRs, the performance wins by a much larger margin than higher SNRs compared to other models. It can be concluded that our proposed method is the most robust against varying SNRs in the transmission environment.

Moreover, Ref. [33] shows that evaluating the same emitter in different channels can cause a serious decrease in the accuracy of the classifier. Receivers have manufacturing differences like emitters, and the collection time-scale gaps between training and testing tend to reduce classification accuracy. Practical deployments of SEI systems may involve multiple receivers and need to work over prolonged periods of times where the channel varies. Therefore, to verify the sensitivity of our method to the receiver and time, an additional experiment is performed on the WiSig dataset [34]. WiSig provides WiFi transmitters (TX) RF signals received by different receivers (RX) on different days. Several settings are shown in

Table 6. Collection settings of WiSig dataset.

TX	RX	Day	Samples/Class	Feature Dimension
6	12	4	1000	2 × 256
Train set	4 classes (TX 1–4, RX 1, Day 1)			3200 samples
Test set	6 classes (TX 1–6, RX 1–12, Day 1–4)			18,000 samples

.

In the experiment, four TXs are trained using data from RX1, day 1. In testing, six TXs are used, and RXs or days are modified in turn. For example, using different days under RX1 or using different RXs on day 1. The purpose of this division is to investigate whether a trained classifier can be used on different RXs than those used to collect the training data. Figure 7a shows the effect of different days. The test data from day 1 is consistent with the training data, so it has the highest accuracy. Compared to day 1, the test results for other days are also satisfactory. Both KKCs and UUCs maintain an accuracy of more than 90%. This proves that our method has superior robustness to the collection time. Figure 7b shows the effect of different RXs. Obviously the RX has a greater influence on the channel, but our method maintained good performance in most of the 12 RXs, keeping the drop within 20%. Compared with the relevant results in [34], our method still has higher accuracy and stability, even in the open set field.

Obviously various factors in the sampling situation can affect the performance of classification. Signals sampled under different RXs have completely different accuracy, and signals sampled at different acquisition times have different performance degradation. The results show that our method has good adaptability to these factors.

5. Conclusions

In this paper, we propose an OSR model PSGRNN for SEI, which solves the challenge of being unable to pre-train all emitters types in practical scenarios. PSGRNN utilizes the structure of a generator and a discriminator to generate pseudo unknown signals. These signals are used for auxiliary training to strengthen the recognition ability of unknown signals when it is impossible to obtain and train them. In addition, the class loss in CvCNN can further improve the performance and the EVT is applied to model open set classification boundaries. Extensive experiments on the ADS-B system IQ signal dataset and WiSig dataset have demonstrated the effectiveness of PSGRNN. The results show that our proposed method outperforms several state-of-the-art open set recognition methods in signal recognition. To further improve the recognition performance, the transfer learning scheme can be adopted in future work to adapt to changes in the wireless environment.