A Review of Research on Signal Modulation Recognition Based on Deep Learning

Abstract

Since the emergence of 5G technology, the wireless communication system has had a huge data throughput, so the joint development of artificial intelligence technology and wireless communication technology is one of the current mainstream development directions. In particular the combination of deep learning technology and communication physical layer technology is the future research hotspot. The purpose of this research paper is to summarize the related algorithms of the combination of Automatic Modulation Recognition (AMR) technology and deep learning technology in the communication physical layer. In order to elicit the advantages of the modulation recognition algorithm based on deep learning, this paper firstly introduces the traditional AMR method, and then summarizes the advantages and disadvantages of the traditional algorithm. Then, the application of the deep learning algorithm in AMR is described, and the identification method based on a typical deep learning network is emphatically described. Afterwards, the existing Deep Learning (DL) modulation identification algorithm in a small sample environment is summarized. Finally, DL modulation is discussed, identifying field challenges, and future research directions.

1. Introduction

In recent years, the rapid development of wireless communication technology, especially the new generation of communication technology led by 5G and 6G technology, has attracted the attention of academic and commercial circles. Among them, 5G technology can support three major application scenarios of enhanced mobile communication, ultra-reliable and low-latency, and massive machine-type communication, and promote the application of artificial intelligence products [1], which drives the development of the Internet of Things and greatly promotes economic growth, which has facilitated the production and life of human beings. With the commercialization of 5G technology, the world’s major communications powers have begun preliminary studies on 6G, noting that 6G communications technology will have faster data rates, greater system capacity and wider network coverage, 6G technology can connect the communication systems of satellites, drones, and submarines to truly realize the internet of everything [2]. Since 5G and 6G communication technologies are capable of generating massive amounts of communication data and deep learning techniques in artificial intelligence are capable of learning features in massive amounts of data and accomplishing problems such as channel prediction and intelligent signal generation and processing that are difficult for humans to handle, researchers in the communication field have begun to introduce deep learning into various layers of wireless communication systems, such as the communication physical layer, communication link layer, and communication network layer. At present, Tsinghua University, Zhejiang University, Southeast University, University of Electronic Science and Technology of China, MIT, Stanford, and other universities have carried out research on deep learning in the communication physical layer. The research content includes automatic modulation recognition based on deep learning, deep learning-based channel state estimation, prediction and feedback based on deep learning, channel coding and decoding technology based on deep learning, etc. [3]. In this article, the automatic modulation identification technology based on deep learning will be surveyed.

Automatic Modulation Recognition (AMR) is an intermediate step between signal detection and signal demodulation. This technique is able to identify the type of modulation of the signal and thus obtain the information contained in the signal without knowing the system parameters. It can be seen that AMR technology is a prerequisite for demodulating signals at the receiving end, and it is also a key link in wireless communication, playing a key role in both civilian and military fields. In the civil field, AMR is mainly used for spectrum monitoring and interference identification [4]. The spectrum resources in wireless communication are very limited, but a large number of communication services occupy most of the spectrum resources, resulting in a shortage of spectrum resources, which may cause some illegal organizations to occupy the public spectrum illegally and seriously interfere with normal communication, so the spectrum resources need to be managed. AMR technology can identify the modulation method of interference signals and legitimate user signals, analyze the properties of signals, and complete the supervision of the spectrum. In the military field, AMR technology can identify the interference information and key military information sent by the enemy and help the military formulate targeted reconnaissance and anti-reconnaissance strategies. It can be seen that the development of AMR technology has important strategic significance.

After decades of development of automatic modulation recognition technology, its traditional recognition methods can be roughly divided into two categories: likelihood-based methods [5,6] and feature-based methods [7]. Likelihood-based methods have theoretical optimality, but the amount of computation is extremely large [8], while feature-based methods require manual feature extraction, resulting in recognition results that rely heavily on experts’ feature extraction experience. Therefore, these two recognition methods are no longer suitable for increasingly complex communication systems. In the past few years, scholars in the field of communication have applied typical networks in deep learning, such as convolutional neural networks [9] and recurrent neural networks [10], etc. AMR has shown deep learning-based modulation recognition algorithms perform better than traditional modulation identification methods. Since then, many researchers have started to study automatic modulation recognition algorithms based on deep learning. Y. Wang et al. combined two convolutional neural networks to complete modulation recognition [11]. In the literature [12], the authors used deep belief networks (DBN) and spiking neural networks (SNN) to successfully complete the AMR task under a low signal-to-noise ratio.

The basic idea of the DL-based AMR algorithm is to input a large number of labeled modulation signal datasets (currently commonly used open-source modulation signal datasets, RadioML2016.10a, RadioML2016.10b, RadioML2016.04c, RadioML2018.10A, and HisarMod2019.1) into the neural network. In the network, the neural network learns and extracts the features of the signal in the labeled dataset [13], and finally completes the classification according to the features. The entire process does not require a manual signal feature extraction [14] and can achieve high recognition accuracy. It can be seen that the DL-based AMR method has the following advantages: less expert knowledge, strong feature extraction ability, and high recognition accuracy. Therefore, many scholars have designed different deep learning networks for recognition.

It is worth noting that although the DL-based AMR method performs well, it also has limitations: it is well known that deep learning models can only achieve good classification results with huge datasets. If the dataset is too small, then the depth of the learning model is prone to overfitting, resulting in a poor final learning classification effect. In general, if the number of samples in each category in the dataset is only tens or hundreds, then the data is a small sample dataset [15]. For modulated signals, if a modulated signal has only a few hundred samples, the modulated signal dataset is a small sample dataset.

In the field of communication, there are many occasions, such as deep-sea communication, radar communication, etc., that cannot obtain huge, labeled datasets, which will lead to poor performance of DL-based AMR algorithms. Therefore, to solve the problem of low modulation recognition accuracy in a small sample environment, researchers need to develop an AMR algorithm suitable for a small sample environment. At present, some scholars have begun to study the DL-based AMR algorithm under the condition of small samples. The authors of [16] proposed an easy-to-train meta-learner, Meta-SGD, which can achieve better learning results with a few data samples. Li et al. used the transfer learning method in their work [17] to improve the modulation recognition accuracy in the case of small samples.

According to the above introduction, we know that the modulation recognition algorithm based on deep learning has a large gap when the amount of sample data is sufficient or insufficient. At present, there are many algorithms in these two cases, so it is of great significance to summarize the DL-based AMR algorithms in these two cases for subsequent research. For the past two years, several researchers have reviewed deep learning-based modulation recognition algorithms for the period from 2017 to 2021 [18,19], where research by [18] focuses on the application of classical neural networks (e.g., CNN, RNN, and FFNN) in deep learning for modulation recognition; and studies by [19] focuses on a review of datasets, signal representation methods, and deep learning-based approaches in the field of modulation recognition. Although research by both [18,19] have made relevant contributions to deep learning-based AMR algorithms, most research in the literature summarizes AMR methods using classical neural networks and ignores the relevant descriptions of deep learning modulation recognition methods in difficult environments, e.g., small sample environments. Therefore, in order to provide a complete reference for subsequent research, it is necessary to summarize the new deep learning modulation recognition algorithms and further summarize the related algorithms in the small sample environment.

This review fills a gap in the analysis of deep learning-based modulation recognition algorithms in difficult environments. The contributions of this paper are as follows:

(1)

The traditional modulation identification algorithm is investigated, and the characteristics of the signal and the advantages and disadvantages of the traditional modulation identification algorithm are summarized in detail;

(2)

This paper introduces the application of CNN, RNN, combined neural network, and other types of neural networks in modulation recognition, and makes a comprehensive summary of new algorithms for modulation recognition based on deep learning from 2020 to 2022;

(3)

This paper describes the challenges faced by deep learning in identifying modulated signals in a small sample environment and summarizes the solutions to these challenges.

The remainder of this paper is organized as follows: In Section 2, the traditional modulation recognition algorithms are described, and the advantages and disadvantages of the two algorithms are summarized at the end; the modulation recognition algorithm based on deep learning is discussed along with the problem that the current deep learning algorithm has poor recognition accuracy in the case of small samples; in Section 4, we introduce the application of deep learning in modulation recognition in a small-sample environment; in Section 5, the challenges and future research directions for automatic modulation recognition are described; finally, the conclusions are summarized in Section 6. The structural layout of the article can be illustrated in Figure 1.

2. Traditional Methods of Modulation Recognition

Early signal modulation identification work mainly relies on experienced experts to carry out manual operations. The final judgment result is more subjective, and the accuracy rate is difficult to guarantee. In this case, automatic modulation identification technology came into being and became the keyway to solving the above problems.

2.1. Identification Method Based on Likelihood Ratio

The modulation identification method based on the likelihood ratio is regarded as a multiple-hypothesis problem, and the probability of signal misclassification is minimized by the maximum likelihood criterion to achieve the theoretical optimal identification. The flow of the method is shown in Figure 2:

As can be seen from Figure 2, the key to the likelihood recognition method is to construct the likelihood function of the signal and set the appropriate threshold. Therefore, when using this method, it takes a lot of effort to construct the likelihood function and then select the appropriate threshold. The authors of [20] explored the performance of the likelihood-based algorithm in linear digital modulation classification, and the results showed that the computational complexity of the quasi-mixed natural ratio test method when using signal amplitude, phase, and noise power as unknown parameters was much smaller than the mixed likelihood ratio test method. The authors of [21] proposed a new modulation recognition algorithm by optimizing the combined log-likelihood function (LLF), which has a correct classification probability close to the theoretical limit. Zhang et al. proposed a likelihood estimation classifier based on an expectation maximization algorithm [22] to solve the modulation classification problem in a blind channel multipath environment. This classifier is able to find maximum likelihood estimates of the location parameters in a tractable manner. Experiments show that the performance of the algorithm in multipath channels is significantly improved under high signal-to-noise ratio environments.

2.2. Feature-Based Recognition Algorithms

The feature-based (FB) modulation identification method is a very important method in the development of AMR technology. This method does not need to strictly deduce the likelihood function of the signal, but only needs to be able to extract the representative features of different signals. The classification can be completed according to the corresponding features, and the classification accuracy (the classification accuracy here can be expressed as $P_{cc}=S_c/S\times 100\%$, where $S_c$ is the number of correctly classified samples, $S$ is the given total sample size) can meet the requirements. Therefore, the feature-based modulation identification algorithm is often applied in practical engineering due to its low computational complexity and high accuracy. The flow of this method can be shown in Figure 3.

As shown in Figure 3, the method first preprocesses the signal, then extracts the features of the signal, and finally uses the classifier to classify. In the whole recognition process, feature extraction and classifier design are two very important steps. Therefore, researchers must choose appropriate signal features and design good classifiers. There are many kinds of signal features, and different features have different characteristics. For example, the transient features of the signal do not require any parameter estimation and have low preprocessing requirements, but in fading channels they are prone to fluctuations, leading to limited learning ability of the classifier. The high-order statistics feature of the signal has strong anti-noise ability, has certain advantages in the extraction of weak features, and has better recognition under low signal-to-noise ratio. We summarize the now commonly used signal features as well as classifiers in Figure 4.

The computational complexity of the feature-based recognition method is greatly reduced, and a satisfactory level of recognition accuracy can be achieved. Therefore, this method is favored by the majority of researchers. Since then, many scholars have been devoted to research on feature-based modulation recognition methods. Liedtke [23] proposed a digital modulation identification method, which uses the histogram of signal amplitude, frequency, and differential phase, as well as characteristic functions such as amplitude variance and frequency variance and adopts the classification method of pattern recognition to automatically classify signals according to the nearest principle. The authors of [24] proposed to extract the signal’s instantaneous features to identify the type of digital modulation. Later, to solve the problem of poor robustness of transient features, some scholars proposed using higher-order accumulations of signals [25,26,27], cyclic spectra [28,29,30], and higher-order spectra [31] as features and using machine learning algorithms to design classifiers to improve recognition accuracy. Among them, the most commonly used classifiers are as follows: K-Nearest Neighbor [32], support vector machine [25,26,30], random forest [33], and artificial neural network [34].

Although the combination of signal features and machine learning algorithms can achieve better recognition results, the complexity of classifier rule design in complex communication systems is still very high, the scalability is poor, and the recognition accuracy is relatively low in low signal-to-noise ratio environments. Therefore, the modulation identification methods based on machine learning still have great limitations. Accordingly, researchers still have to continue to improve the accuracy of AMR algorithms.

According to the above-mentioned introduction concerning the likelihood recognition method (LB) and the feature (FB) recognition method, we summarize the advantages and disadvantages of these two methods in

Table 1. Summary of advantages and disadvantages of traditional methods for automatic modulation identification.

Method	Advantages	Disadvantages
Modulation identification method based on the likelihood ratio	Have a complete theoretical foundation	The derivation of the likelihood function is complicated, and the amount of calculation is large
	The classification effect is very good	Poor applicability
	Better performance at low signal-to-noise ratio	Requires a small amount of prior knowledge Requires a lot of prior knowledge
Feature-based modulation identification	Simple theoretical analysis	The identification system is complex
	Features are easy to extract when the signal-to-noise ratio is high	There is no complete theoretical basis

.

According to Table 1, we can see that both the LB-based recognition method and the FB-based recognition method need to rely on experts’ experience to extract signal features and need to set thresholds and rules when designing classifiers. Such methods are difficult to meet the recognition requirements of modern, complex communication systems. Therefore, how to improve the modulation recognition rate in complex communication systems has become an important problem that scholars need to solve. In recent years, the development of deep learning has brought a turning point to solving the problem of low accuracy of traditional modulation recognition. Many researchers have tried to apply deep learning algorithms in the field of modulation recognition, and now they have achieved many excellent results. Section 3 focuses on deep learning-based modulation recognition methods.

3. The Modulation Recognition Method Based on Deep Learning

Deep learning is a powerful artificial intelligence technology that can learn features from a large amount of data and fit nonlinear networks, so it is widely used in the fields of computer vision, natural language processing, and speech recognition, and has achieved tremendous success. Since mobile communication networks are able to generate large amounts of different types of data at a very fast pace, relevant researchers have applied deep learning to the field of communication [35], bringing opportunities for the development of communication technologies. For example, signal modulation identification in wireless communication can be done using deep learning techniques, and deep learning-based modulation identification methods have better robustness than traditional AMR methods and higher accuracy rates.

Essentially, the modulation recognition algorithm based on deep learning is a feature-based recognition method that needs to extract the features of the signal. However, deep learning can automatically extract and classify the features of signals, replacing the steps of relying on expert experience to identify features in traditional recognition algorithms, and the recognition accuracy is higher. The modulation recognition model based on deep learning can be represented by Figure 5.

There are many excellent neural networks in deep learning, such as CNN, RNN, etc. Among them, CNN is good at processing image data and RNN is good at processing sequence signals. CNN and RNN are widely used in AMR. In this section, we will introduce the application of these neural networks to deep learning in detail.

3.1. Application of Convolutional Neural Network in Modulation Recognition

3.1.1. Convolutional Neural Networks

The convolutional neural network (CNN) is one of the most popular and successful deep learning architectures, which consists of multiple convolutional layers, pooling layers, and fully connected layers, and its structure diagram is shown in Figure 6. Among them, the convolutional layer can extract different features of the input data, the pooling layer can downscale the high-dimensional features after convolution to improve the computation speed, and the fully connected layer can combine the previously extracted local features into global features and finally complete the classification according to the features.

CNN is ideal for image processing because it can accurately extract feature information from images using convolution. Therefore, when identifying the type of the modulated signal, scholars usually process the signal into two-dimensional images such as constellation diagrams and time-frequency diagrams, then use convolutional layers to extract the features of the signal from the image, and the classification is done by the fully connected layer. On the other hand, CNN is also widely used in text and signals, so many scholars use CNN to directly extract features from signals.

3.1.2. The Modulation Recognition Method Based on Convolutional Neural Network

1.

A two-dimensional image recognition method based on CNN.

Converting the signal to a 2D image and then using a CNN to identify the modulation is very popular, so we will introduce this method in this section.

(1) Constellation map

A constellation diagram is a graphical insight into the projection of a signal into an orthogonal vector space, the dimensionality of which is determined by the specific type of modulation. Two-dimensional constellation diagrams are by far the most common, which can reflect different characteristics of different modulation types. This paper shows the constellation diagram of the 8PSK signal at 20 dB, 15 dB, 5 dB, and 0 dB in Figure 7. It can be seen from Figure 7 that the features of the constellation diagram are more obvious under high signal-to-noise ratio and less obvious under low signal-to-noise ratio. Therefore, some scholars will convert the single-channel constellation diagram into a three-channel color constellation diagram to increase the recognition rate, which can achieve better results, so constellation diagrams are often used in modulation identification.

Shengliang Peng et al. [36] converted complex signals into constellation maps and applied two popular CNN models (AlexNet and GoogleNet) to the recognition of complex signals and designed a CNN-based modulation recognition algorithm. Finally, experiments showed that when the signal-to-noise ratio is greater than 6 dB, the recognition rate of the algorithm for eight kinds of modulation signals reaches more than 95%, which is better than the traditional support vector machine. However, when the signal-to-noise ratio is less than 1 dB, the recognition accuracy of the model is less than 80%. It can be seen that the recognition accuracy under low signal-to-noise ratio needs to be improved.

In order to improve the recognition classification accuracy, X. Tian [37] processed the constellation map into a heat map with colored shadows. Then, the typical CNN model is used, namely VGG16, VGG19, InceptionV3, Xception, and ResNet50 to identify the constellation heatmaps of six modulation methods. Among them, ResNet50 has the best classification accuracy, and the accuracy rate can reach more than 95%. However, when the signal-to-noise ratio drops to 2 dB, the recognition accuracy drops to 80%, so this method is only suitable for high signal-to-noise ratio environments.

Due to the lack of relevant research on DL-based AMR in MIMO-OFDM systems, the authors of [38] proposed a series of constellation multimodal feature networks (SC-MFNet) for the modulation recognition problem of MIMO-OFDM systems in their work. The SC-MFNet network has four parts, including the feature extraction module based on Conv1DNet, the constellation feature extraction module based on efficient network, the multimodal feature fusion module, and the full connection classifier. The authors input the waveform diagrams of the five modulated signals (BPSK, QPSK, 8PSK, 16 QAM, and 32 QAM) and the segmented accumulated constellation diagrams into the SC-MFNet network, and the network extracts the features of the signal waveform diagrams and constellation diagrams and fuses the features. Final classification experiments show that the SC-MFNet has a recognition accuracy of 95% when the signal-to-noise ratio is 0 dB.

(2) Eye diagram

The eye diagram is a waveform displayed by a series of digital signals accumulated on the oscilloscope that can reflect the overall characteristics of the digital signal. The eye diagram characteristics of different modulation signals are different, so the signal can be converted into an eye diagram, and then the deep learning network model is used to extract the features in the eye diagram to complete modulation recognition. Converting the modulation signal into an eye diagram is an important step in the modulation identification process. After reviewing the literature, there are three most commonly used conversion methods: The first is to use the dedicated eye diagram generation module of the oscilloscope to convert the modulated signal into the corresponding eye diagram [39]; the second is to use the eye diagram function in MATLAB to generate the eye diagram; and the third is that researchers need to write their own programs to complete the conversion of the eye diagram. Since the eye diagram itself is generated by the afterglow effect of the oscilloscope, the first method mentioned above will be more accurate when generating the eye diagram. Due to the limitations of experimental conditions, this paper uses the eye diagram function in MATLAB to convert QPSK, 8PSK, 16PSK, 4QAM, 16QAM, and 256QAM signals into eye diagrams, as shown in Figure 8.

The authors of [40] demonstrated preliminary results of deep learning for modulation identification through eye diagrams of signals. The paper first convolves the I and Q eye maps of the signal, secondly connects the I and Q eye maps, then performs maximum pooling, and finally experimentally verifies that the model can achieve a 100% recognition rate for OQPSK as well as BPSK. The recognition rate of 16 QAM is less than 80%.

The authors of [41] considered that the original eye diagram did not consider the signal aggregation degree at a specific position, so they enhanced the eye diagram. The authors use the enhanced signal eye diagram as the input to the neural network, and then extract and map features of different dimensions using a multi-input CNN model. Experiments show that the recognition accuracy of the model for BPSK, QPSK, OQPSK, 8PSK, and 16APSK is close to 100% at 2 dB, but the recognition accuracy for 64 QAM, 32 QAM, and 16 QAM is low. Therefore, it is necessary to further improve the intraclass recognition of modulated signal accuracy.

Dan. W et al. [39] proposed a CNN-based optical signal modulation recognition and classification algorithm. The author uses the eye pattern generation module of the oscilloscope to generate four modulated optical signals (return-to-zero on-off keying (RZ-OOK), non-return-to-zero keying (NRZ-OOK), RZ-differential phase shift keying (RZ-DPSK), and four-pulse amplitude modulation (4PAM) are converted into eye diagrams, and then CNN is used to learn the features of the eye diagrams and complete the classification. The recognition rate of the four modulation methods is close to 100%.

(3) Time-frequency diagram

If only the characteristics of the signal in the time and frequency domains are analyzed, the characteristics of the signal may be lost. Therefore, to observe the relevant characteristics of the signal more thoroughly, scholars will perform time-frequency analysis of the signal, such as using wavelet transform and other methods, such as converting the signal to a time-frequency map and extracting the features from the time-frequency map. Therefore, it is also a popular method to use deep learning to extract time-frequency map features to complete modulation recognition. We have drawn the time-frequency diagrams of 2FSK, 4FSK, 2PSK, and 4PSK signals in Figure 9, and we can see that different modulation signals have different time-frequency diagrams.

The authors of [42] used neural networks to process time-frequency images of radar signals to identify the modulation types of radar signals. This paper first uses complex wavelet transform to obtain the time-frequency image of the signal, and then uses image cropping, grayscale, adaptive filter normalization, and other steps to enhance the time-frequency image. The results show that when the signal-to-noise ratio is −7 dB, the recognition rate reaches more than 92%, which fully proves that the time-frequency diagram of the signal can well reflect the characteristics of the modulated signal. The author also proposed the Sep-ResNet model for recognition. After comparison, the Sep-ResNet model is better than the ResNet50 and VGG networks.

The authors of [43] proposed an LPI radar signal recognition method based on dual-channel CNN and feature fusion. The authors used the wavelet transform method to convert the signal into a time-frequency map, and the time-frequency map was processed in grayscale. Subsequently, it was inputted into the two-channel CNN model, which can extract two features, the oriented gradient (HOG) and the depth feature histogram, from the signal time-frequency map and finally fuse the two features and classify them. The classification method has a signal-to-noise ratio of 6 dB, and the recognition rate can reach more than 95%.

The authors of [44] proposed a modulation and identification method of impulse noise communication signals based on fractional low-order Choi–Williams distribution and CNN, aiming at the low recognition rate of a communication signal in non-Gaussian noise. Feature extraction was performed, and then FLO-CWD used to transform the signal time-frequency map by inputting the transformed time-frequency map into the improved CNN for the second feature extraction and classification. The recognition rate of this method reaches 95% at 4 dB. However, this method only recognizes signals of 2ASK, 2FSK, and 2PSK modulation methods and does not know the recognition rate of other modulation methods. Therefore, if we want to apply this method to actual communication systems, we need to continue research and optimization.

(4) Circulation spectrogram

A cyclic spectrum has good anti-noise performance, so it is often used to analyze signals in environments with large noise interference. The 3D graph output by the cyclic spectrum can give an intuitive impression, and the signal can be further analyzed by the cross-sectional view of the 3D graph in different directions. In order to further visualize the cyclic spectrogram of the modulated signal, this paper uses MATLAB to draw the three-dimensional cyclic spectrogram of QPSK and 4ASK and intercept the two-dimensional part when the cyclic frequency alpha is equal to 0, as shown in Figure 10.

In 2019, the author of [45] made improvements to address the poor performance of constellation maps at low signal-to-noise ratios, resulting in low recognition rates and poor adaptation phases, and proposed a modulation recognition method based on innovative CNNs and recurrent spectrograms. The author first maps the three-dimensional cyclic spectrum of the signal into two dimensions, and then takes the two-dimensional cyclic spectrum as the dataset. Then the improved CNN is used to extract the features of the cyclic spectrum. Finally, the softmax layer is used to successfully identify eight modulation modes.

The authors of [46] proposed using a deep learning algorithm to process the cyclic spectrogram of the signal to identify the secondary modulation signal. The author uses AlexNet, vgg16, vgg19, and resnet18 to recognize the two-dimensional cyclic spectrum images of seven modulated signals (BPSK, QPSK, 2FSK, bpsk-pm, qpsk-pm, 2fsk-pm, and DS-BPSK). The experimental results show that the recognition accuracy of VGG19 and ResNet18 is better, but the confusion rate of BFSK-PM and BPSK-PM is high.

(5) Amplitude Histogram

The amplitude histogram of the signal can represent the relationship between the amplitude of the signal and the number of sampling points of different amplitudes. The amplitude histograms of different modulation methods have large differences, as shown in Figure 11, which shows that the amplitude histograms of different modulation methods are all different shapes.

The authors [47], proposed a new scheme of CNN. Joint OSNR monitoring and MFI based on the signal amplitude histogram. The author uses the constant modulus algorithm (CMA) to obtain modulated signal samples after equalization to draw amplitude histograms (Ahs), and then uses the convolutional neural network a design based upon VGG to extract the features of the amplitude histograms for classification and identification. After 2500 times of training, the recognition rate of QPSK, 8-QAM, and 16QAM reached 100%, indicating that CNN can identify the modulation type of the signal according to the histogram of the signal.

The authors [48] proposed a multi-layer neural network modulation pattern recognition method based on cyclic cumulants of communication signals. The authors use the improved CNN to extract the signal features represented in the cyclic spectrograms of MPSK and MQAM, and then use the softmax layer to complete the classification. The algorithm can achieve a recognition accuracy of 92% in a signal-to-noise ratio environment of −5~5 dB.

2.

The signal sequence recognition method based on CNN.

In addition to the CNN-based signal two-dimensional image recognition method, some scholars use CNN to directly extract the features in the signal sequence and complete the classification. Therefore, this section will introduce the CNN-based signal sequence recognition method.

(1) IQ sequence

S. Hong et al. proposed a DL-based AMR algorithm to identify signals in Orthogonal Frequency Division Multiplexing (OFDM) systems [49]. The authors used convolutional neural networks to train IQ samples of OFDM signals. It can be seen through experiments that when the signal-to-noise ratio is 10dB, the correct classification probability is higher than 90%, but when the signal-to-noise ratio is lower than 10dB, the recognition accuracy drops rapidly.

In order to enable CNN to work with a small amount of data, the authors of [50] proposed a data-augmented modulation recognition method. The authors first calculate the amplitude, phase, and frequency according to the input IQ signal, and take them as the most basic signal features. Second, the phase sequence of the signals is rearranged according to the distribution of the modulated signals in the constellation diagram, so as to obtain new features. Then, the higher-order spectral information of the signal is obtained to provide new identification clues. Finally, the IQ signal, the amplitude frequency phase of the signal, the reordered IQ sequence, the reordered amplitude frequency phase, and the high-order spectrum of the signal are input into the improved CNN for classification and identification. The experimental results show that the algorithm achieves an average recognition rate of above 95%. However, the feature extraction process of this method is relatively complicated, which is not conducive to its use in a changeable communication environment. Yu. W et al. applied the DL-based AMR algorithm to multiple-input multiple-output (MIMO) systems, and in their work [51] they proposed a CNN-based zero-forcing (ZF) equalization AMR method. Among them, ZF equalization can improve the signal-to-noise ratio of the received signal under the channel state information (CSI) and enhance the accuracy of modulation identification. Therefore, the author inputs the received signal and CSI into ZF equalization, performs vectorization, and finally inputs them into CNN for classification. Through experiments, it can be seen that the recognition accuracy of the ZF equalization AMR algorithm based on CNN reaches more than 90% when the signal-to-noise ratio is—5 dB, which is better than the traditional algorithm based on ANN and higher-order cumulants. Huynh-The et al. [52] proposed a three-dimensional MIMO-OFDM Convolutional Neural Network (MONet) capable of accomplishing efficient AMR in a Multiple-Input Multiple-Output Orthogonal Frequency Division Multiplexing (MIMO-OFDM) system. The relevant underlying features within and between antennas can be extracted under multi-scale signals through the cubic convolution filter of the network. Through experimental simulation, MONet can achieve 95% recognition accuracy under the condition of 0 dB.

(2) Higher-order cumulants

Research on signal modulation identification based on higher-order cumulants was carried out as early as 1992, when the author [53] used higher-order cumulants as relevant features and used decision trees to make judgments, resulting in the identification of modulation patterns. Subsequently, many scholars began to use high-order cumulants as features for modulation classification [54,55,56] and achieved good results. After the development of deep learning, some scholars combined deep learning with high-order cumulants to study a new AMR algorithm.

Y. Wang proposed a CNN-based CO-AMR method [57]. The author used CNN to extract the high-order cumulant features of the signal in all antenna datasets and identify sub-results. All sub-results were combined, and finally, using decision rules (direct voting and weighted voting) to complete the classification. Experiments showed that when the signal-to-noise ratio is greater than 0 dB, the recognition accuracy of the algorithm can reach 100%, but at −5 dB, the recognition rate is only 82%. Therefore, it is necessary to further improve the recognition rate under a low signal-to-noise ratio.

From the above literature survey, it can be seen that because CNN is more suitable for extracting information from images, there are not many studies on using CNN to directly extract signal sequence features. Relatively speaking, the ability of RNN to extract features directly from the signal sequence is stronger than that of CNN. Therefore, this paper will introduce the modulation recognition algorithm based on RNN in the next section.

3.2. Application of the Recurrent Neural Network in Modulation Recognition

3.2.1. Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are different from typical multi-layer networks with feedforward connections. RNNs are extended by the concept of recursive connections to feedback information to the previous layer (or the same layer). Its structure is shown in Figure 12. The input of the RNN is the data of different moments. Firstly, the information will be input from the moment of the signal. The input gate decides whether the information of this moment is input into the memory neuron. The output gate decides whether the information of this moment is output. The forgetting gate decides whether the information of this moment is forgotten. If it is not forgotten, it can be transmitted to the next moment, and the process is cycled until the whole input signal is processed. According to the above characteristics, it is known that RNN is very good at processing sequence signals, such as time series, text sequences, and audio data [58]. The communication signal is a signal that changes with time, so some scholars began to use RNN to process the communication signal [59]. Therefore, this section presents the application of RNN in modulation recognition.

3.2.2. Modulation Recognition Based on Recurrent Neural Network

The automatic modulation recognition work is very dependent on the characteristics of the signal. For different types of features, the extraction and classification methods are also different. It is mentioned in the previous content that with CNN it is easier to extract the features in the two-dimensional image of the signal, but for other forms of features, the extraction and classification effect of CNN is not so good. For example, pulse repetition interval (PRI) features in radar signals are not conducive to extraction and classification using CNN, because the PRI of the signal will greatly hinder the online application of CNN [60]. Subsequently, some scholars proved that the LSTM model can extract PRI features better than the CNN model [61].

In 2020, the authors of [62], taking into account the sequence characteristics of PRI, proposed a model combining attention mechanism with GRU to identify the modulation type of radar signal. The model can improve the recognition rate in the presence of a high ratio of missing and spurious pulses. This article proposes to use a one-hot vector to represent the PRI sequence and then reduce the dimensionality of the sequence and input it into the attention-based GRU model, which can save the sequential patterns contained in the PRI sequence and finally output it through the GRU. This article proposes to use a one-hot vector to represent the PRI sequence and then reduce the dimensionality of the sequence and input it into the attention-based GRU model, which can save the sequential patterns contained in the PRI sequence and finally output it through the GRU.

Inspired by the fact that the nodes of the potential layer in the LSTM model can retain the dynamic time-domain characteristics of the information, some scholars proposed a low SNR modulation recognition method based on LSTM [63]. This work uses an LSTM network to construct a signal-to-noise ratio classifier, a denoising auto-encoder, and finally a recognition classifier. The signal-to-noise ratio classifier consists of three LSTM layers and a fully connected layer, which can divide the signal into low signal-to-noise ratio signals and high signal-to-noise ratio signals according to the set threshold. The denoising auto-encoder is composed of five bidirectional LSTM layers, which can denoise signals with a low signal-to-noise ratio. Finally, a modulation recognition structure based on LSTM is designed. The experimental results show that the modulation recognition model based on LSTM can achieve an average recognition rate of more than 90% at a signal-to-noise ratio of 0~8 dB, but the recognition rate is still low with a signal-to-noise ratio of −10 dB~−2 dB.

S. Wei et al. used a model combining a self-attention mechanism and bidirectional LSTM to identify the modulation mode of a radar signal [64]. The method can accurately identify eight types of radar modulation signals under low signal-to-noise ratio and the recognition rate is as high as 95% at −10 dB. Experiments show that the recognition accuracy of the model is better than CLDN, DRCNN, Seq-CNN, and Seq-Res networks. Qingfeng Jing et al. [65] designed an end-to-end modulation identification method based on LSTM and GRU, which can directly obtain the modulation type from the sampled signal. The experimental results show that the end-to-end modulation recognition method proposed by the author can achieve a recognition rate of 90% for each type of modulated signal. In order to solve the local dependency constraints of CNN and RNN in extracting signal features, W. Kong et al. proposed a transformer based connected sequential neural network structure (ctdnn) [66]. First, the author uses the convolution layer to map the time-domain sequence of the signal to a high-dimensional space, then uses the transformer encoder to complete the feature extraction of the signal, and finally uses the fully connected layer to complete the signal classification. Experiments show that the model can complete the classification of 10 modulated signals well.

This paper summarizes the last 5 years of RNN-based modulation identification methods in

Table 2. Summary of modulation recognition algorithms based on cyclical neural networks.

Author	Year	Input Signal	Model	Modulation Signal Type	Recognition Accuracy
D. Hong et al. [67]	2017	IQ signal	Two-layer GRU	WB-FM, AM-SSB, AM- DSB, BPSK, QPSK, 8PSK, 16QAM, 64QAM, BFSK, CPFSK, PAM4	91% (−4 dB)
Kim, H et al. [68]	2019	Amplitude phase of IQ signal	LSTM	8PSK, AM-DSB-BPSK, PFSK, GFSK, PAM4, QAM16, QAM64, QPSK, WBFM	80% (6 dB)
Wang, Yan et al. [69]	2020	IQ signal	HSNet	16QAM, 32QAM, BPSK, QPSK, 8PSK, PAM, CPFSK, GFSK, FM, SSB	86% (0 dB)
H. Yang et al. [70]	2021	Polar coordinate representation of the signal	IRS + LSTM	BPSK, QPSK, 8PSK, 16QAM, 64QAM, GFSK, CPFSK, 4PAM, WBFM, AM–DSB	90% (0 dB)
Liu et al. [71]	2021	IQ signal	DCN-BiLSTM	AM-DSB, AM-SSB, WBFM, BPSK, 8PSK, QPSK, CPFSK, GFSK, PAM4, 16QAM, 64QAM	90% (4 dB)
Zi. et al. [72]	2022	IQ signal	LSTM	8PSK, AM-DSB, AM-SSB, BPSK, CPFSK, GFSK, PAM4, QAM16, QAM64, QPSK, WBFM	90% (8 dB)
V.N.Senthil Kumaran [73]	2022	Original signal	EDL-MSC (GRU + Bi LSTM + SSAE)	8PSK, BPSK, 16QAM, 64QAM, QPSK	92% (−4 dB)
UdayaDampage [74]	2022	IQ signal	LSTM + Bi LSTM	BPSK, QPSK, 16QAM, 64QAM, 256QAM	90% (0 dB)
P. Ghasemzadeh [75]	2022	IQ signal	S-QRNN	OOK, 4ASK, 8ASK, BPSK, QPSK, 8PSK, 16PSK, 32PSK, 16APSK, 32APSK, 64APSK, 128APSK, 16QAM et	90% (5 dB)

. Since it is easier for the RNN to extract the information from the sequence signal, inputting the IQ sequence of the signal into the RNN for identification is more conducive to improving the identification accuracy.

3.3. Application of Combination Neural Network in Modulation Recognition

According to the previous description, CNN has a strong ability to process image information, and RNN has excellent performance when processing IQ signals directly. Therefore, some scholars combined CNN and RNN to form the CLDNN model [76], and CLDNN gave full play to the advantages of these two models and further improved the accuracy of modulation recognition. Since then, more and more scholars have devoted themselves to studying the application of combined neural networks in modulation recognition. In this section, we summarize the modulation recognition method based on a combined network.

T. Wang proposed an MCF network composed of the Information Cue Multi-Stream Module (SCMS) and the Visual Cue Recognition Module (VCD) [77]. The network realizes the parallel feature extraction of the IQ, AΦ, and signal constellation of the signal. The SCMS module extracts the features of the IQ signal and the AΦ signal and uses the VCD to extract the features of the signal from the constellation. Finally, the extracted features are fused and classified. The author compared the MCF network with CM+CNN, SCNN, LSTM, and CLDNN network models during the experiment. The results show that the MCF network has better recognition accuracy. When the signal-to-noise ratio is greater than 0 dB, the recognition accuracy is close to 100%.

The authors of [78] proposed a new deep learning-based attention collaboration framework, which includes CNN, RNN, and GAN networks. Among them, ACGAN is used to expand the original data, and then CNN and RNN are used to extract the spatial and temporal characteristics of the signal, respectively. Finally, the spatial and temporal properties of the signal are fused using GAMP, and the classification is done with a fully connected layer. Experiments show that the attention cooperation framework proposed in this paper can recognize 11 kinds of modulated signals with an accuracy of more than 90% at 0 dB. The recognition accuracy is better than that of VGG, RNN-GRU, GoogleNet, and CLDNN. The authors of [79] proposed a CGDNet network structure composed of shallow convolutional networks, GRU, and DNN networks in their work. Among them, a shallow convolutional network and GRU are used to extract the features of IQ sequence signals, and DNN is used to complete the classification task. In the experiments, the author uses the RadioML2016.10a and RadioML2016.10b datasets to verify the performance of the CGDNet structure. When identifying the RadioML2016.10a modulation dataset, it can achieve a recognition accuracy of more than 90% at 0 dB; when recognizing the RadioML2016.10b modulation dataset, it can achieve a recognition accuracy of more than 90% at 18 dB.

The authors of [80] proposed a modulation recognition algorithm based on convolutional long and short-term deep neural networks (CLDNN). The CLDNN network structure consists of four layers of convolution layers, four layers of pooling layers, two layers of LSTM layers, and two layers of fully connected layers. Experiments show that the average recognition rate of the CLDNN model for 11 kinds of modulation signals reaches 90.8%, and it can recognize both QAM16 and QAM64 better.

In recent years, modulation identification methods based on combinatorial networks have become more popular, and many scholars have devoted themselves to researching more reliable combinatorial networks for AMR. Combinatorial network-based modulation identification methods over recent years are summarized in

Table 3. Summary of modulation recognition algorithms based on combinatorial neural networks.

Author	Year	Input Signal	Model	Modulation Signal Type	Recognition Accuracy
B. Tang et al. [81]	2018	Outline constellation map	CNN + GAN	4ASK, BPSK, OQPSK, 8PSK, 16QAM, 32QAM, 64QAM	100% (−2 dB)
F. Liu et al. [82]	2020	IQ sequence, cyclic spectrum	CNN + GRU	2PSK, 2ASK, 2FSK, 4PSK, 4ASK, 16QAM, 64QAM	100% (0 dB)
J. Xu et al. [83]	2021	IQ sequence	MCLDNN(CNN + LSTM + FC)	WBFM, AM-DSB, AM-SSB, BPSK, CPFSK, GFSK, 4-PAM, 16-QAM, 64-QAM, QPSK, and 8PSK	90% (0 dB)
Jiang K et al. [84]	2021	IQ sequence	CNN + Bi LSTM + Attention	2ASK, 4ASK, BPSK, QPSK, 8PSK, 64QAM	93.14% (10 dB)
Duan, Q et al. [85]	2021	Original signal	MCBL (CNN + Bi LSTM + Attention)	8PSK, AM-DSB, BPSK, CPFSK, GFSK, PAM4, QAM16, QAM64, QPSK, WBFM	93% (0 dB)
Z. Liang et al. [86]	2021	Time-frequency diagram	ResNeXt WSL(ResNeXt + Attention)	8PSK, BPSK, AM-DSB, QPSK, QAM16, QAM64, CPFSK, GFSK, PAM4, and WBFM,	90% (0 dB)
S. Chang et al. [87]	2022	Sampled signal	MLDNN(CNN + Bi GRU + SAFN)	AM-DSB, AM-SSB, WBFM. 8PSK, BPSK, CPFSK, GFSK, 4PAM, 16QAM, 64QAM, QPSK	84% (0 dB)
W. Zhang et al. [88]	2022	Sampled signal	GRU + CNN	BPSK, QPSK, BFSK, QFSK, 16QAM, 64QAM, OFDM	99.45% (true channel)
Zou B et al. [89]	2022	IQ sequence	ASCLDNN (Attention + CLDNN)	BPSK, 8PSK, CPFSK, GFSK, PAM4, PAM16, QAM64, QPSK, AM-DSB, AM-SSB, WBFM	90% (0 dB)
Bai J et al. [90]	2022	IQ sequences and GAF images	DMFF-CNN (Complex Value Network + ResNet50)	2FSK, AM, DSB, FM, OFDM, QAM16, QPSK, SSB	91% (−10 dB)

.

3.4. Applications of Other Neural Networks in Modulation Recognition

In the previous three subsections, we introduced the application of classical deep learning networks and combinatorial neural networks in modulation recognition, but in fact, there is much modulation recognition research based on other deep neural networks, for example, Multilayer Perceptron (MLP), Deep Neural Network (DNN), Probabilistic Neural Network (PNN), Deep Belief Network (DBN), and Back Propagation neural network (BP). Therefore, in this section, this paper will introduce the applications of MLP, DNN, PNN, DBN, and BP networks in AMR.

The authors of [91] studied an efficient modulation recognition model, which includes three modules, namely feature extraction, feature optimization, and classifier design. The model uses Principal Component Analysis (PCA) to optimize features in the feature extraction module and uses MLP and Radial Basis Function (RBF) as classifiers, respectively. Experiments show that the recognition accuracy rate of the algorithm based on PCA-MLP is always above 95% in the range of signal-to-noise ratio of −10~10 dB, which has better stability than the algorithm based on PCA-RBF.

The authors of [92], proposed a new method that can quickly complete modulation recognition. The authors use the sixth-order cumulant of the signal as a feature, which is then fed into a DNN to complete the recognition. The entire recognition model can complete online and offline recognition and has high practicability.

Han H et al. designed a PNN-based modulation recognition algorithm in [93]. The authors fused the temporal instantaneous characteristics of the signal with the higher-order cumulants. Finally, PNN is used to complete modulation recognition according to the fused features. Experiments show that the algorithm can achieve a recognition accuracy of 99.8% at 0 dB.

The deep belief network is a typical unsupervised learning network, which can use the stacked neural network to extract features from the received signal and then use a traditional classifier to determine the modulation type [94]. The authors of [95], proposed a spatial laser pass pulse position modulation and demodulation system based on the cascade of DBN and SVM, which can realize modulation identification and demodulation. The DBN network is used to identify the modulation type, and the SVM is used to classify the demodulation results. Experiments show that the DBN-SVM demodulation system can effectively alleviate the phenomenon of channel fading, and it is close to the maximum likelihood detection method in atmospheric turbulence, indicating that the performance of the algorithm is very good. The authors of [96] combined multi-layer DBN and BP networks to design an AMR classifier, that can detect eight kinds of radar modulated signals (CW, PSK, DPSK, and FSK when the signal-to-noise ratio is greater than 10 dB. MP, LFM, and NLFM with a recognition probability of 100%), reflecting the excellent recognition performance of deep learning.

The authors of [97] proposed a BP network model (BP) based on a bird swarm algorithm. First, the bird swarm algorithm was optimized for the BP network, and then the MQAM signal was identified. Experiments show that the recognition accuracy of the algorithm reaches more than 98% and has good classification. Z. Fang et al. used four BP networks to form a modulation recognition classifier [98], which was able to achieve a recognition rate of more than 90% at a signal-to-noise ratio of 11dB. However, the recognition accuracy of this classifier under low SNR is not known, so further research is needed.

In this section, we briefly introduce the applications of MLP, DNN, PNN, DBN, and BP networks in modulation recognition. It can be seen that MLP can be jointly optimized with principal component analysis. DNN, PNN, and BP networks are better at processing statistical features such as instantaneous features and high-order cumulants of signals. The combination of DBN and a traditional classifier can achieve a better recognition rate.

After the introduction of this chapter, we have learned that convolutional neural networks, recurrent neural networks, and various combined neural networks can achieve satisfactory recognition results after training with a large amount of labeled data. However, in some fields, such as spectrum management and national defense information security, a large number of labeled signal data cannot be obtained for neural network training, resulting in the degradation of recognition performance. Therefore, improving the recognition accuracy in a small sample environment is a great challenge. With the continuous development of deep learning technology, some scholars have begun to solve the above problems. In the following section, this paper will introduce the existing modulation recognition algorithm based on deep learning in a small sample environment.

4. Modulation Recognition Based on Deep Learning in Small Sample Environment

In order to overcome the problem of poor modulation recognition accuracy in a small sample environment, some scholars have begun to focus on research on modulation recognition algorithms based on deep learning in a small sample environment. There are two main methods currently being studied. One is to perform data enhancement on the modulated signal to generate enough data and then use the neural network for training and the second is to combine transfer learning with deep learning techniques and apply excellent models in other fields that deal with small samples of data to the field of modulation recognition. Therefore, we will introduce the modulation recognition algorithm based on deep learning in the case of small samples in this section, as shown in Figure 13.

4.1. The Small Sample AMR Algorithm Based on Signal Data Enhancement

Data enhancement is the most intuitive method to solve the low recognition rate in a small sample environment. The current signal data enhancement methods can be divided into two types. One is to use a confrontational generative network (GAN) to generate virtual data, and the other is to use a semi-supervised learning method to jointly train labeled and unlabeled data to ultimately improve the recognition rate. We will introduce two AMR methods based on signal data augmentation.

4.1.1. Generate Dummy Data

When the number of labeled samples is small, generating dummy data to expand the dataset is an important method to deal with the problem of small-sample modulation recognition.

In 2021, Yu Haoyang et al. proposed the enhanced depth convolution generation countermeasure network (SDCGAN) to expand radar signals [99]. The convolutional neural network is used for recognition and classification. The experimental results show that when the signal-to-noise ratio is 8 dB, only 40 samples can make the average recognition rate reach more than 99%.

The authors of [100] introduced a data augmentation method in the small sample case as follows: (i) use GAN to generate pseudo data similar to the original signal, and then use the image quality assessment method PSNR to filter out high-quality pseudo signals, (ii) invert the original signal and the fake signal to further increase the diversity of samples, and (iii) input the enhanced dataset into the designed CNN network to complete the classification and recognition. This method can achieve a recognition rate of more than 80% when the number of signal samples is 10 and a recognition rate of 90% when the number of samples is close to 400.

4.1.2. Data Augmentation Method Based on Semi-Supervised Learning

Semi-supervised learning can jointly optimize unlabeled samples and labeled samples, improve the performance of the model, and improve the recognition accuracy. Therefore, semi-supervised learning is often used to solve the modulation identification problem under few-shot conditions [101]. Zhou et al. designed a semi-supervised learning framework based on Generative Adversarial Networks (GAN) that can directly process IQ signal data and make full use of unlabeled samples to achieve end-to-end accurate classification of a small number of electromagnetic signals [102].

4.2. Small-Sample AMR Algorithm Combining Deep Learning and Transfer Learning

Transfer learning is good at applying learned knowledge or models to other domains to solve similar problems in different domains [103]. At present, there are neural networks designed for small sample problems in the fields of machine vision and handwritten digit recognition. Therefore, some scholars have migrated deep neural networks in these fields to the field of modulation recognition to solve the problem of poor modulation recognition accuracy in a small sample environment. Some scholars also transfer the parameters trained by other networks to the neural network used for modulation recognition as the initial parameters, to improve the recognition accuracy in a small sample environment. In this section, this paper will briefly describe the small-sample AMR algorithm combined with deep learning and transfer learning.

In order to solve the problem of small samples, Li et al. introduced the CapsNet network in the field of handwritten digit recognition into the field of AMR in their work [104]. Experiments were carried out using 5% of the signal data of the modulated signal dataset RML2016.04c. The results show that when the signal-to-noise ratio is greater than 2 dB, the classification accuracy of AMR-CapsNet is greater than 80%. When experimenting with 3% of the signal data of the dataset and the signal-to-noise ratio is greater than 4 dB, the recognition accuracy of the AMR-CapsNet network exceeds 80%, and the average recognition rate is higher than that of CNN. The authors of [105] proposed an adversarial transfer learning architecture (ATLA), which utilizes adversarial training to reduce the differences between data distributions and achieve transfer learning. Experiments show that the ATLA model can achieve a recognition accuracy of 80%, indicating that the algorithm is superior to the existing parameter transfer methods in improving the migration ability and expanding the tolerance of distribution differences.

The authors of [106] proposed a new convolutional neural network based on transfer learning. The authors take the radar signals of three types of intrapulse modulation as the source domain and use the CNN to train the signals in the source domain to obtain relevant parameters; then transfers the trained parameters and models to the target domain with 450 radar signals for training. When the SNR is greater than −1, 90% classification accuracy can be achieved. The authors of [107] proposed a modulation recognition method based on data transfer learning. The authors use the coefficient autoencoder to learn the noise signal in water, thus constructing the received signal dataset and using the constructed dataset to train the neural network. Finally, the trained network is used to identify the modulation categories of real underwater acoustic communication signals. The authors of [108] proposed a new modulation recognition method for OFDM-VLC systems, firstly using a large number of ImagNet images to pre-train AlexNet and GoogleNet, then using the weights in the pre-training stage to retrain the constellation diagram of the small sample signal, and finally completing the modulation identification of the signal.

In this section, we introduced the modulation recognition problem in the small sample environment, mainly using data augmentation and transfer learning methods to solve it. However, the data enhancement method cannot guarantee the quality of the data, which may lead to a decrease in the recognition accuracy. The combination of transfer learning and deep learning can transfer models and parameters from other fields to modulation recognition, but there are fewer models specifically for modulation recognition, and generalization is not strong enough, so how to design a small sample recognition model in the field of modulation recognition is still an important research topic in the future.

5. Hardware Implementation of Modulation Recognition Algorithm Based on Deep Learning

In the previous introduction, we learned that the modulation identifiers based on deep learning are all completed by simulation. However, if you want to better apply the modulation identifiers based on deep learning in actual communication systems, you need to use hardware circuits to implement related algorithms. Through literature review, this paper understands that the use of hardware circuits to implement DL-based AMR algorithms can be roughly divided into three processes, which can be represented by Figure 14.

According to Figure 14, if you want to use a hardware circuit to implement a modulation recognizer based on deep learning, you first need to use a computer programming language, such as Python, C, or MATLAB, etc., to design a deep neural network, and then use simulation software to train the neural network to optimal. Then use hardware description language, such as VHDL, Verilog, etc., to describe the trained modulation recognizer, and use hardware language simulation software, such as ModelSim, etc., to simulate. Next, download the hardware description language to the hardware circuit for debugging, and finally realize the modulation and identification of the signal. At present, the commonly used hardware platforms for signal modulation and identification algorithms include central processing unit (CPU) + image processing chip (GPU), programmable array logic (FPGA), etc., of which FPGA can achieve acceleration and can also simulate parallel operations of CPU and GPU. It has powerful computing power and flexibility.

Using hardware circuits to implement deep learning-based modulation identifiers has important research significance in practical engineering, attracting many scholars to conduct research. Current research includes results by the authors of [109] who proposed an FPGA-based convolutional neural network modulation recognition method. The authors first designed a convolutional neural network in TensorFlow, then used VHDL language to describe the CNN and download it to the FPGA to achieve accelerated modulation recognition. It has been proved by experiments that the recognition accuracy of FPGA reaches more than 80% at 4 dB, which is close to the recognition accuracy of a CPU, but the recognition speed of FPGA is 10 times that of a CPU, which greatly improves the recognition speed. The authors of [110] used FPGA to implement AMR algorithm based on stacked convolutional autoencoders (CAE). In this work, a deep neural network needs to be trained on a GPU, and then the trained neural network is transformed into an FPGA configuration file. Finally, the configuration file is transferred to the FPGA SDR platform to realize the function of AMR. In the final experiment, it is shown that the average score accuracy on FPGA is 70%~80%. Research by the authors of [111], different from the above two works, proposed a real-time modulation recognition system based on software-defined radio and multi-hop residual neural network (MRNN). The authors use RadioML2018.01A to optimize the performance of MRNN and then embed the neural network into the SDR platform composed of GNU radio and USRP radio to achieve real-time modulation recognition. Among them, the USRP model selected by the authors is the USRP B210. The final experiment shows that the system can complete the real-time recognition of modulated signals. The average recognition time is 4.1 ms, and the average recognition accuracy is greater than 90%. The authors of [112] proposed a classification convolutional neural network model for modulation recognition and implemented the proposed classification convolutional neural network on FPGA. The experimental results show that the network is well implemented on the FPGA. When the signal-to-noise ratio is greater than 5 dB, the recognition accuracy of the radioml2016.10a dataset is greater than 80%. It can be seen that it is feasible to use a hardware development board such as FPGA to implement a modulation recognizer composed of a deep neural network.

In recent years, FPGA has become an important hardware platform for realizing modulation recognizers composed of deep neural networks [113]. It brings hope to the hardware implementation of the modulation recognition algorithm based on deep learning. However, since the hardware resources of FPGA are limited, the size, type, and number of layers of the neural network are important factors that affect the performance of hardware recognition. Some studies have proved that the FPGA is the preferred platform for implementing quantitative CNN [114] and has good classification accuracy in the field of modulation recognition, so the modulation recognizer composed of CNN can be the first choice to complete the hardware implementation. On the other hand, due to the limited hardware resources of FPGA, when designing CNN, it is necessary to consider the depth of the network and the size of the convolutional layer and try to design a lightweight neural network [115].

6. Research Challenges and Future Directions

6.1. Further Research on DL-Based AMR Algorithms in Visible Light Communication Systems

It can be seen from the above introduction that most of the existing deep learning-based modulation identification algorithms are proposed for wireless communication, and a few modulation identification algorithms are proposed for visible light communication. Visible light communication is a high-speed communication technology that does not require a licensed frequency band and will play an important role in 6G [116]. Therefore, studying the modulation and identification technology of visible light communication is also an important research direction in the future, with good prospects.

6.2. Improving Modulation Recognition Accuracy in Small Sample Scenarios

The recognition rate of modulation recognition algorithms based on deep learning is not high in the case of small samples, although some scholars have initially solved this problem through data enhancement and transfer learning [99]. However, the data quality cannot be guaranteed, the generalization is not strong, and it is difficult to solve the identification problem in the field of radar communication. Therefore, improving the accuracy of AMR models in small sample scenarios is still an important problem to be solved and has great research value.

6.3. Strengthening the Research on Adaptive Modulation Identification

At present, most researchers only study the modulation identification module, and there are few studies on the joint design of the entire modulation and demodulation system. However, with the development of intelligent communication, it is necessary to jointly study the modulation identification and demodulation of the physical layer and to design an adaptive modulation identification and demodulation system. It is also an important development direction.

6.4. Designing more Lightweight Modulation Recognition Networks

Due to the limited resources of current hardware circuits, it is difficult to deal with huge neural networks, so most of the modulation recognition algorithms based on deep learning are still focused on software simulation, and there are not many studies on applying these networks to hardware circuits. However, in order to apply the excellent deep learning-based modulation recognition algorithm to the actual communication system, it is necessary to solve the difficulty that the neural network is difficult to realize on the hardware circuit. Designing a lightweight neural network is a better way to solve this problem [115]. Therefore, in future research, designing a neural network with lighter weight and higher recognition efficiency has a better development prospect.

6.5. Design the Modulation Identification Method of OFDM Signal

It can be seen from the above introduction that most of the current deep learning-based modulation identification algorithms are mainly aimed at single-carrier systems, and there is relatively little research on modulation identification algorithms for OFDM signals. Orthogonal Frequency Division Multiplexing (OFDM) is a very important transmission method in multi-carrier modulation. It is widely used in communication systems due to its good anti-interference performance and is an important method to complete large data transmissions in the future. Therefore, it is of great significance to study the modulation and identification methods of OFDM signals. However, due to the diversification of modulation types, the channels of multi-carrier systems are very complex, resulting in the identification of sub-carrier modulation of OFDM signals and the identification between OFDM signals and single-carrier signals becoming very difficult [117]. This brings great challenges to future wireless communication. Therefore, it is very necessary to strengthen the research on modulation and identification of OFDM signals.

7. Summary

As a key technology of the physical layer, automatic modulation recognition has important research value and application prospects. With the complexity of communication scenarios, AMR technology will face more severe challenges, so researchers need to continuously develop new AMR methods to solve new problems. This paper expounds the research situation of advanced AMR algorithms based on deep learning in the past five years and organizes some of the references mentioned in this paper into

Table 4. Technical comparison table of some references.

Reference A	Author	Year	Input Signal Type	Model	Recognition Accuracy	Advanced Techniques Mentioned in Reference A	Author	Year	Input Signal Type	Model	Recognition Accuracy
[36]	Pengshengliang et al.	2019	Constellation Chart	GoogleNet, AlexNet	90% (3 dB)	[118]	Fen Wang	2019	Constellation Chart	GCP-DBN	95% (0 dB)
[40]	Zhuolun LI et al.	2019	Eye Chart	Improved ResNet	>95% (5 dB)	[36]	Pengshengliang	2019	Constellation Chart	Goog-le-Net, AlexNet	90% (3 dB)
[41]	Xiong Zha et al.	2021	Constellation and eye charts	Designed multi-input CNN	92% (3 dB)	[36]	Pengshengliang	2019	Constellation Chart	Goog-le-Net, AlexNet	90% (3 dB
[42]	Yongjiang Mao et.al	2021	Time and frequency diagram	Sep-ResNet	93.44% (−10 dB)	[119]	Zhiyu Qu	2020	Time and frequency diagram	CNN	96.1% (−6 dB)
[43]	Daying Quan et al.	2021	Time and frequency diagram	Dual-channel CNN	97% (−6)	[120]	Jian Wan	2019	Time and frequency diagram	CNN+TPOT	94.42% (−4 dB)
[45]	Gao Jingpeng et al.	2019	Circulation spectrogram	CNN	90.48% (−2 dB)	[121]	Xiao Yan	2018	Constellation and eye charts	Modulation classifier using the minimum angle search	>90% (15 dB)
[47]	Hongjing Lv et al.	2021	Magnitude histogram	CNN	100%	[122]	Zhiquan Wan	2019	Magnitude histogram	MTL-ANN	100%
[49]	Sheng Hong et al.	2019	IQ Signal	CNN	98% (8 dB)	[92]	Wenwu Xie	2019	High-order cumulative volume	DNN	99% (−5 dB)
[73]	V.N.Senthil Kumaran et al.	2022	IQ Signal	EDL-MSC (GRU + BI LSTM+SSAE)	92% (−4 dB)	[69]	Duona Zhang	2020	IQ Signal	HsNet	86% (0 dB)
[81]	Pejman Ghasemzadeh et al.	2022	IQ Signal	S-QRNN	90% (5 dB)	[123]	Sai Huang	2020	Signal cyclic correntropy vector (CCV)	LSTM+DenseNet(LSMD)	90% (−6 dB)
[80]	Tuo Wang et al.	2021	IQ Signal, Constellation Chart	SCMS (CNN + IndRNN) +VCD	100% (0 dB)	[124]	Yuan Zeng	2019	Spectrum diagram	SCNN, SCNN2	80% (0 dB)
[79]	Judith Nkechinyere Njokuet al.	2021	IQ Signal	CGDNet	90% (18 dB)	[123]	Sai Huang	2020	Signal cyclic correntropy vector (CCV)	LSTM + DenseNet(LSMD)	90% (−6 dB)
[82]	Fugang Liu et al.	2022	Circulation spectrogram + IQ Signal	CNN + GRU	100% (0 dB)	[119]	Zhiyu Qu	2020	Time frequency diagram	CNN	96.1% (−6 dB)
[83]	Jialang Xu et al.	2020	IQ Signal	MCLDNN (CNN+LSTM + FC)	90% (0 dB)	[125]	Erma Perenda	2019	IQ Signal	1D-CNN	Close to 90% (0 dB)
[84]	Kaiyuan Jiang et al.	2021	IQ Signal	CNN + Bi LSTM + Attention	93.14% (10 dB)	[71]	Kai Liu	2021	IQ Signal	DCN-Bi-LSTM	90% (4 dB)
[88]	W.Zhang et al.	2022	Sampling signal	GRU + CNN	99.45% (Real channel)	[126]	Zufan Zhang	2020	IQ Signal	CNN-LSTM	83% (−2 dB)
[100]	HuajiZHOU et al.	2022	IQ Signal	GAN + CNN	90% (400 signal samples)	[127]	Xiaohui Yao	2019	Sampling signal	GAN	92% (12,000 signal samples)
[104]	Lixin Li et al.	2021	IQ Signal	AMR-CapsNet	80% (8103 signal samples)	[128]	Sabour, S.	2017	Handwritten digital picture	CapsNet	95.7%

. Column 1 in Table 4 is some of the references mentioned in this paper, and column 7 in Table 4 is the state-of-the-art technology mentioned in the references in column 1. We have sorted out these contents and formed a comparison table, so that readers can intuitively understand the technical starting points of the references mentioned in this article and hope to provide some references for future research.