Efficient Jamming Policy Generation Method Based on Multi-Timescale Ensemble Q-Learning

Abstract

With the advancement of radar technology toward multifunctionality and cognitive capabilities, traditional radar countermeasures are no longer sufficient to meet the demands of countering the advanced multifunctional radar (MFR) systems. Rapid and accurate generation of the optimal jamming strategy is one of the key technologies for efficiently completing radar countermeasures. To enhance the efficiency and accuracy of jamming policy generation, an efficient jamming policy generation method based on multi-timescale ensemble Q-learning (MTEQL) is proposed in this paper. First, the task of generating jamming strategies is framed as a Markov decision process (MDP) by constructing a countermeasure scenario between the jammer and radar, while analyzing the principle radar operation mode transitions. Then, multiple structure-dependent Markov environments are created based on the real-world adversarial interactions between jammers and radars. Q-learning algorithms are executed concurrently in these environments, and their results are merged through an adaptive weighting mechanism that utilizes the Jensen–Shannon divergence (JSD). Ultimately, a low-complexity and near-optimal jamming policy is derived. Simulation results indicate that the proposed method has superior jamming policy generation performance compared with the Q-learning algorithm, in terms of the short jamming decision-making time and low average strategy error rate.

1. Introduction

With the rapid development of information technology, the electromagnetic environment of modern battlefields has become increasingly complex, and the role of electronic warfare has become more critical [1]. As a crucial part of electronic countermeasures, the goal of interference decision-making is to generate the optimal interference strategy in real-time based on threat perception and judgment of radar signals, thereby effectively completing the task of interfering with radar and reducing the threat of enemy radar [2]. Driven by advanced electronic technologies, such as digital phased array systems, radar is gradually evolving toward multifunctionality. For example, it has the ability to quickly change the waveform between pulses and can adaptively adjust the working state according to the task and external environment [3,4,5,6,7].

Currently, multifunctional phased array radar (MFPAR) has advantages, such as fast response speed, flexible beam control, strong anti-interference ability, and can achieve various functions, such as search, tracking, recognition, and guidance through multi-dimensional parameter regulation. Due to the inability to directly obtain changes in the radar operating status, it poses great challenges to the jamming decision-making of the jammer. If the jammer lacks sufficient prior knowledge of the radar state, it will be difficult to establish the optimal correspondence between the radar state and existing jamming styles, and thus impossible to form an optimal jamming strategy for different radar states. Therefore, it is of great significance to study efficient jamming strategy generation methods for MFPAR with complex systems.

1.1. Cognitive Radar Countermeasure Process

Electronic warfare (EW) is crucial in modern warfare for gaining electromagnetic superiority, which aims to control the electromagnetic spectrum by attacking enemy electronic systems while protecting friendly ones [2,8,9,10]. Traditional adaptive electronic warfare employs pre-set or environmentally adaptive countermeasures to respond to enemy electronic systems, which relies heavily on human expertise without the capability for reasoning and learning. The emergence of cognitive electronic warfare (CEW) can improve the rapid response capability and reliability of jammers [11,12,13,14]. CEW perceives enemy signals through prior knowledge and interactive learning, and then uses artificial intelligence algorithms to quickly generate optimal jamming decisions [15,16,17,18,19]. Figure 1 illustrates the cognitive radar countermeasure process, where cognitive jamming, cognitive reconnaissance, cognitive evaluation, and the target environment form a closed loop [20,21].

Combining cognitive theory with jamming decision-making concepts to form a cognitive-based jamming decision system will significantly enhance the accuracy and real-time capabilities of jamming and meet the need for effective jamming against advanced MFR. Therefore, the jamming decision system must be capable of real-time selection of the optimal jamming strategy based on the reconnaissance-obtained parameters of the radar. It should utilize an evaluation module to assess jamming effectiveness in real-time and adjust the jamming strategy accordingly. Thereby, it forms a closed-loop system from jamming application to real-time feedback, which aligns with the real-time and dynamic requirements of CEW.

1.2. Traditional Jamming Strategy Generation Methods

Researchers have extensively studied jamming decision-making from three aspects: template matching, game theory, and reasoning algorithms [22,23]. Nonetheless, these techniques significantly depend on extensive pre-existing knowledge, which poses a strong challenge when trying to adjust to the swiftly evolving and intricate electromagnetic environment. The template-matching method compares radar signal samples with those in a template library and selects the jamming pattern corresponding to the most similar sample. This approach is mainly suitable for radars with fixed and unchanging parameters, and the quality of the template library directly affects the accuracy of jamming decisions. However, it lacks timeliness and often lags behind the rapidly changing battlefield environment [24]. The game theory method treats the radar and jammer as two players in a game, where the strategy set and payoff matrices are defined. The jammer picks a jamming pattern from the available strategy set and then obtains a payoff based on the payoff matrix. This method can optimize the selection of jamming patterns to a certain extent [25,26]. However, the game-theory-based jamming decision method is highly dependent on the payoff matrix, which also requires extensive expert experience for its construction and cannot guarantee real-time jamming of the radar. The reasoning algorithms examine past events to identify causal links, thereby estimating the likelihood of future events occurring under specific conditions and supporting jamming decision-making [26,27]. However, this method also lacks real-time capability, and the easily accumulated errors will affect the accuracy of jamming decisions.

The three aforementioned methods can play a role in jamming decision-making to some extent. However, it is challenging to obtain prior information for new system radars, such as MFR, which makes it difficult for accurate jamming in actual combat. As shown in Figure 2, the prior knowledge refers to the information about the electromagnetic environment and target radar, which is already known before making jamming decisions. This includes the operation parameters of the target radar, environmental information, friendly resources and capabilities, historical data, and experience. This prior knowledge influences the jamming template library, the payoff matrix, and the calculation of event probabilities after taking different jamming measures, ultimately affecting the generation of jamming strategies.

1.3. Jamming Strategy Generation Methods Based on Reinforcement Learning

To develop a strategy for jamming that effectively counters MFRs, it is meaningful to explore an efficient jamming decision-making approach that does not rely on prior data. The advent of reinforcement learning addresses the shortcomings of traditional jamming decision-making methods that overly depend on prior data [28]. Popular reinforcement learning algorithms encompass methods such as temporal difference (TD), policy gradient, Q-learning, and SARSA [29,30,31]. Additionally, researchers have proposed improvements to these algorithms to address issues, such as high noise sensitivity and large estimation biases [32]. Llorente et al. provided a comprehensive overview of the application of Monte Carlo methods in noisy and costly densities and applied them to reinforcement learning and approximate Bayesian computation (ABC) [33]. Liu, W. et al. utilized the Q-ensemble approach to derive optimal policies from static datasets, employing techniques such as large-batch punishment and binary classification networks to differentiate in-distribution from out-of-distribution data, thereby reducing the Q-ensemble size and enhancing the performance and training speed [34]. Essentially, reinforcement learning entails an interaction between the agent and its environment, wherein the agent adapts its strategy according to the rewards received from its actions, aiming to maximize future rewards.

The concept of cognition was first introduced into the field of radar electronic warfare in [35], which applies the Q-learning algorithm to the cognitive radar countermeasure process. The authors of [36] developed a cognitive jamming decision-making system, concentrating on the creation of jamming strategies and the evaluation of their effectiveness. The authors of [37] improved the jamming decision problem model by modeling the problem as a Markov decision process with rewards. The authors of [38,39,40,41] improved the Q-learning algorithm significantly by enhancing the decision-making efficiency and reducing the training time. To handle larger radar state spaces, the authors of [42,43,44] employed deep reinforcement learning techniques to address the jamming decision problem, which significantly enhanced the efficiency and accuracy of jamming decisions within high-dimensional state spaces. The authors of [45] introduced a method for generating jamming strategies using heuristic planning combined with reinforcement learning. This method constructs a heuristic reward function using a potential function, which improves sample utilization while reducing sample complexity and shortening the jamming decision time. The authors of [46,47,48,49] began to consider various radar countermeasure scenarios, which addressed different issues, such as optimal jamming parameter selection and multi-agent jamming systems.

To address issues such as low jamming efficiency and accuracy against advanced MFRs, an efficient jamming policy generation method based on multi-timescale ensemble Q-learning is proposed in this paper. The main contributions of this paper are summarized as follows:

• A realistic radar countermeasure scenario was established, in which an aircraft carries a jammer for self-protection to avoid being locked by radar during mission execution. The transition modes of the MFR operating modes were analyzed, and mode-switching rules were designed for scenarios both with and without jamming.
• The problem of generating jamming strategies was modeled as a MDP, with the goal of identifying the optimal jamming strategy in the dynamic interaction between the MFR and the jammer.
• An efficient jamming strategy generation method based on MTEQL was proposed. First, the adversarial process between the radar and the jammer was sampled to construct multiple structurally related Markov environments. The Q-learning algorithm was then run in parallel across these environments. A JSD-based adaptive weighting mechanism was employed to combine the multiple output Q-values, ultimately obtaining the optimal jamming strategy.
• Numerical simulations and experimental investigations were carried out to demonstrate the effectiveness of the proposed method. Results highlighted that the proposed method achieved faster jamming strategy generation and higher decision accuracy.

The primary structure of this paper is outlined as follows: Section 2 designs the radar countermeasure scenario and models the jamming strategy problem, Section 3 proposes the jamming strategy generation algorithm, Section 4 presents the simulation experiments, and Section 5 provides the conclusions.

2. Radar Countermeasure Scenario Design and Problem Modeling

2.1. Radar Countermeasure Scenario Design

This paper discusses a scenario in which an aircraft carries a jammer for self-protection, releasing jamming signals to avoid being locked by a MFR during missions. The focus is on the one-on-one confrontation between the jammer and the radar, as illustrated in Figure 3. The MFR on the fighter jet tracks the target aircraft, which is equipped with a jammer capable of intercepting radar signals in real time for reconnaissance. The jammer evaluates the effectiveness of its jamming based on changes in the intercepted radar signals. Through continuous confrontation with the radar, the jammer learns the radar’s mode transition patterns and jamming strategies. It then selects the optimal jamming method based on these strategies and releases jamming signals accordingly. The goal of the jammer is to continuously reduce the threat level posed by the radar’s operational modes, thereby protecting the target from being locked by the radar.

At the present time step, the jammer detects that the MFR is in state $s_t$. The jammer obtains a reward, $r_t$, based on the transformation of the radar state. After updating its jamming strategy, it employs jamming pattern $a_t$ to jam with the MFR. As a result of the jamming, the radar state transitions to $s_{t+1}$. The jammer detects the change in the radar state and obtains another reward, $r_{t+1}$. The jammer then updates its policy and makes a new decision, iterating this process continuously. Based on the above analysis, the task of generating jamming policies is essentially a sequential decision-making problem, which can be modeled using the MDP. The MDP is described by the tuple $\left\{S,A,P,R\right\}$, in which S, A, P, and R represent the radar states, jamming patterns, radar state transition probabilities, and immediate rewards, respectively.

2.2. Radar Operation Mode Modeling

To effectively jam the MFR, it is crucial to accurately model its operational modes. Such modeling not only enhances the understanding of the radar’s performance across different modes but also provides a theoretical foundation for developing effective jamming strategies.

The state space, denoted as $S=\left[s_1,s_2,\dots ,s_N\right]$, comprises various operation modes of the multi-function radar (MFR), totaling N modes. The MFR can operate in several modes, including velocity search (VS), range while search (RWS), track while scan (TWS), track and scan (TAS), multi-target track (MTT), and single-target track (STT). These operation modes can be flexibly switched. This paper designed the switching rule of the MFR’s operation modes in the absence of jamming, based on changes in the target’s threat level or distance, as shown in Figure 4.

When a fighter jet equipped with a MFR is at a considerable distance from the target and has not yet detected it, the radar initially operates in VS mode, conducting a comprehensive search of the mission area. At this stage, the threat level is minimal, as no target has been detected. Upon detecting a target, the MFR switches to RWS mode to continue tracking the target, taking into account resource constraints. As the target approaches, the MFR enters the coarse-tracking phase, with the fighter jet selecting either TWS or TAS mode based on real-time conditions. When the fighter jet is very close to the target and the threat level is high, the radar transitions to STT or MTT mode, entering a continuous fine-tracking phase. This phase provides robust support for subsequent precise weapon strikes. Consequently, the threat level is highest at this stage, and jamming efforts must focus on preventing the MFR from operating in this mode.

The action space consists of various jamming patterns, denoted as $A=\left[a_1,a_2,\cdots ,a_M\right]$, with a total of M patterns. These include narrowband noise jamming (NNJ), wideband noise jamming (WNJ), swept-frequency noise jamming (SFNJ), comb spectrum jamming (CSJ), dense false-target jamming (DFTJ), slice repeater jamming (SRJ), range gate pull-off jamming (RGPJ), velocity gate pull-off jamming (VGPJ), and doppler noise jamming (DNJ). These patterns are primarily categorized into three types: blanket jamming, deception jamming, and combined blanket–deception jamming. Each type exerts distinct effects on the radar, and successful jamming can reduce the threat level of the MFR’s operational modes.

The state transition probability, $p\left(s_{t+1}|s_t,a\right)$, represents the likelihood of the radar transitioning from state $s_t$ to state $s_{t+1}$ after being subjected to jamming. Changes in the external electromagnetic environment can cause shifts in the radar’s operational modes, with the switching rules for these modes illustrated in Figure 5. Different types of jamming have varying effects on the radar’s mode transitions, and selecting effective jamming can reduce the threat level of these modes. For instance, when the radar is operating in TWS mode and subjected to RGPJ, the target position changes, prompting the radar to re-acquire the target and switch to RWS mode. If the radar encounters SFJ and detects a new target, it can still switch to TAS mode, and if multiple targets are confirmed, it can transition to MTT mode. When subjected to NNJ, if the target is successfully suppressed, the radar loses the target and needs to search again, thus switching to VS mode.

Online jamming effectiveness assessment, a critical component in the closed-loop generation of jamming strategies, has always faced significant challenges. Accurately evaluating jamming effectiveness requires real-time processing of complex electromagnetic environment data and considering the dynamic response and adaptive changes of the target radar. This evaluation process demands efficient algorithms and robust computational power to provide reliable feedback within a short timeframe, thereby optimizing the jamming policy. Since the jammer can intercept radar signals in real time and identify the radar’s state, and different radar states pose varying levels of threat to the jammer, it is possible to determine the effectiveness of the jamming in real time based on changes in the radar state. As shown in Figure 6, this paper defined the reward based on the changes in the radar state, where $r\left(s_{t+1}|s_t\right)$ represents the reward obtained from the transition of the radar operation mode from $s_t$ to $s_{t+1}$. The reward, $r\left(s_{t+1}|s_t\right)$, is the same as the jamming effectiveness evaluation value, $r_t$, in Figure 3. In this paper, the effectiveness evaluation value was obtained based on the changes in the radar state observed after jamming. The current policy was adjusted and optimized according to the effectiveness evaluation value, ultimately identifying the optimal strategy that reduced the radar threat level the fastest. The difference in threat levels between two states is defined as:

$$\Delta TL=TL\left(s_{t+1}\right)-TL\left(s_t\right).$$

(1)

Establishing a function, $f$, between the threat level difference and the reward, the reward, $r\left(s_{t+1}|s_t\right)$, is:

$$r\left(s_{t+1}|s_t\right)=f\left(\Delta TL|s_{t+1},s_t\right).$$

(2)

The objective of this study was to minimize the threat level of the radar. Therefore, if the target threat level decreases, a positive reward is obtained, and the greater the decrease in the threat level, the higher the reward. Conversely, if the target threat level increases, a negative reward is received, and the greater the increase in the threat level, the lower the reward.

2.3. Problem Modeling

This paper modeled the problem of jamming policy generation as an MDP model. The core objective was to identify the optimal jamming pattern for each radar operation mode, ultimately transitioning the radar’s operation mode to the lowest threat level, i.e., finding the optimal jamming strategy, $\pi :S\to A$. After each jamming release, the jammer will perform an online evaluation of the jamming effectiveness, obtaining an instantaneous reward value, $r_{a_t}\left(s_t\right)$. $v_{\pi }\left(s\right)$ is the value function under policy $\pi$, indicating the anticipated cumulative reward achieved by following the policy $\pi$ from the initial state $s$, and its expression is given by the Bellman equation:

$$v_{\pi }\left(s\right)={\mathrm{E}}_{\pi }\left[{\displaystyle \sum _{t=0}^{\infty }{\gamma }^t{r}_{a_t}\left(s_t\right)|s_0=s}\right]={\mathrm{E}}_{\pi }\left[r_{a_t}\left(s_t\right)+\gamma v_{\pi }\left(s_{t+1}\right)|s_0=s\right],$$

(3)

where $\gamma \in \left(0,1\right)$ represents the discount factor, which determines the importance of future rewards in current decision-making. The jammer’s objective is to solve the Bellman optimality equation to obtain the optimal policy as:

$$v^{*}\left(s\right)=\underset{\pi }{\max } v_{\pi }\left(s\right)=\underset{\pi }{\max } {\mathrm{E}}_{\pi }\left[{\displaystyle \sum _{t=0}^{\infty }{\gamma }^t{r}_{a_t}\left(s_t\right)|s_0=s}\right],$$

(4)

$${\pi }^{*}\left(s\right)=\underset{\pi }{\arg \max } v_{\pi }\left(s\right),$$

(5)

where $v^{*}$ is the optimal value function, and ${\pi }^{*}$ is the optimal policy. According to the aforementioned Bellman equation, solving the value function is a dynamic iterative process. Each state of the MFR can yield a value, and the continuous increase in the value function corresponds to the continuous optimization of the policy. Multiple different jamming actions can be executed for each radar state, resulting in the state–action value function, $Q\left(s,a\right)$. For the jammer, the state transition rules of the MFR are unknown. Through Q-learning, the jammer can learn these rules during the adversarial process with the MFR to solve for the optimal policy. By learning the value function, $Q\left(s,a\right)$, for each state–action pair $\left(s,a\right)$, the optimal policy can be found. The value function $Q\left(s,a\right)$ is updated according to the following formula:

$$Q\left(s,a\right)\leftarrow Q\left(s,a\right)+\alpha \left(r_a\left(s\right)+\gamma Q\left(s^{\prime },a^{\prime }\right)-Q\left(s,a\right)\right),$$

(6)

where $\alpha \in \left(0,1\right)$ is the learning rate, representing the reward value obtained after performing action $a$ in state $s$. Throughout the learning process, an $\epsilon$-greedy policy was employed to balance exploration and exploitation, selecting a random action with a probability of $\epsilon$ and the current optimal action with a probability of $1-\epsilon$. Its greedy action selection strategy has similarities with simulated annealing and the Metropolis–Hastings algorithm in terms of balancing exploration and exploitation. The greedy action selection strategy in the $\epsilon$-greedy algorithm is similar to the temperature control mechanism in simulated annealing, which includes a temperature parameter that initially allows for the acceptance of poorer solutions to avoid local optima [50]. As the temperature decreases, the algorithm increasingly favors greedy choices to achieve global optimality. The Metropolis–Hastings algorithm constructs a Markov chain by accepting or rejecting candidate states, with the acceptance probability proportional to the relative probability of the candidate state [51,52]. The $\epsilon$-greedy strategy is akin to the acceptance probability strategy in the Metropolis–Hastings algorithm, particularly in introducing randomness to explore the state space. These algorithms introduce randomness and a gradual optimization strategy to avoid local optima, thereby increasing the probability of achieving a global optimum. The specific expression of the $\epsilon$-greedy policy is shown as follows:

$$a=\{\begin{array}{c}\underset{a}{\arg \max Q\left(s,a\right)},with\mathrm{a}\mathrm{probability}\mathrm{of}1-\epsilon \\ random\mathrm{selection},\mathrm{with}\mathrm{a}\mathrm{probabilty}\mathrm{of}\epsilon \end{array}.$$

(7)

The jammer interacts with the radar and collects samples $\left\{s,a,s^{\prime },r\right\}$ to update the $Q\left(s,a\right)$. The $Q\left(s,a\right)$ eventually converges to the optimal value with probability 1, from which the optimal policy and value function can be derived as:

$${\pi }^{*}\left(S\right)=\underset{a\in A}{\arg \max } Q^{*}\left(s,a\right),$$

(8)

$$v^{*}\left(s\right)=\underset{a\in A}{\max } Q^{*}\left(s,a\right),$$

(9)

The jamming strategy generation algorithm table based on Q-learning is shown in Figure 7. The jammer detects the radar state and selects a jamming style. After obtaining the performance evaluation value, the Q-table is updated using Equation (1). The process iterates continuously, and after each Q-value calculation, it checks whether the Q-values have converged. If convergence is achieved, the optimal strategy is produced; if not, the process moves on to the next learning iteration.

3. Jamming Policy Generation Method Based on MTEQL

To enhance the efficiency and accuracy of jamming policy generation, this paper proposed an efficient jamming strategy generation method based on MTEQL [53]. The overall flowchart of the algorithm is shown in Figure 8, which primarily includes four parts: sampling to generate multiple Markov environments, jamming pattern selection, jamming effectiveness evaluation, and weight coefficient calculation. The jamming style selection was performed using a greedy method, the jamming effectiveness evaluation was based on the change in threat level of the radar operating mode before and after jamming, and the weight coefficients for each Q-table were calculated based on the JS divergence. The jamming style selection and jamming effectiveness evaluation were applied to the Q-table generated for each Markov environment. Finally, all Q-tables were combined into a composite Q-table using the calculated weight coefficients, and the optimal jamming strategy was obtained after the algorithm converged. The following sections will provide a detailed introduction to this method.

This method used an $\epsilon$-greedy policy at the algorithm level and employed multiple structurally related Markov environments at the environment level to provide the jammer with radar state transition relationships over different time dimensions, thereby significantly enhancing the algorithm’s exploration capability in unknown environments. Different Markov environments correspond to different radar state transition probabilities, so the radar state transition probabilities are stored in a probability transition tensor (PTT), $P$, as Markov environments. $P$ consists of $\left|A\right|$ probability transition matrices (PTMs) of size $\left|S\right|\times \left|S\right|$. First, approximate radar state transition matrices were obtained through sampling. Then, different Markov environments were generated by multiplying the original radar state transition matrices, constructing environments with multiple timescales that correspond to the n-hop transition matrices of the Markov chain. This approach enhanced the jammer’s exploration capability and accelerated the exploration phase of Q-learning.

Figure 9 illustrates the relationship between the original Markov environment and the synthesized Markov environment. Utilizing n-hop Markov environments can help the jammer better understand and adapt to the dynamically changing radar environment. Firstly, the jammer can accelerate its understanding of the environment by considering state transitions and reward signals over longer periods, discovering new valuable state–action pairs, and reducing unnecessary interactions with the radar. Secondly, n-hop Markov environments can simulate the potential trajectories of mode transitions that are not directly observable, allowing the jammer to learn the radar’s state transition patterns from indirect experiences.

The MTEQL algorithm uses the interactions between the jammer and the radar to obtain K Markov environments, including the original Markov environment and K-1 synthesized Markov environments. The difference between the MTEQL algorithm and the original Q-learning method is shown in the Figure 10. The MTEQL algorithm runs the Q-learning algorithm in parallel across multiple environments and finally merges the Q-functions obtained from different environments into a single Q-function estimate, resulting in an approximately optimal policy.

Algorithm 1 is the pseudocode for the jamming policy generation method based on MTEQL.

Algorithm 1 The pseudocode for the jamming policy generation method based on MTEQL

Input: Sampling path length, $l$, the total number of Markov environments, $K$, minimum number of times for each state transition pair, $v$, training episodes, $E_{num}$, learning rate, $\alpha$, discount factor, $\gamma$, replacement rate, ut, and Q-tables for K different environments, $Q^{\left(n\right)}$, $n\in \left\{1,2,3,\cdots ,K\right\}$.

Output: The resultant Q-table, $Q^{it}$, and jamming policy, $\widehat{\pi }$.

01. Initialize each element of ${\widehat{P}}_a$ to ${\displaystyle \frac{1}{\left|S\right|}}$

02. while the number of samples for $\left(s,s^{\prime }\right)$ does not exceed $v$ times do

03.          Randomly select an initial radar operation mode $s$

04.          repeat $l$ times

05.                    The jammer obtains $\left\{s,a,s^{\prime }\right\}$ after randomly applying jamming

06.                    ${\widehat{P}}_a\left(s,s^{\prime }\right)\leftarrow {\widehat{P}}_a\left(s,s^{\prime }\right)+1$

07.          end

08. end while

09. Normalize the sum of each row in ${\widehat{P}}_a$ to 1, and obtain $M^{\left(1\right)}$ through sampling

10. for n = 2,3,$\cdots$,K do

11.          Compute the $n^{th}$ power of the matrix to obtain ${\widehat{P}}_a^n$

12.          Use $M^{\left(n\right)}$ to represent the synthesized Markov environment corresponding to ${\widehat{P}}_a^n$

13. end for

14. Randomly initialize the weights $w_0$, $Q_0^{it}\leftarrow 0$, $t\leftarrow 0$

15. for episode = 1,2,$\cdots$,$E_{num}$ do

16.          for n = 1,2,$\cdots$,K do

17.                    Reconnaissance obtains the initial radar operation mode $s_0$

18.                    while $s^{\prime }\ne s_{aim}$ do

19.                              Select the jamming pattern $a$ according to the $\epsilon$-greedy policy

20.                              Obtain the next radar operation mode $s^{\prime }$ from $M^{\left(n\right)}$

21.                              Obtain the reward $r$ based on the radar state transition

22.                              Update $Q_t^{\left(n\right)}$ using (5)

23.                    end while

24.                    Use SoftMax to convert $Q_t^{\left(n\right)}$ to state probabilities ${\widehat{Q}}_t^{\left(n\right)}$

25.                    $w_t^{\left(n\right)}\leftarrow 1-AJSD\left({\widehat{Q}}_t^{\left(1\right)}\Vert {\widehat{Q}}_t^{\left(n\right)}\right)$

26.          end for

27.          $w_t\leftarrow soft\max \left(w_t\right)$

28.          $Q_{t+1}^{it}\leftarrow u_t{Q}_t^{it}+\left(1-u_t\right){\displaystyle \sum _{n=1}^K{w}_t^{\left(n\right)}}{Q}_t^{\left(n\right)}$

29.          $t\leftarrow t+1$

30. end for

31. $\widehat{\pi }\left(s\right)\leftarrow \arg {\max }_{a^{\prime }} Q^{it}\left(s,a^{\prime }\right)$

Since the jammer does not initially know the radar’s state transition matrix, it is necessary to sample the environment first and create multiple synthesized Markov environments. $\widehat{P}$ represents the estimated PTT, and ${\widehat{P}}_a$ represents the PTM corresponding to each jamming pattern, $a$, in the PTT. Initialize as a matrix where each element value is ${\displaystyle \frac{1}{\left|S\right|}}$. Record the number of times each state pair is experienced during the adversarial interaction between the jammer and the radar, ensuring that each state pair is experienced at least $v$ times. Finally, normalize the sum of each row in ${\widehat{P}}_a$ to 1 to obtain an approximate estimate of the radar state transition matrix. Compute the $n^{th}$ power of matrix ${\widehat{P}}_a$ to obtain ${\widehat{P}}_a^n$, which is the PTM corresponding to the synthesized Markov environment $M^{\left(n\right)}$.

Next, run the Q-learning algorithm for each Markov environment separately. The training ends when the radar operation mode reaches the target mode (i.e., the mode with the lowest threat level). The instantaneous reward, $r$, obtained from the interactions between the jammer and the radar is used to update the Q-table for each environment. Use the SoftMax function to convert the Q-function into probabilities for each state. The objective of the jammer is to maximize the reward, so larger Q-values correspond to optimal actions. Here, $w_t^{\left(n\right)}$ is the weight vector, representing the weight of the $n^{th}$ Markov environment when updating the final Q-table. This paper used the average Jensen–Shannon divergence (AJSD) to calculate the distance between the probability distributions ${\widehat{Q}}_t^{\left(1\right)}$ and ${\widehat{Q}}_t^{\left(n\right)}$:

$$AJSD\left({\widehat{Q}}_t^{\left(1\right)}\Vert {\widehat{Q}}_t^{\left(n\right)}\right)={\displaystyle \frac{1}{s}}{\displaystyle \sum _{s}JSD\left({\widehat{Q}}_t^{\left(1\right)}\left(s,:\right)\Vert {\widehat{Q}}_t^{\left(n\right)}\left(s,:\right)\right)},$$

(10)

where $Q_t^{\left(n\right)}\left(s,:\right)$ represents the probability of selecting a certain action in state $s$ in the Q-table trained under Markov environment $M^{\left(n\right)}$. It is a probability vector of size $\left|A\right|$. The formula for calculating the JSD between two probability distributions is as follows:

$$JSD\left(p,q\right)={\displaystyle \frac{1}{2}}\left[KL\left(p\Vert {\displaystyle \frac{p+q}{2}}\right)+KL\left(q\Vert {\displaystyle \frac{p+q}{2}}\right)\right],$$

(11)

where $KL(\cdot )$ in the equation represents the Kullback–Leibler divergence (KL divergence), which is an asymmetric measure used to assess the difference between two probability distributions. It quantifies the divergence between one probability distribution and another, reflecting the information loss when distribution Q is used to approximate distribution P. When P and Q are very similar, the KL divergence approaches zero; when they differ significantly, the KL divergence increases. The definition of the KL divergence is as follows:

$$KL\left(P\parallel Q\right)={\displaystyle \sum _{x}P\left(x\right)}\log {\displaystyle \frac{P\left(x\right)}{Q\left(x\right)}}$$

(12)

AJSD is used to measure the difference between two distributions. The greater the difference between the two Q-tables, the smaller the weight during the update, so $1-AJSD\left({\widehat{Q}}_t^{\left(1\right)}\Vert {\widehat{Q}}_t^{\left(n\right)}\right)$ is used to update the weight vector. After each Markov environment completes an episode of training, a weight vector of size K is obtained. Normalize the weight vector using the SoftMax function. Then, update $Q^{it}$ with an update rate of $u_t$. Output the optimal policy when the algorithm converges.

The corresponding algorithm table is shown in Figure 11.

4. Simulation Experiments and Results Analysis

This paper established a radar state transition diagram using six operation modes of a MFR as states, as shown in Figure 12. The states $S_1\sim S_6$ represent the STT, MTT, TAS, TWS, RWS, and VS modes, with their threat levels defined as 10, 8, 6, 5, 3, and 1, respectively. The VS mode had the lowest threat level and was set as the radar’s target state. During training, the STT mode was set as the initial state. The rewards corresponding to the differences in threat levels of −4, −3, −2, −1, 1, 2, 3, and 4 were 20, 10, 5, 1, −1, −5, −10, and −20, respectively.

In the radar state transition diagram, $a_{ij}$ represents the jamming pattern that can transition the radar state from $S_i$ to $S_j$, causing the radar state to change with probability $p_{ij}$. To examine the effectiveness of the proposed algorithm across various environments, three different radar state transition models were defined: env_1, env_2, and env_3. We set the radar state transition probabilities in env_1, env_2, and env_3 to 0.6, 0.8, and 0.9, respectively, as shown in Figure 13. In env_1, the uncertainty of radar state transitions was the highest, making the environment the most unstable and posing the greatest challenge to the jammer.

The simulation experiments were conducted in a Python environment. The hardware resources included an AMD Ryzen 7 6800H processor with a clock speed of 3.2 GHz and 16 GB of memory, sourced from AMD Inc., Santa Clara, USA. Experimental parameter settings are shown in

Table 1. Simulation parameter settings.

Parameters	Value
Sampling path length, $l$	200
Minimum number of times for each state transition pair, $v$	40
Training episodes, $E_{num}$	12,000
The total number of Markov environments, $K$	5
Learning rate, $\alpha$	0.1
Discount factor, $\gamma$	0.9
Replacement rate, $u_t$	0.5

.

In reinforcement learning, the sum of Q-values can be used to evaluate whether the algorithm has converged. The algorithm is considered to have converged when the difference in the sum of Q-values between consecutive training episodes is less than a threshold. After each round of training, we computed the sum of all elements in the Q-table, defined as follows:

$${\displaystyle {\sum }_Q=}{\displaystyle \sum _{i=1}^{\left|S\right|}{\displaystyle \sum _{j=1}^{\left|A\right|}{q}_{ij}}},$$

(13)

where $\left|S\right|$ represents the number of radar states, and $\left|A\right|$ represents the number of jamming patterns. The Q-value sum curves of the Q-learning algorithm and the MTEQL algorithm in the three different environments are shown in Figure 14a–c.

It can be observed that in all three environments, the MTEQL algorithm converged earlier than the Q-learning algorithm. The number of training iterations required for convergence in the three environments was 6100, 6400, and 7800, respectively, representing reductions of 28.23%, 20%, and 14.29% compared to the convergence iterations of the traditional Q-learning algorithm. Moreover, the more unstable the environment, the better the performance improvement in the MTEQL algorithm. Due to the parallel processing nature of MTEQL, each iteration may involve a greater computational load, which varies with the number of Markov environment sets. The computational load is equivalent to a constant multiple of that in the traditional Q-learning algorithm. However, the MTEQL method compensates for this by significantly reducing the number of iterations required for convergence. This results in a net decrease in convergence time, thereby improving the efficiency of jamming strategy generation. Taking environment env_1 as an example, the Q-table generated after convergence is shown in Figure 15. It can be seen that the optimal jamming patterns corresponding to the $s_1\sim s_5$ state were $a_{13}$, $a_{24}$, $a_{34}$, $a_{46}$, and $a_{56}$. The optimal policy can be identified as $S_1\stackrel{a_{13}}{\to }{S}_3\stackrel{a_{34}}{\to }{S}_4\stackrel{a_{46}}{\to }{S}_6$, as shown in Figure 16. The red arrows represent the optimal jamming actions chosen by the jammer, demonstrating that the jammer transitioned the radar from the initial state to the target state in the fewest steps.

To investigate the impact of different numbers of Markov environments on algorithm performance, we varied the number of Markov environments by setting K to 2, 3, 4, and 5, respectively. Experiments were conducted in the env_1 environment, and the resulting Q-value sum curves are shown in Figure 17.

The experimental results indicated that the algorithm converged when K was set to 2, 3, 4, and 5. The Q-values represent the expected cumulative rewards for taking certain actions in given states. The faster convergence of MTEQL allowed it to stabilize and optimize its strategy more quickly, which is the primary advantage we highlighted here. However, the convergence speed was slightly faster when K was set to 5 compared to the other three cases. This is because, with K set to 5, the jammer can learn more knowledge from indirect experiences, enhancing its exploration capability in unknown environments. The accuracy of the policy determines whether the jammer can complete the jamming task, and the average policy error (APE) can measure the accuracy of the strategy. The APE is the probability that the jamming pattern chosen in each state is not the optimal jamming pattern out of all states. Its calculation formula is as follows:

$$APE={\displaystyle \frac{1}{\left|S\right|}}{\displaystyle \sum _{s=1}^{\left|S\right|}1\left({\pi }^{\ast }\left(s\right)\ne \widehat{\pi }\left(s\right)\right)},$$

(14)

where ${\pi }^{\ast }$ is the optimal policy calculated according to Equation (5), and $\widehat{\pi }$ is the policy learned from different Markov environments. Figure 18a shows the average strategy error obtained by the MTEQL algorithm, and Figure 18b–f shows the average strategy error for Markov environments $M^{\left(1\right)}$ to $M^{\left(5\right)}$.

The experimental results indicated that the strategy derived using the MTEQL algorithm exhibited an APE of zero per round. In contrast, the strategy generated from the original Markov environment, M1, contained errors, suggesting that the traditional Q-learning algorithm may encounter decision errors during the training process. Furthermore, as the value of K increased, the APE of the training strategy also rose. These findings demonstrated that integrating strategies obtained from training multiple Markov environments is a feasible approach, yielding optimal strategies.

5. Conclusions

Reinforcement-learning-based jamming strategy generation methods can adjust jamming strategies in real time, understanding the patterns of radar operation modes in adversarial scenarios to find the optimal jamming strategy for each mode. However, traditional Q-learning algorithms face challenges, such as limited exploration capabilities in unknown environments and long convergence times. To address these issues and improve the jamming decision efficiency, enhance the jammer’s exploration capability in the strategy space, and reduce decision error rates, we introduced an efficient method for generating jamming strategies based on multi-timescale ensemble Q-learning. To confirm the algorithm’s effectiveness in real battlefield environments, we first constructed an adversarial scenario between the jammer and the radar and thoroughly analyzed the rules of radar operation mode transitions. Additionally, multiple Markov environments were utilized to accelerate the jammer’s understanding of the environment and speed up the algorithm’s convergence. Simulation experiments demonstrated that the proposed algorithm converged faster than the Q-learning algorithm in various environments and performed better in unstable environments. Moreover, the algorithm exhibited a very low average strategy error, capable of generating optimal jamming strategies.

Of course, the model established in this paper was relatively simple, considering only the impact of jamming on radar states and not accounting for factors such as targets. Therefore, the next step could be to improve the model to make it more representative of real-world adversarial scenarios. Additionally, the electromagnetic environment in actual battlefields is complex, and merely selecting the optimal jamming pattern may not effectively interfere with the radar. Optimizing jamming parameters based on the selected jamming pattern is also a direction for future research.