Frequency Diversity Array Radar and Jammer Intelligent Frequency Domain Power Countermeasures Based on Multi-Agent Reinforcement Learning

Abstract

With the development of electronic warfare technology, the intelligent jammer dramatically reduces the performance of traditional radar anti-jamming methods. A key issue is how to actively adapt radar to complex electromagnetic environments and design anti-jamming strategies to deal with intelligent jammers. The space of the electromagnetic environment is dynamically changing, and the transmitting power of the jammer and frequency diversity array (FDA) radar in each frequency band is continuously adjustable. Both can learn the optimal strategy by interacting with the electromagnetic environment. Considering that the competition between the FDA radar and the jammer is a confrontation process of two agents, we find the optimal power allocation strategy for both sides by using the multi-agent deep deterministic policy gradient (MADDPG) algorithm based on multi-agent reinforcement learning (MARL). Finally, the simulation results show that the power allocation strategy of the FDA radar and the jammer can converge and effectively improve the performance of the FDA radar and the jammer in the intelligent countermeasure environment.

1. Introduction

In recent years, with the development of radar jamming and anti-jamming technology, the electronic countermeasures between the radar and jammer have become increasingly fierce [1,2]. Because the jammer and the target are in the radar antenna’s main lobe and share the radar antenna’s main lobe gain, the main lobe jamming is complicated to deal with in all kinds of jamming [3,4].

In 2006, Antonik et al. proposed the FDA radar [5], which has a small frequency increment between each transmitting array element and produces a directional graph with range–angle–time coupling characteristics [6,7,8]. The FDA radar’s performance is enhanced by unique features, such as beamforming [9], target parameter estimation [10,11], detection [12], tracking [13], imaging [14], clutter suppression [15], and low intercept performance [16,17,18,19]. In addition, the FDA radar can effectively suppress forwarded spoofing jamming located in the main lobe. [20,21,22]. Due to the frequency increment of each array element of the FDA radar, it also has excellent potential to suppress the spectrum-blocking jamming in the main lobe, and it can avoid fixed spectrum-blocking jamming through the transmitting power allocation [23,24]. Compared with the traditional suppression of spectrum-blocking jamming by spectrum notching [25,26], the FDA radar does not require complex optimization methods to design radar waveforms, nor does it require arbitrary waveform generators to generate optimized waveforms [27,28]. The above methods focus on the passive suppression of jamming signals by signal processing after the radar receives jamming signals. Compared with passive jamming signal suppression, with radar-active countermeasures the radar can take measures in advance to avoid jamming itself; thus, it has higher flexibility and freedom [3,29].

Aiming at the spectrum-blocking jamming caused by communication systems to radar, ref. [30] modeled the jamming suppression process as a Markov decision process. It sought the optimal transmitting strategy through strategy iteration. Aiming at intermittent spectrum jamming, [31] applies deep reinforcement learning (DRL) to spectrum jamming suppression to realize the real-time learning and online updating of a radar transmitting strategy. Aiming at fixed spectrum-blocking jamming, [32] uses DRL to avoid fixed spectrum-blocking jamming by transmitting the power allocation of an FDA-MIMO radar. Most studies assume that the jammer is a non-agent and can only adopt a fixed jamming strategy. However, by using learning radar strategies, jammers can adjust their jamming strategies [33,34]; so, studying the intelligent countermeasures between the radar and the jammer is significant.

Some scholars use the static game analysis framework to model the relationship between the radar and the jammer [35,36], and their model cannot represent the sequential decision problem between the radar and the jammer. The confrontation between the radar and the jammer should be a multi-round interactive sequential decision. In [37,38], the authors use the extensive-form game (EFG) model to model the multi-round interaction between the frequency-agile radar and the jammer. It uses the neural fictitious self-play (NFSP) method to seek the approximate Nash equilibrium strategy, but the limited interaction rounds and discrete strategies still limit its model. It is an essential premise to establish a more realistic radar and jamming counteraction model to realize radar-active counteraction.

This paper first introduced a parametric measurement model in which the target response changes with the transmitting frequency of the FDA radar [39]. Under the condition that the total energy is unchanged, the FDA radar can continuously adjust the energy distribution of the corresponding frequency of each array element to avoid spectrum jamming and improve the radar signal-to-jamming-plus-noise ratio (SJNR). The jammer can transmit a broadband jamming signal, and its center frequency aligns with the radar center frequency. The jamming energy of each frequency is continuously adjustable when the total jamming energy is unchanged. The MARL simulates the multi-round sequence decision process between the radar and the jammer. Under this model, we study the frequency–power confrontation strategy between the FDA radar and the jammer under the dynamic distribution of jamming power. The main contributions of this paper are as follows:

(1)

The FDA radar signal model with target response varying with frequency is established based on continuously adjustable jamming and radar power. Targets respond differently to different frequencies, and FDA radars and jammers can use target response characteristics to allocate power in different frequency bands.

(2)

The adversarial relationship between the FDA radar and the jammer is mapped to a MARL model. Power allocation in the frequency domain realizes the confrontation between the FDA radar and the jammer. In designing a reward function based on SJNR, the radar optimization strategy maximizes SJNR, while the jammer adjustment strategy minimizes SJNR.

(3)

In the DRL framework, the power allocation strategies of the FDA radar and jammer were fixed, respectively, and the performance of the frequency domain power countermeasures using the deep deterministic policy gradient (DDPG) algorithm was analyzed. When one side of the FDA radar or jammer had a fixed transmission strategy, the other side adopted the DRL framework to optimize the transmission strategy, which could significantly improve the performance.

(4)

The intelligent frequency domain power countermeasure of the FDA radar and jammer is analyzed using the MADDPG algorithm and the method of centralized training with decentralized execution (CTDE) based on MARL.

The structure of this paper is as follows. Section 2 gives the FDA radar signal model and jammer power allocation model and establishes the relationship between the FDA radar power distribution and SJNR. In Section 3, the meaning of each element of the FDA radar and jammer mapping to the DRL framework is designed; a single-agent environment interaction model is established; the DDPG algorithm is described; and the transmitting strategy optimization method is given. In Section 4, the meaning of each element of the FDA radar and jammer mapping to the MARL framework is designed in detail; the multi-agent environment interaction model is established; the MADDPG algorithm is described; and the performance and effect are analyzed. Section 5 shows the simulation results, and the theoretical analysis is verified. The Section 6 summarizes the paper.

2. Signal Model

We consider that the transmitting array and receiving array of the FDA radar are uniform linear arrays (ULAs); the transmitting array includes $M_t$ transmitting array elements; the receiving array includes $M_r$ receiving array elements; and the array unit spacing is $d$. The basic framework of the FDA radar is shown in Figure 1. With the first transmitting array element as the reference array element, the carrier frequency of the $m$ transmitting array element is:

$$f_m=f_0+(m-1)\Delta f$$

(1)

where $f_0$ represents the carrier frequency of the reference array, and $\Delta f$ represents the frequency increment between adjacent arrays. The signal transmitted by the $m$ transmitting array can be expressed as:

$${\phi }_M(t)=w_m\varphi (t)e^{j2\pi f_{m}t}$$

(2)

where $w_m$ is the transmitting weight value representing the power distribution of each transmitting element. $\varphi (t)$ is a baseband pulse waveform, and its bandwidth meets $B\ge \Delta f$; so, the signals transmitted by each transmitting array meet the orthogonality in the frequency domain:

$${\displaystyle \int \varphi (t){\varphi }^{\ast }(t-\tau )e^{j2\pi (m-m^{\prime })}dt=\left\{\begin{array}{c}1,\forall \tau ,m=m^{\prime }\\ 0,\forall \tau ,m\ne m^{\prime }\end{array}\right.}$$

(3)

where $\ast$ stands for conjugation.

Considering that a point target in space is located in the far field $(r_0,{\theta }_0)$, the received signal of the $n$-th receiving array element of the FDA radar can be expressed as:

$$\begin{array}{cc}\hfill x_n(t)& =\alpha e^{j2\pi \frac{d}{{\lambda }_0}(n-1)\sin {\theta }_0}{\displaystyle \sum _{m=0}^{M_t-1}\varphi (t-{\tau }_m)w_m{\sigma }_m^t{e}^{j2\pi f_m(t-{\tau }_m)}}\hfill \\ & \approx {\alpha }_0\varphi (t-\tau )e^{j2\pi \frac{d}{{\lambda }_0}(n-1)\sin {\theta }_0}{\displaystyle \sum _{m=0}^{M_t-1}{w}_m{\sigma }_m^t{e}^{j2\pi f_{m}t}{e}^{j2\pi m\frac{d}{{\lambda }_0}\sin {\theta }_0}{e}^{-j4\pi m\Delta f\frac{r_0}{c}}}\hfill \end{array}$$

(4)

where ${\alpha }_0=\alpha e^{-j4\pi f_0\frac{r_0}{c}}$ includes the path loss of signal propagation, $e^{j2\pi \frac{d}{{\lambda }_0}(n-1)\sin {\theta }_0}$ represents the phase difference caused by the spacing of the received array elements, $w_m$ represents the energy weight, ${\sigma }_m^t$ represents the target response of each frequency channel, ${\tau }_m=(2r_s-md\sin {\theta }_0)/c=\tau -md\sin {\theta }_0/c$ represents the two-way delay of signal propagation, and $c$ represents the speed of light. The signals received by the FDA radar array can be expressed as:

$$x_t(t)={\alpha }_0{a}_r({\theta }_0){[w\odot a_t(r_0,{\theta }_0)\odot {\sigma }^t]}^{T}diag\{e(t)\}\Psi (t-\tau )$$

(5)

where $\odot$ represents the Hadamard product, ${(\cdot )}^T$ represents the transpose, and $diag\{\cdot \}$ represents the diagonal matrix. $a_t(r_0,{\theta }_0)$ and $a_r({\theta }_0)$ represent the transmit and receive array guide vectors, respectively. $e(t)$ represents the transmission vector related to the time and frequency increments, $\Psi (t-\tau )$ is the transmitted baseband signal vector, and $x_t(t)$ is the received signal vector. They can be expressed as:

$$a_r({\theta }_0)={[1,e^{j2\pi \frac{d}{{\lambda }_0}\sin {\theta }_0},\cdots ,e^{j2\pi \frac{d}{{\lambda }_0}(M_r-1)\sin {\theta }_0}]}^T$$

(6)

$$w={[w_0,w_1,\cdots ,w_{M_t-1}]}^T$$

(7)

$$a_t(r_0,{\theta }_0)=a_t({\theta }_0)\odot a_t(r_0)$$

(8)

$$a_t({\theta }_0)={[1,e^{j2\pi \frac{d}{{\lambda }_0}\sin {\theta }_0},\cdots ,e^{j2\pi (M_t-1)\frac{d}{{\lambda }_0}\sin {\theta }_0}]}^T$$

(9)

$$a_t(r_0)={[1,e^{-j4\pi \Delta f\frac{r_0}{c}},\cdots ,e^{-j4\pi (M_t-1)\Delta f\frac{r_0}{c}}]}^T$$

(10)

$${\sigma }^t={[{\sigma }_0^t,{\sigma }_1^t,\cdots ,{\sigma }_{M_t-1}^t]}^T$$

(11)

$$e(t)={[e^{j2\pi f_{0}t},e^{j2\pi f_{1}t},\cdots ,e^{j2\pi f_{M_t-1}t}]}^T$$

(12)

$$\Psi (t)=\varphi (t)·I_{M_t\ast 1}$$

(13)

$$x_t(t)={[x_0(t),x_1(t),\cdots ,x_{M_r-1}(t)]}^T$$

(14)

where $I_{M_t·1}$ represents a $M_t·1$ dimensional vector in which the elements are all 1.

The receiving end of the array adopts the processing architecture of a multi-channel mixing and matching filter. The multi-channel processing scheme is shown in Figure 2.

After multi-channel mixing and matching filtering, $x(t)$ can be expressed as $Y_t(w)$.

$$Y_t(w)={\alpha }_0{a}_r({\theta }_0){[w\odot a_t(r_0,{\theta }_0)\odot {\sigma }^t]}^T$$

(15)

Further, convert the matrix $Y_t(w)$ to vector $y_t(w)$.

$$y_t(w)=vec(Y_t(w))={\alpha }_0(w\odot a_t(r_0,{\theta }_0)\odot {\sigma }^t)\otimes a_r({\theta }_0)$$

(16)

By observing (15) and (16), the received signal of the FDA radar is related to the transmission power allocation, and the spectrum of the transmitted signal can be designed by adjusting it to avoid spectrum jamming.

The jammer is assumed to align the jamming signal carrier frequency with the FDA radar signal carrier frequency and to implement the spectrum jamming of the radar. After passing through the $m$-th matching filter channel of a single receiving array, the jamming signal can be represented as [24]:

$$j_m={\displaystyle \int i(t)e^{-j2\pi f_{m}t}{\varphi }^{\ast }(t-\tau )dt}$$

(17)

The jamming signal processed by a single receiving array can be represented as a vector $j$.

$$j=[i_0,i_1,\cdots i_m,\cdots ,i_{M_t-1}]$$

(18)

Further, jammer signals and noise signals, after being processed by the receiving array, can be expressed as:

$$y_{j+n}=j\otimes a_r({\theta }_j)+n$$

(19)

where ${\theta }_j$ indicates the direction of the jammer. $n\in {ℂ}^{M_t{M}_r·1}\sim (0,{\sigma }_n^2{I}_{M_t{M}_r·M_t{M}_r})$ represents the noise vector.

The signals transmitted by each transmitting array are orthogonal in the frequency domain, and the jamming signals also meet the orthogonality after passing through the multi-matching filter. The correlation matrix of the jamming signal can be expressed as $P$.

$$P=diag(p)$$

(20)

$$p=[p_0,p_1,\cdots ,p_{M_t-1}]$$

(21)

$$p_m=E\left\{{\left|i_m\right|}^2\right\}$$

(22)

where $E\{·\}$ represents calculation expectation.

The covariance matrix of the jamming signal and noise signal can be expressed as $R_{j+n}$.

$$R_{j+n}=P\otimes [a_r({\theta }_j)a_r^H({\theta }_j)]+{\sigma }_n^2{I}_{M_t{M}_r·M_t{M}_r}$$

(23)

The total signal received by the FDA radar can be expressed as the sum of the target echo signal, the jamming signal, and the noise signal.

$$y(w)=y_t(w)+y_{j+n}={\alpha }_0(w\odot a_t(r_0,{\theta }_0)\odot {\sigma }^t)\otimes a_r({\theta }_0)+j\otimes a_r({\theta }_j)+n$$

(24)

The adaptive beamforming method adopted by FDA radar at the receiving end is minimum variance distortionless response (MVDR). After signal processing, the SJNR can be expressed as $SJNR(w)$.

$$SJNR(w)={\sigma }_0^2(w\odot a_t(r_0,{\theta }_0)\odot {\sigma }^t)\otimes a_r({\theta }_0){)}^H{R}_{j+n}^{-1}(w\odot a_t(r_0,{\theta }_0)\odot {\sigma }^t)\otimes a_r({\theta }_0))$$

(25)

The FDA radar transmitting power allocation optimization model can be formulated as an optimization problem with the optimization objective of maximizing SJNR and a constant energy constraint, as shown in Equation (26).

$$\begin{array}{ll}\max \hfill & SJNR(w)\hfill \\ s.t.\hfill & {‖w‖}^2=1\hfill \\ \hfill & {‖j‖}^2=1\hfill \end{array}$$

(26)

3. Single-Agent Reinforcement Learning

3.1. DDPG Algorithm

For the parametric measurement model in which target response varies with the transmitting frequency of the FDA radar, the frequency domain power allocation between the FDA radar and the jammer should not be regarded as a set of discrete actions [40]. Power allocation should be a continuous control problem; the action space is a continuous set [41]. For the continuous control method, this paper considers the application of the DDPG algorithm to realize the power allocation between the FDA radar and the jammer [42].

The DDPG algorithm is an actor–critic algorithm, which consists of a strategy network and a value network. The policy network control agent takes an action according to the state. The value network scores the action based on the state and guides the iterative optimization of the strategy network according to this. Training the strategy network involves improving the parameters to obtain higher values in the value network. The value network is trained by optimizing the parameters so that the value network’s prediction is closer to the actual value function. The relationship between the policy and value networks is shown in Figure 3.

Both the policy and value networks adopt the multi-layer perceptron (MLP), and the network information of the two is shown in

Table 1. Policy network parameter information.

Layer	Input	Output	Activation Function
MLP1	state dimension	hidden dimension	Relu
MLP2	hidden dimension	hidden dimension	Relu
MLP3	hidden dimension	action dimension	softmax

and

Table 2. Value network parameter information.

Layer	Input	Output	Activation Function
MLP1	state and action dimension	hidden dimension	Relu
MLP2	hidden dimension	hidden dimension	Relu
MLP3	hidden dimension	1	/

, respectively.

The DDPG is an off-policy method, and its behavior strategy differs from the target strategy, which is $a=u(s;{\theta }^{-})$. The off-policy method can collect experience data through one strategy, train another strategy network, and reuse the collected experience during training. At the same time, the DDPG algorithm is similar to the deep Q network (DQN) algorithm, which will cause an overestimation of the real action value. The main reason for overestimation is that the temporal difference (TD) target overestimates the actual value, and bootstrapping leads to overestimation propagation. In order to solve the overestimation problem, the target strategy network $u(s;{\theta }^{-})$ and the target value network $q(s,a;{\omega }^{-})$ must be introduced during the training. We increase the exploratory nature of the action by adding noise $\mathbb{N}$ to the policy network. The detailed flow of the DDPG algorithm applied to power allocation is shown in Algorithm 1.

Algorithm 1 DDPG algorithm applied to power allocation

Initialize policy network $u(s;\theta )$ and value network $q(s,a;\omega )$ with random network parameters $\theta$ and $\omega$. Initialize the target network by copying the same parameters, ${\theta }^{-}\leftarrow \theta$ and ${\omega }^{-}\leftarrow \omega$.

Initialize the experience playback pool $R$

Input the maximum number of rounds $E$, time step $T$, discount factor $\gamma$, policy network learning rate $l{r}_{actor}$, value network learning rate $l{r}_{critic}$, soft update parameter $\tau$, $R$ storage capacity $N_{total}$, minimum amount of data required for sampling $N_{\min }$, and batch size $N$, Gaussian noise variance ${\sigma }^2$.

For $e=1\to E$ do

Initialize Gaussian noise $\mathbb{N}$

Get initial state $s_1$

For $t_0=1\to T$ do

Select action $a_t=u(s_t;\theta )+\mathbb{N}$ according to the current policy.

Act $a_t$, get the reward $R_t$, and the environment state changes to $s_{t+1}$.

Store $(s_t,a_t,R_t,s_{t+1})$ in the experience playback pool $R$

Sample $N$ tuples ${\left\{(s_i,a_i,R_i,s_{i+1})\right\}}_{i=1,\cdots ,N}$ from $R$

Calculate the TD target for each tuple $y_i=R_i+\gamma q(s_{i+1},u(s_{i+1};{\theta }^{-});{\omega }^{-})$.

Minimize loss function $L=\frac{1}{N}{\displaystyle \sum _{i=1}^N{(y_i-q(s_i,a_i;\omega ))}^2}$ and update value network

Calculate the policy gradient ${\nabla }_{\theta }J\approx \frac{1}{N}{\displaystyle \sum _{i=1}^N{\nabla }_{\theta }u(s_i;\theta ){\nabla }_{a}q(s_i,a;\omega ){|}_{a=u(s_i;\theta )}}$ and update the policy network

Soft update target strategy network ${\theta }^{-}=\tau \theta +(1-\tau ){\theta }^{-}$

Soft update target value network ${\omega }^{-}=\tau \omega +(1-\tau ){\omega }^{-}$

End for

End for

3.2. FDA Radar Agent

The background of this section is that the power allocation of the jammer is constant in each frequency band, and the jammer is a non-intelligent agent that cannot achieve intelligent power allocation by interacting with the electromagnetic environment. The FDA radar is an agent that can use the DDPG algorithm to achieve intelligent allocation of the FDA radar transmitting power by interacting with an electromagnetic environment. The interaction process between the FDA radar and the electromagnetic environment is shown in Figure 4.

As shown in Figure 4, the status $s$ perceived by the FDA radar is the frequency domain power allocation $p$ of the jamming in an electromagnetic environment, where the frequency domain power is arbitrarily allocated in the jamming frequency band. The FDA radar action $a$ is that it transmits power allocation $w$. The FDA radar’s reward $R$ is the SJNR received by the radar, and it determines the power allocation preference of FDA radars. The FDA radar continuously adjusts the transmitting power allocation, obtains a reward, and updates the state. The FDA radar transmitting strategy is optimized with the maximum reward as the optimization objective and the policy network as the optimization variable. Finally, the FDA radar’s strategy network $u(p;\theta )$, which is obtained by interacting with the electromagnetic environment, is its learned power allocation strategy model.

3.3. Jammer Agent

The background of this subsection is considered to be the fact that the FDA radar transmitting power is constant at each frequency band. The jammer is an agent that can realize the intelligent allocation of jamming power by interacting with an electromagnetic environment and using the DDPG algorithm. The interaction process between the jammer and the electromagnetic environment is shown in Figure 5.

As shown in Figure 5, the state $s$ sensed by the jammer is the transmitting power distribution $w$ of the FDA radar in the electromagnetic environment, where the transmitting power of the radar can be arbitrarily allocated in the signal frequency band. The jammer’s action $a$ is its transmitted jamming power distribution $p$. The reward $R$ obtained by the jammer is set to a negative value of SJNR, and the larger the reward function, the smaller the SJNR.

4. Multi-Agent Reinforcement Learning

In an electromagnetic environment, both radars and jammers are intelligent systems, and both can learn through sequential interaction with the environment to improve their strategies. Therefore, the FDA radar and jammer together constitute a multi-agent system. The FDA radar and the jammer share the electromagnetic environment. The two interact with each other, and the change in the action of one party causes a change in the electromagnetic environment, which causes the other party’s state to change.

Under the framework of MARL, the FDA radar and jammer are in the electromagnetic environment together, and both interact with the electromagnetic environment independently, using the obtained rewards to improve their strategies to obtain higher cumulative rewards. The strategy of the FDA radar and jammer depends not only on their observations and actions but also on the perception and actions of the other party. At the same time, the relationship between the two belongs to the perfect competition relationship; the loss of one party is the gain of the other party; the two parties constitute a zero-sum game, and the sum of the reward obtained is 0.

The observation of the FDA radar is the jammer’s power allocation $o^{FDA}=p$, and the action is its own transmitting power allocation $a^{FDA}=w$. The observation of the jammer is the FDA radar’s power allocation $o^{jammer}=w$, and the action is the power allocation $a^{jammer}=p$ of its power allocation. The state of the entire electromagnetic environment is $s=[o^{FDA},o^{jammer}]$, and the action is $a=[a^{FDA},a^{jammer}]$. Since the FDA radar and jammer constitute a zero-sum game, $R^{FDA}=-R^{jammer}$. The interaction diagram between the FDA radar and the jammer is shown in Figure 6.

For the sequential decision optimization problem of the FDA radar and jammer, this paper considers the application of the MADDPG algorithm [43], which is an algorithm based on the actor–critic method under the CTDE framework. In this framework, all the agents share a centralized value network, simultaneously evaluating each agent’s policy network during training. The policy network of each agent is entirely independent, enabling decentralized execution. The MADDPG algorithm shows the relationship between the policy and value networks in Figure 7.

Both the strategy and value networks adopt MLP, and the network information of the two is shown in

Table 3. Policy network parameter information of MADDPG.

Layer	Input	Output	Activation Function
MLP1	observation dimension	hidden dimension	Relu
MLP2	hidden dimension	hidden dimension	Relu
MLP3	hidden dimension	agent action dimension	softmax

and

Table 4. Value network parameter information of MADDPG.

Layer	Input	Output	Activation Function
MLP1	state and environment action dimension	hidden dimension	Relu
MLP2	hidden dimension	hidden dimension	Relu
MLP3	hidden dimension	1	/

, respectively.

Like the DDPG algorithm, it solves the overestimation problem using the target strategy networks and target value networks and increasing the exploitability of the action by adding noise $\mathbb{N}$ to the policy network. The specific process of the MADDPG algorithm applied to the FDA radar and jamming intelligence is shown in Algorithm 2.

Algorithm 2 MADDPG algorithm applied to power allocation

Initialize the FDA radar and jammer strategy network $u(o^i;{\theta }^i)$ and value network $q(s,a;{\omega }^i)$, with random network parameters ${\theta }^i$ and ${\omega }^i$. Initialize the target network by copying the same parameters, ${\theta }^{i-}\leftarrow {\theta }^i$ and ${\omega }^{i-}\leftarrow \omega$.

Initialize the experience playback pool $R$

Input the maximum number of rounds $E$, time step $T$, discount factor $\gamma$, policy network learning rate $l{r}_{actor}$, value network learning rate $l{r}_{critic}$, soft update parameter $\tau$, $R$ storage capacity $N_{total}$, minimum amount of data required for sampling $N_{\min }$, and batch size $N$, Gaussian noise variance ${\sigma }^2$.

For $e=1\to E$ do

Initialize Gaussian noise $\mathbb{N}$

Initial observations of FDA radars and jammers obtained constitute the initial state $s_1=[o_1^{FDA},o_1^{jammer}]$

For $t_0=1\to T$ do

For the $i$-th agent, select an action based on the current policy $a_t^i=u(o^i;{\theta }^i)+\mathbb{N}$.

Act $a_t=[a_t^{FDA},a_t^{jammer}]$, obtain the reward $R_t$, and the environment state changes to $s_{t+1}=[o_{t+1}^{FDA},o_{t+1}^{jammer}]$.

Store $(s_t,a_t,R_t,s_{t+1})$ in the experience playback pool $R$

Sample $N$ tuples ${\left\{(s_t,a_t,R_t,s_{t+1})\right\}}_{t=1,\cdots ,N}$ from $R$

Let the FDA radar and jammer target strategy network make predictions ${\widehat{a}}_{t+1}^{i-}=u(o_{t+1}^i;{\theta }^{i-})+\mathbb{N}$, $\forall i=FDA,jammer$. Obtain predictive actions acting on the electromagnetic environment ${\widehat{a}}_{t+1}^{-}=[{\widehat{a}}_{t+1}^{FDA-},{\widehat{a}}_{t+1}^{jammer-}]$.

Let the FDA radar and jammer target value network make predictions ${\widehat{q}}_{t+1}^{i-}=q(s_{t+1},{\widehat{a}}_{t+1}^{-};{\omega }^{i-})$, $\forall i=FDA,jammer$.

Calculate the TD target for each tuple $y_t^i=R_t^i+\gamma {\widehat{q}}_{t+1}^{i-}$, $\forall i=FDA,jammer$.

Let the value network of FDA radars and jammers make predictions $q_t^i=q(s_t,a_t;{\omega }^i)$, $\forall i=FDA,jammer$.

Calculate TD error ${\delta }^i=q_t^i-y_t^i$, $\forall i=FDA,jammer$.

Update value network ${\omega }^i={\omega }^i-\alpha {\delta }^i{\nabla }_{{\omega }^i}q(s_t,a_t;{\omega }^i)$, $\forall i=FDA,jammer$.

Let the FDA radar and jammer strategy network make predictions ${\widehat{a}}_t^i=u(o_t^i;{\theta }^i)+\mathbb{N}$, $\forall i=FDA,jammer$. Obtain predictive actions acting on the electromagnetic environment ${\widehat{a}}_t=[{\widehat{a}}_t^{FDA},{\widehat{a}}_t^{jammer}]$.

Update policy network ${\theta }^i={\theta }^i-\beta {\nabla }_{{\theta }^i}u(o_t^i;{\theta }^i){\nabla }_{a_t^i}q(s_t,{\widehat{a}}_t;{\omega }^i)$, $\forall i=FDA,jammer$.

Soft update target policy network ${\theta }^{i-}=\tau {\theta }^i+(1-\tau ){\theta }^{i-}$, $\forall i=FDA,jammer$.

Soft update target value network ${\omega }^{i-}=\tau {\omega }^i+(1-\tau ){\omega }^{i-}$, $\forall i=FDA,jammer$

End for

End for

5. Simulation Results

Set the number of transmitting and receiving array elements $M_t=M_r=6$, array spacing $d={\lambda }_0/2$, carrier frequency $f_0=10 \mathrm{GHz}$, frequency increment $\Delta f=100 \mathrm{MHz}$, and baseband pulse waveform bandwidth $B_{\varphi (t)}=100 \mathrm{MHz}$. The target is in $({30}^{\circ },50 \mathrm{km})$, and the jammer is in $({30}^{\circ },40 \mathrm{km})$. The signal-to-noise ratio is $SNR=-10 \mathrm{dB}$; the jamming-to-noise ratio is $JNR=20 \mathrm{dB}$. Different frequencies will cause changes in the radar cross-sectional (RCS). Set the target RCS for the FDA radar $M_t=6$ transmitting bands as ${\sigma }^t={[20,16,12,8,4,0.1]}^T$. The remaining simulation parameters are set in each section.

This simulation is divided into three parts. The first part is for the fixed jammer power allocation, which uses the DDPG algorithm to realize the FDA radar intelligent frequency domain power allocation; the second part is for the fixed radar transmitting power allocation, which uses the DDPG algorithm to realize the jamming power intelligent allocation; the third part is for the MARL framework, which verifies and analyzes the performance of the FDA radar intelligence against the jammer in the frequency domain power allocation using the MADDPG algorithm.

5.1. FDA Radar Intelligent Frequency Domain Power Allocation

The jamming power allocation of the jammer is set in two cases, respectively. Case 1: the jamming power is evenly allocated $p_1=[\frac{1}{6},\frac{1}{6},\frac{1}{6},\frac{1}{6},\frac{1}{6},\frac{1}{6}]$. Case 2: align jamming to the maximum target scattering frequency $p_2=[1,0,0,0,0,0]$. The other simulation parameters are set to: $l{r}_{actor}=0.0003$, $l{r}_{critic}=0.003$, $E=400$, $T=25$, $hidden dimension=64$, $\gamma =0.98$, $\tau =0.005$, $N_{total}=10000$, $N_{\min }=1000$, $N=64$, and ${\sigma }^2=0.05$.

The power allocation of the two jamming cases in the frequency–time two-dimensional plane is shown in Figure 8.

To compare the effect of improving the FDA radar power allocation strategy before and after training, the power allocation schemes of the first and last episode of the FDA radar under two jamming conditions are given, as shown in Figure 9.

We can see from Figure 9 that in the face of different jamming environments, the FDA radar does not know the optimal power allocation strategy at the beginning and conservatively adopts the power allocation strategy of uniform allocation as far as possible. Then, after interacting with the electromagnetic environment, the FDA radar realizes the optimization and improvement of its power allocation strategy by learning the power allocation strategy of the jammer. When faced with a uniform allocation of jamming power, the FDA radar concentrates most of its power on the bands with the largest RCS. When focusing jamming power on the band with the largest RCS, the FDA radar concentrates most of its power on the band with the next largest RCS. In both cases, the trend of return (SJNR) with the episodes is shown in Figure 10.

As shown in Figure 10, in the face of different jamming power allocation situations, the FDA radar uses the DDPG algorithm to improve SJNR by optimizing transmitting power allocation strategies. At the same time, we can find that when facing the fixed allocation of jamming power, the FDA radar can learn the jamming strategy quickly to achieve the purpose of transmitting power to avoid the jamming band. However, we can also find that the final convergence of the SJNR value is closely related to different jammer power allocation strategies.

5.2. Jammer Intelligent Frequency Domain Power Allocation

The FDA radar transmitting power allocation is set in two cases, respectively. Case 1: the transmitting power is evenly allocated $w_1=[\frac{1}{6},\frac{1}{6},\frac{1}{6},\frac{1}{6},\frac{1}{6},\frac{1}{6}]$. Case 2: the allocated power is in three frequency bands $w_2=[0,\frac{1}{3},\frac{1}{3},0,\frac{1}{3},0]$. The other simulation parameters are set to: $l{r}_{actor}=0.0003$, $l{r}_{critic}=0.003$, $E=2500$, $T=25$, $hidden dimension=64$, $\gamma =0.98$, $\tau =0.005$, $N_{total}=10000$, $N_{\min }=1000$, $N=64$, and ${\sigma }^2=0.05$.

The power allocation of the two FDA radar transmitting strategies in the frequency–time two-dimensional plane is shown in Figure 11.

To compare the effect of improving the power allocation strategy of the jammer before and after training, the jamming power allocation scheme of the first and last episode of the jammer under two FDA radar transmission strategies is given, as shown in Figure 12.

As shown in Figure 12, when faced with different FDA radar transmitting strategies, the jammer initially adopted a strategy of evenly allocating jamming power as much as possible. Then, after interacting with the electromagnetic environment, the jammer realized the optimization and improvement of its power allocation strategy by learning the power allocation strategy of the FDA radar. When evenly allocating the FDA radar transmitting power, the jammer allocates the power proportionally according to the size of the RCS in each frequency band. When concentrating the transmitting power of the FDA radar in a few frequency bands, the jammer aligns the jamming power with the corresponding frequency band, and the allocated jamming power is proportional to the size of the RCS of each frequency band. In both cases, the trend of SJNR with the episodes is shown in Figure 13.

As shown in Figure 13, in the face of different FDA radar transmitting power allocation conditions, the jammer uses the DDPG algorithm to reduce SJNR by optimizing jamming power allocation strategies. At the same time, when facing the FDA radar transmitting power fixed allocation, the jammer can learn the FDA radar transmitting power strategy in a very short time to achieve the purpose of reducing SJNR by adjusting the jamming strategy. However, we can find that the final SJNR value is also closely related to the FDA radar power allocation strategy.

5.3. FDA Radar and Jammer Intelligent Frequency Domain Power Countermeasures

In the simulation in this section, the power of the FDA radar and jammer is set to be continuously adjustable in each frequency band. The other simulation parameters are set to: $l{r}_{actor}=0.01$, $l{r}_{critic}=0.01$, $E=600000$, $T=25$, $hidden dimension=64$, $\gamma =0.95$, $\tau =0.01$, $N_{total}=10000$, $N_{\min }=4000$, $N=1024$, and ${\sigma }^2=0.05$.

The allocation of the transmitted power of the FDA radar and jammer in the initial episode in the frequency–time two-dimensional plane is shown in Figure 14.

As shown in Figure 14, the FDA radar and jammer evenly and approximately allocate the power throughout the spectrum in the initial episode. At the same time, the FDA radar and jammer explore the electromagnetic environment. Each interaction changes the transmission power allocation to seek the optimal power allocation strategy. To compare the effect of improving the power allocation strategy of the FDA radar and jammer before and after training, the power allocation scheme of the FDA radar and jammer in the last episode is given, as shown in Figure 15.

By observing Figure 15, we can find that after multiple rounds of interaction, the power allocation of the jammer is closely related to the target frequency response. The frequency band with a significant target frequency response has more jamming power, while the one with a small target frequency response has less. The power allocation of the FDA radar is also related to the target response characteristics. The transmitting power is mainly concentrated in the frequency band with the maximum target response, and the power of each frequency band is closely related to the target RCS. At the same time, the FDA radar considers exploration and utilization, allocating transmitting power in other frequency bands with a small probability of exploring whether the jammer can realize frequency domain perception and jamming. Therefore, the jammer’s power allocation should be related to the target response characteristics, and the jammer concentrates power in the frequency band with significant target response characteristics. The FDA radar concentrates most of its energy in the frequency band with the more significant target response characteristics while considering exploring the electromagnetic environment.

In [30], the authors use the neural fictitious self-play (NFSP) algorithm to propose that in the frequency domain confrontation between the radar and jammer, the jammer concentrates power in the frequency band with the most significant target response, and the radar concentrates transmission power in the frequency band with the second largest target response. Based on the limited interactions between radar and jammer, this conclusion was drawn in [30]. In practice, the sequence interaction process between the radar and jammer is approximately infinite. Once the radar fixed its power in a particular frequency band, the jammer must also aim its power at this frequency band. In the interaction process between the FDA radar and jammer, both can sense the electromagnetic environment and allocate the transmitting power in real time; so, it is an excellent strategy to correlate the transmitting power allocation with the target RCS. They can also explore the strategy space with a small probability of sensing whether the electromagnetic environment model has changed. The return obtained by the FDA radar is SJNR, and the return obtained by the jammer is -SJNR. The trend of the two returns with the episodes is shown in Figure 16.

It can be seen from Figure 16 that the gains of the FDA radar and jammer are the losses of each other. They continuously optimize their power allocation strategy through interaction with the electromagnetic environment, learn the electromagnetic environment information through continuous sequential decision making, and finally find their optimal power allocation strategy. At the same time, we can find that when the FDA radar transmitting power and jamming power change dynamically, the convergence time of the two will be significantly prolonged. Because the electromagnetic environment changes dynamically, it is difficult for the FDA radar and jammer to learn a fixed policy. Hence, their cognition of the electromagnetic environment is a probability function, and their goal is to maximize their profit. As shown in Figure 15, the power allocation of the FDA radar is related to the target frequency response characteristics, and the jamming power allocation is related to the power allocation of the FDA radar. When the FDA radar fully learns the target frequency response characteristics, both benefits converge and achieve the maximum expected benefit. Therefore, one way to shorten the convergence time is to input the target frequency response characteristics as a priori knowledge to the FDA radar.

6. Conclusions

In electronic warfare, the confrontation between the radar and the jammer is a sequence decision with multiple rounds of interaction. This paper aims to solve the policy optimization problem of the FDA radar and the jammer in the process of intelligent frequency domain power countermeasures. To accurately describe the intelligent countermeasure relationship between the FDA radar and the jammer, the sequential decision problem of the FDA radar and jammer is mapped to MARL, and the MADDPG algorithm is used to find the optimal power allocation strategy for the FDA radar in the case of a dynamic electromagnetic environment. In the training process, CTDE is used to accelerate the convergence of the algorithm. The simulation results demonstrate the effectiveness of the resulting power allocation strategy for the FDA radar, regardless of whether the jammer is intelligent.