Reinforcement Learning Based Dual-UAV Trajectory Optimization for Secure Communication

Abstract

Unmanned aerial vehicles (UAV) can serve as aerial base stations for users due to their flexibility, low cost, and other characteristics. However, due to the high flight position of UAVs, the air-to-ground (ATG) channels usually dominate with line-of-sight (LoS), which can be easily eavesdropped by multiple eavesdroppers. This poses a challenge to secure communication between UAVs and ground users. In this paper, we study a UAV-aided secure communication in an urban scenario where a legitimate UAV Alice transmits confidential information to a legitimate user Bob on the ground in the presence of several eavesdroppers around it and a UAV Jammer sends artificial noise to interfere with the eavesdroppers. We aim to maximize the physical layer secrecy rates in the system by jointly optimizing the trajectories of UAVs and their transmitting power. Considering the time-varying characteristics of channels, this problem is modeled as a Markov decision process (MDP). An improved algorithm based on double-DQN is proposed in the paper to solve this MDP problem. Simulation results show that the proposed algorithm can converge quickly under different environments, and the UAV transmitter and UAV jammers can find the optimal location correctly to maximize the information secrecy rate. It also shows that the double-DQN (DDQN) based algorithm works better than the Q-learning and deep Q-learning network (DQN).

1. Introduction

Due to their flexibility and low cost, UAVs are promising to provide reliable data transmission services as mobile nodes in various scenarios in the next-generation (6G) network [1]. For example, UAVs can serve as a communication node of intelligent city transportation systems for traffic management and monitoring. In case of emergencies that lead to partial blockage or paralysis of communication systems, UAVs can be deployed as mobile air communication nodes in these areas to help ground base stations provide communication [2,3]. Compared with conventional communication that suffers from shadowing and multi-path fading, the channel of UAV experiences LoS fading, which allows the UAV to provide better communication quality and larger communication coverage with lower energy consumption [4]. However, due to the broadcast characteristics of the wireless channel, the privacy and security of UAV communication are challenging [5,6]. Although traditional key encryption techniques can improve the security of communication, they are computationally complex and require additional resource overhead for keys. Different from traditional methods, physical layer security (PLS) uses secrecy rate as an evaluation criterion and is considered to be a promising technology. Currently, the main research directions of UAV-assisted physical layer security techniques include mmWave communication, trajectory design approach, resource allocation and others [5,7,8]. Trajectory design approach, one of the most popular research directions, has achieved some meaningful research results [9,10,11,12,13]. Trajectory design approach is an optimization problem that can be solved by mathematical tools such as convex optimization. However, the optimization problem is usually non-convex, which makes the problem difficult to solve directly. Therefore, the original optimization problem is often converted into a convex optimization problem or multiple sub-convex optimization problems. For example, Zhang et al. [9] proposed a model in which a UAV serves as a communication node (Alice) to transfer data to legitimate users on the ground. The aim of the model is to maximize the average secrecy rate by jointly optimizing the UAV’s trajectory and the transmit power of the legitimate transmitter. Then, they used the block coordinate descent and successive convex optimization methods to solve the optimization problem. Cai et al. [10] also divided the non-convex optimization problem into two sub-nonconvex optimization problems. Then they used the successive convex approximation (SCA) [14] algorithm and block coordinate descent to get a good approximate solution.

However, most conventional optimization methods can only achieve a local optimal result. When the number of optimization variables of the UAV increases and the environment becomes more complex, the nonconvex problem will be more difficult to solve and the algorithm may not converge [4].

With the rapid development of artificial intelligence, techniques such as deep learning (DL) and reinforcement learning (RL) have proven to be effective in solving high complexity optimization problems, such as V2V communications, UAV trajectory optimization, and resource allocation, which are difficult to solve with traditional methods [15,16,17,18,19,20,21]. In addition, RL is attracting more and more scholars’ interest because it can solve complex optimization problems with a simplified model but requires no training samples like deep learning. Zhang [15] successfully used Q-learning to optimize the trajectories of UAV under the condition of multiple users and one eavesdropper. Zhang et al. [17] solved the trajectory optimization problem by modeling the problem as a constrained Markov decision process (CMDP) using the safe-DQN model. Fu et al. [16] proposed the curiosity-driven DQN algorithm C-DQN, which constringes faster, and the simulation results indicated that the system is more stable at the trajectory corners. In addition, for the trajectory optimization problem under multi-UAV cooperation, Zhang et al. [18] proposed a continuous action attention MADDPG (CAA-MADDPG) model that adopts centralized training and distributed execution, and the algorithm achieved good results in the simulation.

Regarding the optimization problem of physical layer security, it is important to maximize the information secrecy rate. At the same time, the energy consumption of UAVs is also a problem to consider due to the limited energy. In addition, due to the multiple eavesdroppers on the ground, a single transmitting UAV may be difficult to satisfy the secure communication in the system, so the help of jamming UAVs can effectively solve this problem. Therefore, in this paper, we consider a multi-agent model. The existing multi-agent algorithm not only requires each agent to maintain its own neural network model but also requires an airborne computing platform to train the critic neural network with all agents’ input.

A system with a small number of UAVs does not need such a complex system because the high cost of an airborne computing platform. Therefore, we propose an optimization algorithm under the situation of a single transmitter and a single jammer. The algorithm is based on double-DQN [22]. For ease of illustration, this paper adopts a model in which two UAVs work in cooperation: one UAV (Alice) transmits information and the other UAV (Jammer) sends jamming signals. In this algorithm model, each UAV is an agent that maintains its own DDQN network and executes actions based on observations and information transmitted from other UAVs. Results indicate that the algorithm in this paper has good scalability, and performs well in the case of different numbers of eavesdroppers (Eves). Our goal is to maximize the information secrecy rate by optimizing both the UAV trajectory and the UAV transmit power. For readers convenience, The main notations used in the paper are summarized in

Table 1. Part of notations.

Notation	Definition
T	Total flight time
$\tau$	Number of time slots
t	Index of time slots
$P{L}_{\xi }\left(t\right)$	Path loss of ATG
$FSPL\left(t\right)$	Free space path loss
$r\left(t\right)$	Distance between the UAV and the ground receiver
$h\left(t\right)$	Height of the UAV
f	Carrier frequency of the system
c	Speed of light
${\eta }_{\xi }$	Value of excessive path loss
$\xi$	Propagation group
$\overline{PL\left(t\right)}$	Channel path loss
$P_{LoS}\left(t\right)$	Probability of LoS
$P_{NLoS}\left(t\right)$	Probability of NLoS
$\theta$	Angle between the UAV and the ground user
$P{L}_{A,B}\left(t\right)$	Path loss between Alice and Bob
$P{L}_{A\left(J\right),e}\left(t\right)$	Path loss between Alice (Jammer) and Eve
$P{L}_{J,A}\left(t\right)$	Path loss between Jammer and Alice
$R_{A,B}\left(t\right)$	Data rate of legitimate channel
$R_{A,e}\left(t\right)$	Data rate of wiretap channel
$R_c\left(t\right)$	Secrecy rate of the system
$N_0$	Power of the natural Gaussian noise
E	Number of Eves
$p_A\left(t\right)$	Transmitting power of Alice
$p_J\left(t\right)$	Artificial noise power generated by Jammer
$\mathcal{S}$	State space of MDP
$\mathcal{A}$	Action space of MDP
$\mathcal{P}$	State-transition probability of MDP
$\mathcal{R}$	Reward space of MDP

.

The rest of this paper is organized as follows. Section 2 introduces the system model. In Section 3.1 and Section 3.2, we present the problem formulation and model the optimization problem as an MDP. In Section 3.3, we present the DQDN-based approach for multi-UAV networks. Section 4 discusses the simulation results. Finally, the paper concludes in Section 5.

2. System Model

2.1. Network Model

As shown in Figure 1, we mainly focus on UAV-aided secure communication in an urban scenario. In this model, the UAV Alice serves as a mobile base station to provide a reliable data transmission service to the legitimate user Bob, with the assumption that there are E eavesdroppers Eve on the ground, which eavesdrop on the information between Alice and Bob. The UAV Jammer serves to send artificial noise to confuse Eves and can move as required. We suppose there are single or multiple mobile eavesdroppers and several stationary eavesdroppers in the system. There are two types of eavesdroppers: active and passive. The former actively send signal to attack the legitimate user, so it can be easily detected [23]. The latter does not send signals, only eavesdrops. The location of the passive eavesdropper can be detected from the local oscillator power leaked from its radio frequency (RF) front end [24]. Therefore, in this paper, we assume the UAV can detect the position of Bob and all Eves, and the UAVs can communicate with each other to transmit the position information. Therefore, the UAV can acquire the position information of Bob and all Eves, which guarantees that the RL algorithm can learn the optimum position. The movement range of Alice and Jammer in the model is in two separate two-dimensional planes in the air.

2.2. Mobility Model

The total flight time T can be divided into $\tau$ time slots, indexed by t, t = 1, 2, …, $\tau$. Considering the three-dimensional coordinate system, the coordinates of Alice and Jammer at the beginning of moment t are denoted as $s_A\left(t\right)=\left[x_A\left(t\right),y_A\left(t\right),z_A\left(t\right)\right]$ and $s_J\left(t\right)=\left[x_J\left(t\right),y_J\left(t\right),z_J\left(t\right)\right]$; $s_e\left(t\right)=\left[x_e\left(t\right),y_e\left(t\right),z_e\left(t\right)\right]$, $e\in \left\{1,2,...,E\right\}$ denote the coordinates of each Eve. The coordinate of a stationary Bob can be denoted by $s_b=\left[x_b\left(t\right),y_b\left(t\right),z_b\left(t\right)\right]$.

2.3. Transmission Model

The channel between the UAV and the user on the ground can be regarded as an ATG channel. As shown in Figure 2, the transmission path loss suffered by the ATG channel consists of two parts: one part is the free space path loss between the UAV and the receiver on the ground, and the other part is the excessive path loss caused by the buildings and trees in the city. Therefore, the path loss of ATG per unit time slot can be denoted as [25].

$$P{L}_{\xi }\left(t\right)=FSPL\left(t\right)+{\eta }_{\xi }\left(t\right).$$

(1)

In (1), $FPSL\left(t\right)$ denotes the free space path loss, $FPSL\left(t\right)=20{log}_{}\left(d\left(t\right)\right)+20{log}_{}\left(f\right)+20{log}_{}\left(4\pi /c\right)$, $d\left(t\right)=\sqrt{h{\left(t\right)}^2+r{\left(t\right)}^2}$, $r\left(t\right)$ is the distance between the UAV and the ground receiver, $h\left(t\right)$ is the height of the UAV above the ground, f denotes the carrier frequency, c is the speed of light, ${\eta }_{\xi }$ is the value of excessive path loss, and $\xi \in \left\{NLoS,LoS\right\}$ refers to the propagation group. Assuming that the transmitting and receiving antennas are isotropic, the channel path loss can be written as

$$\overline{PL\left(t\right)}=P_{LoS}\left(t\right)\times P{L}_{LoS}\left(t\right)+P_{NLoS}\left(t\right)\times P{L}_{NLoS}\left(t\right),$$

(2)

where $P_{LoS}$ and $P_{NLoS}$ are the probability of LoS and NLoS, respectively, $P_{NLoS}=1-P_{LoS}$, $P_{LoS}$ can be approximated to a simple Sigmoid function [25],

$$P_{LoS}\left(t\right)=\frac{1}{1+aexp(-b\left[\theta -a\right])}.$$

(3)

In (3), a and b are constants that depend on the specific environment (e.g., urban, suburban, or rural), and the value of $\theta$ can be calculated by

$$\theta =180{\pi }^{-1}arctan\left(\frac{z\left(t\right)}{\sqrt{{(x\left(t\right)-x_b\left(t\right))}^2+{(y\left(t\right)-y_b\left(t\right))}^2}}\right).$$

(4)

Supposing the channel path loss between Alice and Bob is $P{L}_{A,B}\left(t\right)$, and the channel path loss between Alice and Jammer and Eve is $P{L}_{A,e}\left(t\right)$ and $P{L}_{J,e}\left(t\right)$, $e\in \left\{1,2,...,E\right\}$, then the formula of channel path loss can be obtained by combining (1) and (2)

$$P{L}_{A,B}\left(t\right)=20{log}_{}\frac{\left(4\pi f{d}_{A,B}\right)}{c}+{\eta }_{LoS}{P}_{LoS}+{\eta }_{NLoS}{P}_{NLoS},$$

(5)

$$P{L}_{A\left(J\right),e}\left(t\right)=20{log}_{}\frac{\left(4\pi f{d}_{A\left(J\right),e}\right)}{c}+{\eta }_{LoS}{P}_{LoS}+{\eta }_{NLoS}{P}_{NLoS},$$

(6)

$$P{L}_{J,A}\left(t\right)=20{log}_{}\frac{\left(4\pi f{d}_{J,A}\right)}{c}+{\eta }_{LoS}{P}_{LoS}+{\eta }_{NLoS}{P}_{NLoS}.$$

(7)

In (5)–(7), $\left[{\eta }_{LoS},{\eta }_{NLoS}\right]$ are constants that are set according to the environment [25]; the specific parameters are given below. We assume that the instantaneous transmitting power of Alice and Jammer are $p_A\left(t\right)$ and $p_J\left(t\right)$ and the information rates of Alice-to-Bob and Alice-to-Eve are $R_{A,B}\left(t\right)$ and $R_{A,e}\left(t\right)$, $e\in \left\{1,2,...,E\right\}$. Then, the achievable rate of the channel can be given by

$$R_{A,B}\left(t\right)=B{log}_2(1+\frac{p_A\left(t\right)}{P{L}_{A,B}\left(t\right)(N_0+p_J\left(t\right)/P{L}_{J,A}\left(t\right))}),$$

(8)

$$R_{A,e}\left(t\right)=B{log}_2(1+\frac{p_A\left(t\right)}{P{L}_{A,e}\left(t\right)(N_0+p_J\left(t\right)/P{L}_{J,e}\left(t\right))}),$$

(9)

where $N_0$ is the power of the natural Gaussian noise. According to [26], the secrecy rate from Alice to Bob can be given by the difference between the achievable rate of the legitimate channel and the channel:

$$R_c\left(t\right)={\left[\sum _{e=1}^E\left[R_{A,B}\left(t\right)-R_{A,e}\left(t\right)\right]/E\right]}^{+},$$

(10)

where ${\left[x\right]}^{+}\triangleq max\left[0,x\right]$.

3. System Problem Formulation and Markov Decision Process

3.1. Problem Formulation

The optimization goal of the model is to optimize the trajectory and transmitting power of the UAV in a limited space to achieve the maximum information secrecy rate. Therefore, the optimization problem can be described as

$$\begin{array}{cc}\hfill & \underset{S_A,S_J,p_A,p_J}{max}{R}_c\left(t\right)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.X^{min}\le x_A\left(t\right),x_J\left(t\right)\le X^{max}\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill & Y^{min}\le y_A\left(t\right),y_J\left(t\right)\le Y^{max}\hfill \end{array}$$

(11b)

$$\begin{array}{cc}\hfill & Z^{min}\le z_A\left(t\right),z_J\left(t\right)\le Z^{max}\hfill \end{array}$$

(11c)

$$\begin{array}{cc}\hfill & P_A^{min}\le p_A\left(t\right)\le P_A^{max}\hfill \end{array}$$

(11d)

$$\begin{array}{cc}\hfill & P_J^{min}\le p_J\left(t\right)\le P_J^{max}\hfill \end{array}$$

(11e)

where (11a)–(11c) give the boundaries of the limited area for UAV movement, and (11d) and (11e) give constraints on the transmission power of Alice and Jammer.

3.2. Markov Decision Process

In a multi-UAV environment, the decisions of each UAV are influenced by the other ones. For each UAV, the optimization problem can be modeled as a Markov decision process (MDP). Therefore, the optimization problem of multi-UAV can be regarded as an extension of the MDP and can be formalized by a tuple $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{R}$ is the reward space, and $\mathcal{P}$ is the transfer probability. In a time slot, each agent has coordinate $s^i\left(t\right)\in \mathcal{S}$ and takes action $a^i\left(t\right)\in \mathcal{A}$ based on a certain policy. Then a new state $s^{i+1}\left(t\right)$ is generated with transition probability $\mathcal{P}$. The reward that UAV i receives from the environment is defined as $r^i\left(t\right)\in \mathcal{R}$. The specific meanings of $\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}$ are expressed as follows.

• Each UAV can be regarded as an agent. An independent model is used to solve the MDP problem for each UAV. However, because the agent can be influenced by other agents, from its own point of view, other agents make the environment unstable. Therefore, UAVs need to communicate during the flight to inform each other of the position information. Then they take action and get environmental rewards according to the position information of all agents. In this paper, we only consider two agents, named Alice and Jammer. The complete model structure is shown in Figure 3.
• $\mathcal{S}$ is the state space; $\mathcal{S}$ includes the location of each UAV, Bob, and Eves. The coordinates of Alice and Jammer are described as $s_A\left(t\right)=\left[x_A\left(t\right),y_A\left(t\right),z_A\left(t\right)\right]$, $s_J\left(t\right)=\left[x_J\left(t\right),y_J\left(t\right),z_J\left(t\right)\right]$; the coordinate of Bob is $s_b\left(t\right)=\left[x_b\left(t\right),y_b\left(t\right),z_b\left(t\right)\right]$. The coordinate of Eve e is $s_e\left(t\right)=\left[x_e\left(t\right),y_e\left(t\right),z_e\left(t\right)\right]$, $e\in \left\{1,2,\dots ,E\right\}$.
• $\mathcal{A}$ is the action space, which contains the moving speed of the UAV at each moment and its transmitting power. To reduce the complexity of the network model, we discretize the speed into a set $\mathcal{V}=\left\{v_1,v_2,...,v_I\right\}$. The speed of Alice and Jammer in each time slot is denoted as $V_{Ai}\left(t\right)$, $V_{Ji}\left(t\right)$, and the transmitting power is $p_A\left(t\right)$ and $p_J\left(t\right)$, $p_{min}\le p_A\left(t\right)\le p_{max}$, $p_{min}\le p_J\left(t\right)\le p_{max}$.
• $\mathcal{P}$ denotes the state-transition probability. As it is difficult to predict $\mathcal{P}$, we adopt the model-free RL method to address the above MDP problem.
• $\mathcal{R}$ is the reward space. For all UAVs, the reward obtained in each time slot consists of the scenario penalty, the energy penalty, the information secrecy rate reward, and the distance reward. The information secrecy rate reward is equal for each agent. For the scene penalty, due to the complex arrangement of tall buildings and vegetation in the city, the free flight space of the UAV is limited, so we need to give the UAV an environmental penalty when the UAV wants to fly away from the limited area. The coordinate of the UAV i is denoted as $[x_i\left(t\right),y_i\left(t\right),z_i\left(t\right)]$, $i\in \left\{A,J\right\}$. Then the scenario penalty $U_i$ is

  $$U_i=\left\{\begin{array}{cc}-C;\hfill & x_i<X_{min}or{x}_i>X_{max}\hfill \\ -C;\hfill & y_i<y_{min}or{y}_i>y_{max}\hfill \\ 0;\hfill & else\hfill \end{array}\right.$$

  (12)

  where C is a constant. Since the battery capacity of the UAV is finite, it is necessary to consider the energy cost of the UAV without wireless charging. The energy consumption penalties of Alice and Jammer are defined as $E_A$ and $E_J$. The transmitting powers of Alice and Jammer in the current time slot are denoted as $p_A$ and $p_J$. Therefore, the energy penalty can be given by

  $$E_A=-k_E^A{p}_A,$$

  (13)

  $$E_J=-k_E^J{p}_J,$$

  (14)

  where $k_E^A$ and $k_E^J$ are hyperparameters. In the initial state of the model, Alice may be far away from Bob, which makes the information secrecy rate very small. The too-small reward leads to the model being unable to learn the features of the environment effectively. To avoid this, we introduce a distance penalty and define the distance penalty for Alice and Jammer as $D_A$ and $D_J$. We use the average of the distances as the reward, as shown in (16). The distance reward according to the Euclidean distance from the UAV to the target can be expressed as

  $$D_A=-{\lambda }_D^A{∥s_A-s_b∥}_2,$$

  (15)

  $$D_J=-\frac{{\lambda }_D^J}{E}\sum _{e=1}^E{∥s_J-s_e∥}_2,$$

  (16)

  where ${\lambda }_D^A$, ${\lambda }_D^J$ are the distance reward factors of Alice and Jammer. Thus, the total reward function for each UAV can be given by

  $$r_A=R_c+U_A+E_A+D_A,$$

  (17)

  $$r_J=R_c+U_J+E_J+D_J,$$

  (18)

  where $r_A$ and $r_J$ are the reward for Alice and Jammer separately.

3.3. Double DQN

The trajectory optimization problem of each UAV can be modeled as a Markov process with the knowledge of other changing coordinates. We use a separate DDQN network for each UAV and assume that each UAV can acquire the positions of other UAVs through real-time communication and estimate the coordinate information of Eve through the camera. Before describing the DDQN algorithm, let us begin with the DQN algorithm [27]. The objective of discrete reinforcement learning is to fit a Q table, but with the increase in state space $\mathcal{S}$ and action space $\mathcal{A}$, the Q table will have a dimensional explosion problem. Therefore, the main idea of DQN is using a deep neural network (DNN) $Q(s,a,w_Q)$ to approximate the Q table, where s is the state and a is the action. With the reward $r_Q$ from environment, the target Q value $y_Q$ can be expressed as [27]

$$y_Q=r_Q+\gamma \underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime },w_Q),$$

(19)

where $\gamma \in \left[0,1\right]$ is the discount factor, and $s^{\prime }$ and $a^{\prime }$ are the state and the action in the next time slot. DQN has an experience replay buffer ${\mathcal{M}}_Q$ to get samples and uses the mean square error as the loss function $Loss{\left(w\right)}_Q$,

$$Loss{\left(w\right)}_Q=\mathbb{E}{\left[(y_Q-Q(s,a,w_Q))\right]}^2.$$

(20)

However, using $Q(s,a,w_Q)$ directly for calculating the target Q value $y_Q$ will cause the $Loss{\left(w\right)}_Q$ to change at each time slot, in which case the $Q(s,a,w_Q)$ will diverge if the environment returns a large reward. At the same time, the max function in (19) makes the target Q value overestimated [22].

Like DQN, double DQN also uses DNN to approximate the Q function.

However, DDQN uses a target Q network to reduce overestimation, which can enhance the stability of the model. The same as DQN, DDQN also adopts an experience replay technique to utilize the data efficiently. For each UAV, the predict Q network is denoted as $Q(s,a,w)$ and the target Q network is denoted as $Q^{\prime }(s,a,w^{\prime })$; s is the joint state of the UAV, a is the action selection, and r is the reward after executing the action given by

$$r=R_c+U+E+D.$$

(21)

After the communication between UAVs to acquire the new joint position information $s^{\prime }$ in the next time slot, the target Q value can be expressed by the symbol y:

$$y=\left\{\begin{array}{cc}r\hfill & \mathrm{terminal}\hfill \\ r+\gamma Q^{\prime }(s^{\prime },{argmax}_{a^{\prime }}Q(s^{\prime },a^{\prime },w),w^{\prime })\hfill & \mathrm{else}\hfill \end{array}\right.$$

(22)

where $\gamma \in \left[0,1\right]$ is the discount factor, and $a^{\prime }$ is the action in the next time slot. The same as DQN, DDQN also uses the mean square error as the loss function $Loss\left(w\right)$, and the samples come from the experience replay buffer $\mathcal{M}$

$$Loss\left(w\right)=\mathbb{E}{\left[(y-Q(s,a,w))\right]}^2,$$

(23)

Each UAV updates parameters w of $Q(s,a,w)$ by gradient back-propagation,

$$w=w-\alpha {\nabla }_{w}Q(s,a,w)(y-Q(s,a,w)),$$

(24)

where $\alpha$ is learning rate, $0<\alpha <1$. When the number of updates of $Q(s,a,w)$ reaches a certain value, w of $Q(s,a,w)$ is copied to $w^{\prime }$ in $Q^{\prime }(s,a,w^{\prime })$. The Q and Q′ networks of different agents are distinguished by the subscript i, where $i\in \{A,J\}$. The procedures of the algorithm are illustrated in Algorithm 1.

Algorithm 1 A DDQN-based optimization algorithm for dual UAVs.

Input:  episode K, total flight time T, vector dimension n, action space $\mathcal{A}$, action noise N, batch size m, network parameters update frequency $\phi$, hyperparameters $\alpha$, $\gamma$, and $ϵ$. Output:  Network parameters w, $w^{\prime }$.   1: Initialize w in $Q_i(s,a,w)$, $i\in \{A,J\}$, then $w^{\prime }⟵w$.   2: Initialize experience replay buffer $\mathcal{M}$ with capacity m.   3: for $k=1$ $to$ K do   4:    for $t=1$ $to$ T do   5:        Initialize s to the first state of the current agent;   6:        Select action a from Qi(s, a, w) using the ϵ—greedy method;   7:        UAV i executes action a in state s.   8:        Each UAV transmits the state to other ones so that each UAV gets a new joint state $s^{\prime }$, the reward r by (21) and termination sign $is\_end$;   9:        Each UAV stores ($s,a,r,s^{\prime },is\_end$) into its own experience replay buffer $\mathcal{M}$, $s⟵s^{\prime }$;   10:      Sampling m items from replay buffer $\mathcal{M}$ and calculating y values by (22);   11:      Update the parameters of $Q_i(s,a,w)$ by (23) and (24);   12:      if (t % $\phi$ = 0) then   13:         Update $Q_i^{\prime }(s,a,w^{\prime })$ parameters $w^{\prime }⟵w$;   14:      end if   15:      if ($is\_endistrue$) then   16:         break;   17:      end if   18:    end for   19: end for   20: return w, $w^{\prime }$

4. Simulation Results and Analysis

In this section, we analyze the performance of the algorithm based on DDQN and discuss the simulation results under different situations. The algorithm is implemented based on the PyTorch framework, and the model is trained on a server with a GTX 1080ti. We suppose that Alice, Bob, Jammer, and Eves are randomly distributed in a 500 m × 500 m environment. One or more Eves move along the road in the city around the Bob, and the rest of the Eves are on the sides of the roads and in the buildings. Because the model quantizes the speed of the UAV, speed can be converted into step length per unit time slot. The step length of the UAV in this model is hyperparametric, and it affects the convergence performance of the algorithm, which is discussed below. To avoid the collision between UAVs, which will complicate the model, the flight heights of Alice and Jammer are fixed at 15 m and 20 m, and UAVs can move in a 500 × 500 area with freedom. System carrier frequency is 5 GHz, Alice and Jammer’s transmitting power are $p_A=10$ dBm and $p_J=20$ dBm, the power of natural Gaussian noise in the channel is $N_0=-97$ dBm, coefficients in (13) and (14) are set as $k_E^A=0.05$, $k_E^J=0.05$, and the learning rate is $\alpha$ = 0.001. In (5) and (6), $\left[{\eta }_{LoS},{\eta }_{NLoS}\right]$ is set as $\left[1,20\right]$. By the formula in [25], we can calculate a = 55, b = 5. The coordinate position of Bob is (280, 250, 0), which is the center of the two-dimensional plane of the area. The step of Eve is set to 1 m.

Figure 4a,b denote the rewards versus episodes with a 90% confidence level for different initial positions of Alice and Jammer. We set the maximum step length of the UAV to 2 m, and there is only one moving Eve. From Figure 4a, we can see that the normalized average reward value increases gradually in each episode, and the curve fluctuates slightly after 200 episodes with a stable value for four initial positions of the UAV. It indicates that the algorithm can reach convergence for different initial positions of the UAV. It can also be found that the closer position Alice is to Bob, the fewer episodes to reach convergence. The convergence speed of the model can be accelerated by appropriately increasing the distance reward factor in (15) and (16). It can also reduce the fluctuation amplitude of the model in a stable state.

In Figure 4b, the trend of the four convergence curves is approximately the same, which indicates that different Jammer positions do not have too much influence on the convergence performance of the model, and the model reaches convergence before 150 episodes.

Figure 5a illustrates rewards versus episodes for different step lengths. For the situation where step lengths are greater than 1 m and less than 8 m, the convergence speed is almost the same for all four groups of simulations. The model converges around 170 episodes. However, as the step length increases, the max reward decreases and the curves become more unsmooth. When the step length is 9 m, the model reaches a steady state with a lower maximum reward. When the step length is 0.5 m, the convergence speed of the system decreases dramatically, and the system converges after 600 episodes. Compared with the other step lengths, the phenomenon is poor. As the model assumes that Eve moves in steps of 1 m, a too-short step length causes UAVs to not to reach the best point instantly. Therefore, step length determines the stability and convergence speed of the system. We recommend setting the step length between 1 m and 5 m. In Figure 5b, we compare the convergence of DDQN with DQN and Q-learning algorithms. Because the action in the scenario is discrete, we use the value-based RL algorithms as the baseline. As can be seen from Figure 5b, the DDQN-based algorithm reaches stability before 100 episodes, whereas the Q-learning algorithm reaches stability after 150 episodes and the DQN algorithm reaches stability after 200 episodes. The reward of the DDQN-based algorithm is always better than that of other discrete RL algorithms, which indicates that the DDQN-based algorithm has good convergence performance and stability.

Consider a scenario where one Eve is in a moving state; Figure 6a,b show the motion trajectories of Alice and Jammer with initial positions [(250, 250), (150, 150)], [(200, 200), (400, 150)]. The red curve denotes Alice’s trajectory, the black curve denotes Jammer’s trajectory, and the green x denotes Bob’s position. Suppose Eve moves along the street around Bob from the position of (100, 210); other Eves’ positions are shown in Figure 6a. In the beginning, Jammer and Alice approach quickly. After they meet each other, Jammer and Alice will go to the area near Bob. When Eve is close to Bob, Alice will hover away from Eve around Bob to reduce Eve’s achievable rate. It can also increase the information secrecy rate of the system. On the contrary, Jammer moves only in a 10 m × 10 m area above Bob, because Jammer needs to consider the position of all Eves. Figure 6c shows the trajectory of the UAV in the environment with only one moving Eve, and we can clearly see that Alice’s trajectory is approximately the same as that in Figure 6a,b; however, Jammer follows the moving Eve. The phenomenon illustrates that Jammer’s motion trajectory in a multi-Eve environment is influenced by all Eves. In addition, the reward curves in Figure 6a–c are approximately the same as those in Figure 4a, in which the system reaches a stable state when Alice and Jammer arrive near Bob, and then Alice and Jammer change their positions according to the environment. The results illustrate that the different initial locations of Alice and Jammer do not affect the convergence of the algorithm. Figure 6d describes the system secrecy rate curves under different RL algorithms. It can be seen that before 200 episodes, the DDQN-based algorithm has achieve the best system security rate, which is approximately 4.3 bps/Hz. However, the DQN and Q-learning have not converged yet. They can only obtain a secrecy rate of 2.5 bps/Hz. The system secrecy rate under DQN and Q-learning algorithms did not reach 4.3 bps/Hz until 300 episodes. From Figure 6d, it can be seen that the DDQN-based algorithm converges twice as fast as the other two algorithms, and if the number of UAVs is increased, the gap of convergence speed will be further expanded.

Consider the scenarios with two moving Eves or three moving Eves, as shown in Figure 7a,b. Figure 7a shows the motion trajectories of Alice and Jammer when two Eves move around Bob along the road. We can see that, although the number of moving Eves increases, Alice and Jammer still approach each other quickly at the beginning without getting too close to Eves, which is the same as the UAV’s trajectory with a single Eve. However, after the system is steady, Jammer moves in a near-elliptical trajectory near Bob, mainly following the Eve that is closest to Bob. This phenomenon indicates that more moving Eves lead to Jammer’s following target switching more frequently, thus, we introduce the average distance in the reward function to smooth the reward value in (21). Figure 7b adds a moving Eve compared to Figure 7a. In Figure 7b, Alice’s trajectory goes around the moving Eve to the left and keeps a distance from it, and the trajectory of Jammer is similar to that in Figure 7a. However, Jammer is more conservative in moving around Bob and hovers in a smaller area but is generally similar to the trajectory in Figure 7a.

5. Conclusions

In the environment with moving Eves, we model the problem as a Markov decision process and use the DDQN-based algorithm to solve it. The main objective of this study is to optimize the UAV trajectory and transmit power to maximize the information secrecy rate of the system. In this paper, the convergence of the model is discussed for UAV and Eve with different step lengths and initial positions. The simulation results show that the model has excellent convergence performance and rapid convergence speed. We also compare the core algorithm double-DQN with several discrete DRL networks. The results show that double-DQN has better stability and faster convergence speed.

The future challenge is about improving the algorithm with multiple agents. The multi-agent environment will result in a larger state space, which makes the computational complexity grow exponentially. At the same time, the data transmission between agents will also be complicated with an increase in agents, so the data transmission assignment is also an issue to be considered.