A Hierarchical Deep Reinforcement Learning Approach for Throughput Maximization in Reconfigurable Intelligent Surface-Aided Unmanned Aerial Vehicle–Integrated Sensing and Communication Network

Abstract

Integrated sensing and communication (ISAC) is considered a key technology supporting Beyond-5G/6G (B5G/6G) networks, which allows the spectrum resources to be used for both sensing and communication. In this paper, we investigate an unmanned aerial vehicle (UAV)-enabled ISAC scenario, where the UAV sends ISAC signals to communicate with multiple users (UEs) and senses potential targets simultaneously, and a reconfigurable intelligent surface (RIS) is deployed to enhance the communication performance. Aiming at maximizing the sum-rate throughput of the system, we formulate the joint optimization problem of the trajectory and the beamforming matrix of the UAV, the passive beamforming matrix of the RIS. Currently, many researchers are working on using deep reinforcement learning (DRL) to address such problems due to its non-convex nature; however, as the environment becomes increasingly complex, high-dimensional state space and action space lead to a decrease in the performance of DRL. To tackle this issue, we propose a novel hierarchical deep reinforcement learning (HDRL) framework to solve the optimization problem. Through decomposing the original problem into the trajectory optimization problem and the sum-rate throughput optimization problem, we adopt a hierarchical twin-delayed deep deterministic policy gradient (HTD3) structure to optimize them alternately. The experimental results demonstrate that the obtained system sum-rate throughputs of the proposed HDRL with an HTD3 structure are 33%, 50%, and 10% higher than those obtained by TD3, twin-TD3 (TTD3), and TD3 with hovering only (TD3HO), respectively.

1. Introduction

With the emergence of various Beyond-5G(B5G)/6G concepts [1], numerous applications requiring sensing functionalities such as wireless localization, 4D imaging, autonomous driving, and activity sensing have aroused great interest in both academia and industry. Integrated sensing and communication (ISAC), which allows wireless devices to share spectrum resources for joint radar sensing and data communications, has been regarded as one of the key technologies to support B5G/6G networks [2]. ISAC provides significant advantages in many scenarios with high spectrum utilization efficiency. In addition, with the deployment of multiple-input-multiple-output (MIMO), which plays an important role in improving the performance of ISAC systems, ISAC networks could perform sensing and communication works synchronously with ultra-high sensing accuracy and communication performance [3]. However, the existence of severe channel degradation caused by complex environmental factors such as obstruction from trees and buildings results in unsatisfactory performance [4]. Thanks to the assistance of the reconfigurable intelligent surface (RIS), which adjusts the amplitude and phase shift of incident signals, the communication or sensing capabilities are significantly facilitated even when the direct path is blocked [5,6,7].

Unmanned aerial vehicles (UAVs) are regarded as a new type of aerial ISAC platform due to their flexibility and high line-of-sight (LoS) probability [8], whereas conventional terrestrial ISAC networks can only provide communication and sensing services within a limited distance, leading to a poor sensing performance. Many recent studies have demonstrated the superiority of UAV-ISAC systems (e.g., [2,9,10,11]). Lyu et al. [2] studied a UAV-enabled ISAC system and jointly optimized the beamforming matrix and trajectory of the UAV for the maximum average sum-rate throughput. Liu et al. [9] studied the joint optimization of task scheduling, transmit power allocation, and trajectory for maximum average sum-rate throughput of a UAV-aided ISAC system; notably, they considered 3D flight parameters to better utilize the flexibility of the UAV. Meng et al. [10] proposed a periodic ISAC system which is able to balance resources effectively in multi-user, multi-objective scenarios. Zhang et al. [11] studied a multi-UAV-assisted ISAC system; by utilizing the advantages of multi-UAVs, the performance of the UAV-ISAC system was further improved. Additionally, some studies have focused on secure transmission in UAV-ISAC networks (e.g., [12,13]). Wu et al. [12] utilized UAV-assisted ISAC to locate users and ensure secure transmission. Liu et al. [13] jointly optimized the scheduling policy, transmit power, and trajectory of the UAV to maximize the secure rate in the UAV-assisted ISAC system. Although these works provided outstanding results, the above studies still face many challenges; to be specific, most of them only considered ideal channel conditions (e.g., [2,9,10,12,13]), while imperfect channel conditions must be considered in specific UAV-ISAC applications. Moreover, the use of new technologies (e.g., RIS) provides further opportunities for improvement in the performance of UAV-ISAC systems, but many studies have not taken this into consideration.

RIS is a promising solution which can be used to enhance both communication and sensing, and improve channel quality affected by environmental factors. Some research studies have focused on RIS-aided UAV-ISAC networks (i.e., [14,15,16]). Yu et al. [16] studied the ISAC via RIS-UAV, where the UAV is not used as an ISAC platform, but rather as a carrier for RIS. Zhang et al. [15] investigated secure transmission for an RIS-aided UAV-ISAC network; they jointly designed the transmit power allocation, the scheduling policy, the passive beamforming matrix of RIS, and the trajectory and velocity of the UAV to maximize the average achievable rate (AAR). Wu et al. [16] jointly optimized the phase shift, the UAV’s trajectory, the beamforming matrix of the UAV, and the scheduling policy to maximize the weighted sum of average sum-rate and sensing signal-to-noise ratio (SNR). These studies have demonstrated the performance improvement of RIS on UAV-ISAC networks through reasonable comparison schemes. However, most works adopted convex optimization to solve approximate solutions to these problems, which may lead to local optimal solutions. On the one hand, such problems involve highly non-convex and intricate multi-variable coupling; on the other hand, channels affected by the environment are always time-varying, and traditional algorithms cannot adapt well to this characteristic. Hence, new approaches should be proposed to overcome the shortcomings of existing solutions.

Fortunately, the development of the deep learning field provides new solutions. Deep reinforcement learning (DRL), which combines deep neural networks (DNNs) and reinforcement learning (RL), is suitable for resolving the time-segmented tasks by abstracting them as a Markov decision process (MDP). DRL overcomes the complex time-varying characteristic of the environment, and has been widely used in research on UAVs [17,18,19,20,21]. Zhang et al. [17] proposed a safe deep Q-learning network (DQN) to ensure energy-efficient secure video streaming in UAV-enabled wireless networks. Miao et al. [18] applied deep deterministic policy gradient (DDPG) to optimize the energy efficiency of UAV-assisted mobile edge computing (MEC) video transmission. Yan et al. [19] combined long short-term memory (LSTM) with DDPG and jointly optimized the trajectory of the UAV and the task offloading strategy to minimize the system delay. Yao et al. [20] proposed a TD3 framework with Lyapunov optimization, which ensures the stability of the learning process in RIS-aided UAV networks. Wang et al. [21] proposed a TD3 approach for minimizing the completion time of data collection of UAV-based Internet of Things (IoT). These works provide sufficient conclusions and simulation results to demonstrate that DRL is an effective solution for solving UAV optimization problems. Other studies have focused on the application of DRL on optimization problems in ISAC (e.g., [22,23]). Liu et al. [22] proposed a DRL approach for an RIS-aided secure ISAC system; although they employed DRL in adjusting the transmit beamforming matrix and passive beamforming matrix of the RIS, they only considered the traditional ground ISAC scenario. Qin et al. [23] applied DRL to ISAC scenarios with multiple UAVs; nonetheless, they only considered the UAVs with a single antenna while ignoring the balance of communication and sensing resources. Generally speaking, few studies have considered a complete solution scheme of DRL applied to RIS-aided UAV-ISAC networks, which is a non-convex problem with high-dimensional state space and action space.

Even though DRL approaches have good adaptability to environments with high complexity, the use of DRL still faces challenges. On the one hand, normal DRL approaches suffer from inefficient exploration. On the other hand, large action space and state space may lead to low solving performance, and it is easy to fall into local optimal solutions [24]. Some studies improved the structure of DRL to alleviate these issues [25,26]. Tham et al. [25] proposed a twin-delayed deep deterministic policy gradient (TTD3) approach, where two agents are set to perform trajectory optimization and transmission performance optimization synchronously. Yang et al. [26] proposed a hybrid-proximal policy optimization (HPPO) approach to solve the optimization of mixed variables in UAV networks. These works adopted a parallel structure DRL; they first split the action space and state space, and then adopted multiple agents to solve the subproblems separately. However, such a structure ignores the correlation between the agents used to solve subproblems, leading to some limitations in solving performance. Fortunately, hierarchical deep reinforcement learning (HDRL), which decomposes the original problem into several subproblems, was proposed to resolve these problems. Not only can HDRL reduce the complexity of the problem but also it considers the correlation between hierarchical agents [27]. Susarla et al. [28] proposed a hierarchical deep Q-learning approach to consider mmWave beams with different beam width resolutions in a hierarchical manner in the beam pair alignment problem. Ren et al. [27] studied a UAV-assisted MEC scenario where the whole problem was decomposed into the location optimization problem and offloading optimization problem, and a hierarchical trajectory optimization and offloading optimization (HT3O) method was proposed to solve them alternately.

Although there are many studies focusing on UAV-ISAC networks, few studies have considered the RIS-aided UAV-ISAC networks and how to efficiently balance communication and sensing resources in such networks. Furthermore, there is still a lack of efficient DRL methods to solve such non-convex problems with high-dimensional state space and action space. Motivated by these challenges, we propose an RIS-aided UAV-ISAC system, where the UAV senses multiple potential targets and communicates with multiple users (UEs) synchronously within ISAC signals; the original problem is decomposed and solved with HDRL. Our detailed contributions can be listed as follows:

• We investigate a RIS-aided UAV-ISAC system, where the UAV is equipped with the ISAC devices and sends ISAC signals, and an RIS is deployed to improve the link quality for better system performance. Subject to constraints on the maximum transmission power, maximum flight speed, and minimum sensing beampattern gain requirements, we jointly optimize the trajectory of the UAV, the beamforming matrix of the UAV, and the passive beamforming matrix of the RIS to maximize the sum-rate throughput of the system.
• We transform the non-convex problem into a semi-Markov decision process (SMDP); the orignal problem is first decomposed into two subproblems, which are trajectory optimization and sum-rate throughput optimization. We then propose an HDRL framework with a hierarchical twin-delayed deep deterministic policy gradient (HTD3) structure for the joint optimization of the trajectory and beamforming matrix of the UAV as well as the passive beamforming matrix of the RIS.
• We simulate the proposed algorithm and other benchmarks for comparison. Extensive results represent the effectiveness and superiority of the proposed HTD3. Specifically, we verify the convergence of the proposed algorithm via different environment parameters based on numerous experiments, and the experimental results demonstrate that HTD3 achieves a higher sum-rate throughput compared to other benchmarks.

The rest of the paper is organized as follows. Section 2 introduces the system model and formulates the sum-rate throughput optimization problem of the RIS-aided UAV-ISAC system. HDRL with an HTD3 structure is introduced in Section 3. Section 4 shows and analyzes the simulation results to represent the effectiveness and superiority of the proposed HTD3. The paper is concluded in Section 5.

2. System Model

2.1. System Overview

The model of the RIS-aided UAV-ISAC system is shown in Figure 1. One UAV with ISAC devices is deployed for synchronous radar sensing and transmission; to be specific, the UAV senses ground targets and synchronously sends the generated sensing data to UEs. Considering the obstruction of trees, buildings, and other obstacles, the channel between the UAV and UEs is the non-line-of-sight (NLOS). Moreover, the user’s location is far from the sensing area, and hence an RIS, which reflects the signals transmitted from the UAV and adjusts their electromagnetic wave characteristics, is adopted to enhance the performance of communication through improving the channel’s quality and the system’s performance. The sensing area contains J potential targets, which can be represented as a set $\mathcal{J}=\{1,\dots ,J\}$. The ISAC devices in the UAV perform joint radar sensing and communication through sending combined communication and sensing signals. The set of UEs can be represented as $\mathcal{K}=\{1,\dots ,K\}$. The UAV is equipped with an A-element uniform linear array (ULA) and the RIS is equipped with a uniform planar array (UPA) which consists of M passive reflecting elements. The finite ISAC mission duration is divided into N time slots, which can be denoted as $\mathcal{N}=\{1,\dots ,N\}$. In each time slot, the UAV could move horizontally to adjust its position and perform sensing and communication synchronously.

2.2. Transmission Model

Let ${\mathit{q}}_u\left[n\right]=(x_u\left[n\right],y_u\left[n\right],H)$ denote the time-varying position of the UAV, where H is a fixed value representing the flight altitude. Let ${\mathit{q}}_k\left[n\right]=(x_k\left[n\right],y_k\left[n\right],z_k\left[n\right])$, ${\mathit{q}}_R=(x_R,y_R,z_R)$ denote the time-varying position of the $k_{th}$ UE and the fixed position of the RIS, respectively; RIS is fixed on a building [20,25] and each UE randomly moves in a small area. Assuming that the duration of each time slot is ${\delta }_{fly}$, the average flight speed of the UAV in the $n_{th}$ time slot can be expressed as

$$v_u\left[n\right]={\displaystyle \frac{\sqrt{\left|\right|{\mathit{q}}_u\left[n\right]-{\mathit{q}}_u[n-1]{\left|\right|}^2}}{{\delta }_{fly}}}.$$

(1)

The channel from the UAV to the RIS can be regarded as an LoS link due to few obstacles, and hence the channel gain from the UAV to the RIS in time slot n can be expressed as

$${\mathit{h}}_{U,R}\left[n\right]=\sqrt{{\rho }_0{d}_{U,R}^{-2}\left[n\right]}{\mathit{a}}_p({\phi }_{U,R}\left[n\right],{\theta }_{U,R}\left[n\right]){\mathit{a}}_L^H\left({\theta }_{U,R}\left[n\right]\right),$$

(2)

where ${\rho }_0$ denotes the channel power gain at reference distance $d_0=1\mathrm{m}$, and $d_{U,R}\left[n\right]=\left|\right|{\mathit{q}}_u\left[n\right]-{\mathit{q}}_R{\left|\right|}_2$ denotes the distance between the UAV and the RIS in the $n_{th}$ time slot. ${\phi }_{U,R}\left[n\right]$ and ${\theta }_{U,R}\left[n\right]$ denote the azimuth angle and the elevation angle of the RIS considering the position of the UAV. The steering vector of ULA and UPA can be calculated as:

$$\begin{array}{cc}\hfill & {\mathit{a}}_L\left(\theta \right)={[1,e^{-j\pi cos\left(\theta \right)},\dots ,e^{-j\pi (A-1)cos\left(\theta \right)}]}^T,\hfill \end{array}$$

(3)

and

$$\begin{array}{cc}\hfill & {\mathit{a}}_P(\phi ,\theta )={[1,\dots ,e^{j\pi sin\left(\theta \right)\left[\right(m-1)cos(\phi )+(a-1)sin(\phi \left)\right]},\dots ,e^{j\pi sin\left(\theta \right)\left[\right(M-1)cos(\phi )+(A-1)sin(\phi \left)\right]}]}^T,\hfill \end{array}$$

(4)

respectively, where $m\in \{1,\dots ,M\}$, $a\in \{1,\dots ,A\}$.

For the channel from the UAV to UEs and from the RIS to UEs, the NLoS link is considered due to the obstruction of obstacles such as trees and buildings. A kind of probabilistic LoS channel is adopted [29], which can be expressed as

$$\begin{array}{c}\hfill PL\left[n\right]={\displaystyle \frac{({ϵ}_{LoS}-{ϵ}_{NLoS})}{1+a_{pl}{e}^{-b_{pl}(\theta \left[n\right]-a_{pl})}}}+20log{\displaystyle \frac{4\pi f_{c}d\left[n\right]}{v_c}}+{ϵ}_{NLoS},\end{array}$$

(5)

where ${ϵ}_{LoS}$, ${ϵ}_{NLoS}$, and $a_{pl}$, $b_{pl}$ denote the coefficients determined by the environment, and $\theta \left[n\right]$ denotes the elevation angle from the UEs to the UAV or RIS. Then, the channel from the UAV to the $k_{th}$ UE and from the RIS to the $k_{th}$ UE in time slot n can be expressed as

$${\mathit{h}}_{U,k}\left[n\right]=\sqrt{{10}^{{\displaystyle \frac{P{L}_{U,k}\left[n\right]}{20}}}}{\mathit{a}}_L\left({\theta }_{U,k}\left[n\right]\right),\forall k\in \mathcal{K},$$

(6)

and

$${\mathit{h}}_{R,k}\left[n\right]=\sqrt{{10}^{{\displaystyle \frac{P{L}_{R,k}\left[n\right]}{20}}}}{\mathit{a}}_P({\phi }_{U,k}\left[n\right],{\theta }_{U,k}\left[n\right]),\forall k\in \mathcal{K},$$

(7)

respectively, and $P{L}_{U,k}\left[n\right]$ and $P{L}_{R,k}\left[n\right]$ can be expressed as

$$\begin{array}{c}\hfill P{L}_{U,k}\left[n\right]={\displaystyle \frac{({ϵ}_{LoS}-{ϵ}_{NLoS})}{1+a_{pl}{e}^{-b_{pl}({\theta }_{U,k}\left[n\right]-a_{pl})}}}+20log{\displaystyle \frac{4\pi f_c{d}_{U,k}\left[n\right]}{v_c}}+{ϵ}_{NLoS},\end{array}$$

(8)

and

$$\begin{array}{c}\hfill P{L}_{R,k}\left[n\right]={\displaystyle \frac{({ϵ}_{LoS}-{ϵ}_{NLoS})}{1+a_{pl}{e}^{-b_{pl}({\theta }_{R,k}\left[n\right]-a_{pl})}}}+20log{\displaystyle \frac{4\pi f_c{d}_{R,k}\left[n\right]}{v_c}}+{ϵ}_{NLoS},\end{array}$$

(9)

respectively, where $d_{U,k}\left[n\right]$ denotes the time-varying distance between the UAV and the $k_{th}$ UE, $d_{R,k}\left[n\right]$ denotes the time-varying distance between the RIS and the $k_{th}$ UE, ${\theta }_{U,k}\left[n\right]$ denotes the time-varying elevation angle from the $k_{th}$ UE to the UAV, and ${\theta }_{R,k}\left[n\right]$ denotes the time-varying elevation angle from the $k_{th}$ UE to the RIS. In particular, the cascaded channel gain from the UAV to the $k_{th}$ UE in time slot n can be represented as

$${\mathit{H}}_{U,k}\left[n\right]={\mathit{h}}_{U,k}^H\left[n\right]+{\mathit{h}}_{R,k}^H\left[n\right]\Theta \left[n\right]{\mathit{h}}_{U,R}\left[n\right],\forall k\in \mathcal{K},$$

(10)

where $\Theta \left[n\right]$ denotes the passive beamforming matrix of RIS in time slot n, which can be expressed as $\Theta \left[n\right]=diag({\xi }_1\left[n\right]e^{j{\varphi }_1\left[n\right]},\dots ,{\xi }_m\left[n\right]e^{j{\varphi }_m\left[n\right]},\dots ,{\xi }_M\left[n\right]e^{j{\varphi }_M\left[n\right]})$, and ${\xi }_m\left[n\right]$ and ${\varphi }_m\left[n\right]$ are the amplitude coefficient and phase shift of the $m^{th}$ RIS element, with ${\xi }_m\left[n\right]\in [0,1]$ and ${\varphi }_m\left[n\right]\in [0,2\pi ]$. To maximize the performance of RIS, we fix the ${\xi }_m\left[n\right]$ as 1.

Let $\mathit{s}\left[n\right]={[s_1\left[n\right],\dots ,s_K\left[n\right]]}^T\in {\mathbb{C}}^{K\times 1}$ with $E\left[\right|s_k\left[n\right]{|}^2]=1$ denoting the transmitted symbol in time slot n, and $s_0\left[n\right]$ is a zero mean random vector with covariance matrix $E\left[s_0\left[n\right]s_0^H\left[n\right]\right]⪰0$ denoting the dedicated radar sensing signal [2]. $\mathit{W}\left[n\right]=[{\mathit{w}}_1\left[n\right],\dots ,$${\mathit{w}}_K\left[n\right]]\in {\mathbb{C}}^{A\times K}$ denotes the beamforming matrix of the UAV, assuming that the communication signal could be used to facilitate sensing; the UAV transmits communication signals together with sensing signals, and hence the transmitted signal of the UAV in time slot n can be expressed as

$${\mathit{x}}_u\left[n\right]=\mathit{W}\left[n\right]\mathit{s}\left[n\right]+s_0\left[n\right],$$

(11)

and overall, the received signal at the $k_{th}$ UE can be expressed as

$${\mathit{y}}_k\left[n\right]={\mathit{H}}_{U,k}\left[n\right]{\mathit{x}}_u\left[n\right]+z_k\left[n\right],\forall k\in \mathcal{K},$$

(12)

where $z_k\left[n\right]\sim \mathcal{CN}(0,{\sigma }_k^2)$ represents the Gaussian white noise.

For the $k_{th}$ UE, interference comes from the undesired signal, sensing signal, and noise, and thus the received signal-to-interference-plus-noise ratio (SINR) at the $k_{th}$ UE can be expressed as

$$\begin{array}{c}\hfill SIN{R}_k\left[n\right]={\displaystyle \frac{|{\mathit{H}}_{u,k}\left[n\right]{\mathit{w}}_k{\left[n\right]|}^2}{{\sum }_{k^{\prime }}|{\mathit{H}}_{u,k}\left[n\right]{\mathit{w}}_{k^{\prime }}\left[n\right]{|}^2+{\mathit{H}}_{u,k}\left[n\right]E[s_0\left[n\right]s_0^H\left[n\right]{\mathit{H}}_{u,k}^H\left[n\right]+{\sigma }_k^2}},\forall k\in \mathcal{K},k^{\prime }\in \mathcal{K},k^{\prime }\ne k,\end{array}$$

(13)

and then, the achievable rate of the $k_{th}$ UE can be calculated by

$$r_k\left[n\right]=lo{g}_2(1+SIN{R}_k\left[n\right]),\forall k\in \mathcal{K}.$$

(14)

2.3. Sensing Model

The sensing performance can be described as the beam pattern gain of the radar towards the locations of the potential targets [2,10], and we assume that the communication signals are also used for sensing. The transmit beam pattern gain from the UAV to the $j_{th}$ target with its location ${\mathit{q}}_j=(x_j,y_j,z_j)$ can be given by

$$\begin{array}{cc}\hfill & \Gamma ({\mathit{q}}_u\left[n\right],{\mathit{q}}_j)=E\left[\right|{\mathit{a}}_L^H\left({\theta }_{U,j}\left[n\right]\right){\mathit{x}}_u\left[n\right]{|}^2]\hfill \\ \hfill & ={\mathit{a}}_L^H\left({\theta }_{U,j}\left[n\right]\right)(tr\left({\mathit{W}}_u\left[n\right]{\mathit{W}}_u^H\left[n\right]\right)+E\left[s_0\left[n\right]s_0^H\left[n\right]\right){\mathit{a}}_L\left({\theta }_{U,j}\left[n\right]\right),\hfill \end{array}$$

(15)

where ${\theta }_{U,j}\left[n\right]$ denotes the elevation from the $j_{th}$ target to the UAV in time slot n; in this case, ISAC devices extract information from the reflected signals from the target, and its power is related to the distance between the UAV and the target. To ensure effective sensing, the beam pattern gain threshold ${\Gamma }_{th}$ is set to represent the lowest sensing performance. Thus, the following constraint must be satisfied:

$${\displaystyle \frac{\Gamma ({\mathit{q}}_u\left[n\right],{\mathit{q}}_j)}{d_{U,j}^2\left[n\right]}}\ge {\Gamma }_{th},$$

(16)

where $d_{U,j}\left[n\right]$ denotes the distance between the UAV and the $j_{th}$ target in time slot n. This constraint first ensures sufficient power allocation in the target direction to achieve successful sensing; then, considering that a larger beam pattern gain necessarily means better overall sensing performance, in this paper, this constraint is sufficient to represent sensing performance. In general, we maximize the communication performance of the system while satisfying this constraint.

UAVs extract sensing information from echo signals. Hence, we use average sensing SNR, which represents the quality of echo signals, to express the sensing performance throughout the entire transmission process. The average sensing SNR can be calculated by [30]

$$SN{R}_s={\displaystyle \frac{1}{N}}\sum _{n=1}^N\sum _{j=1}^J{\displaystyle \frac{s_e{\rho }_0^2\Gamma ({\mathit{q}}_u\left[n\right],{\mathit{q}}_j)}{16\pi d_{U,j}^4\left[n\right]{\sigma }_k^2}},$$

(17)

where $s_e$ denotes the radar cross-section of the targets.

2.4. Problem Formulation

In the proposed RIS-aided UAV-ISAC system, we aim to optimize the accumulated sum-rate throughput of N time slots. The optimization is formulated as (P1):

$$\begin{array}{}\mathrm{(18a)}& \hfill \left(P1\right)& :\underset{\mathcal{Q},\mathit{W}\left[n\right],\Theta \left[n\right],{\mathit{R}}_s\left[n\right]}{max}{\displaystyle \frac{1}{N}}\sum _{n=1}^N\sum _{k=1}^K{r}_k\left[n\right],\hfill \mathrm{(18b)}& & \hfill s.t.tr\left(\mathit{W}\left[n\right]{\mathit{W}}^H\left[n\right]\right)+tr\left({\mathit{R}}_s\left[n\right]\right)\le P_{max},\forall n\in \mathcal{N},\hfill \mathrm{(18c)}& \hfill & v_u\left[n\right]\le v_{max},\forall n\in \mathcal{N},\hfill \mathrm{(18d)}& \hfill & {\displaystyle \frac{\Gamma ({\mathit{q}}_u\left[n\right],{\mathit{q}}_j)}{d_{U,j}^2\left[n\right]}}\ge {\Gamma }_{th},\forall n\in \mathcal{N},\hfill \mathrm{(18e)}& \hfill & {\varphi }_m\left[n\right]\in [0,2\pi ),m\in \left\{1,\dots ,M\right\},\forall n\in \mathcal{N},\hfill \end{array}$$

where $\mathcal{Q}=\{{\mathit{q}}_u\left[1\right],\dots ,{\mathit{q}}_u\left[N\right]\}$ denotes the trajectory of the UAV, ${\mathit{R}}_s\left[n\right]=E\left[s_0\left[n\right]s_0^H\left[n\right]\right]$ denotes the covariance matrix of the sensing signal, Equation (18b) is the transmission power constraint, Equation (18c) indicates that the maximum UAV speed, Equation (18d) defines sensing threshold to ensure the sensing performance, and Equation (18e) defines a range of the phase shift of the RIS.

3. The Proposed Algorithm

In this section, we propose an HTD3-structure HDRL framework for joint-trajectory beamforming optimization. The original problem is decomposed into two subquestions, which solve the trajectory optimization and beamforming task offloading optimization, respectively. The entire process is constructed as an SMDP, and the mechanism of our HTD3 is discussed.

3.1. SMDP

To reduce the dimension of action space and state space of the original problem, in this paper, the original problem can be decomposed into two parts, which are:

$$\begin{array}{}\mathrm{(19a)}& \hfill \left(\tilde{P}1\right)& :\underset{\mathit{Q}}{max}\tilde{P}2,\hfill \mathrm{(19b)}& \hfill & s.t.v_u\left[n\right]\le v_{max},\forall n\in \mathcal{N},\hfill \end{array}$$

and

$$\begin{array}{}\mathrm{(20a)}& \hfill & \left(\tilde{P}2\right):\underset{\mathit{W}\left[n\right],\Theta \left[n\right],{\mathit{R}}_s\left[n\right]}{max}{\displaystyle \frac{1}{N}}\sum _{n=1}^N\sum _{k=1}^K{r}_k\left[n\right],\hfill \mathrm{(20b)}& \hfill & s.t.tr\left(\mathit{W}\left[n\right]{\mathit{W}}^H\left[n\right]\right)+tr\left({\mathit{R}}_s\left[n\right]\right)\le P_{max},\forall n\in \mathcal{N},\hfill \mathrm{(20c)}& \hfill & {\displaystyle \frac{\Gamma ({\mathit{q}}_u\left[n\right],{\mathit{q}}_j)}{d_{U,j}^2\left[n\right]}}\ge {\Gamma }_{th},\forall n\in \mathcal{N},\hfill \mathrm{(20d)}& \hfill & {\varphi }_m\left[n\right]\in [0,2\pi ),m\in \left\{1,\dots ,M\right\},\forall n\in \mathcal{N}.\hfill \end{array}$$

These subproblems are regarded as a hierarchical structure. In the time slot n, we adjust the UAV’s location and for the next $\Delta$ time slots, we adjust the beamforming matrix, the sensing signal of the UAV, and the passive beamforming matrix of the RIS to perform sensing and communication. As shown in (19a) and (20a), we solve (20a) to achieve a higher sum-rate throughput, while we solve (19a) (i.e., adjust the UAV’s location) to maximize (20a).

The hierarchical structure mentioned earlier can be abstracted as an SMDP, and the UAV can be treated as an agent which executes options and actions. An SMDP consists of several MDPs, and each MDP starts from the agent’s option; an option set $\mathcal{O}$ is characterized by the state set $\mathcal{I}$, the policy function ${\mathit{\pi }}_o$, and the termination probability of an option $\beta$. In this article, the option $o_n\in \mathcal{O}$ denotes the adjustment of the locations of the UAV (i.e., $\Delta {\mathit{q}}_u\left[n\right]$, the next location for the UAV to perform sensing and communication). We use ${\mathcal{M}}_n=<\mathcal{S},\mathcal{A},\mathcal{R}>$ to represent the MDP of the option, which denotes the process completing the sensing and communication tasks in the location chosen by option $o_n$ of the UAV. The details of the option set $\mathcal{O}$ can be described as follows:

(1)

State space $\mathcal{I}$: CSI is the most influential factor, which includes the estimated CSI between the UAV and the RIS, and between the UAV and the UEs. To evaluate the sensing performance, the distance between the UAV and the targets is also considered, and thus the state set of the options can be described as:

$$\mathcal{I}=\{d_{U,j}\left[n\right],{\mathit{h}}_{U,R}\left[n\right],{\mathit{h}}_{U,k}\left[n\right]\},\forall j\in \mathcal{J},\forall k\in \mathcal{K}.$$

(21)

(2)

Policy set ${\mathit{\pi }}_o$: ${\mathit{\pi }}_o$ denotes the policy set under an option $o_n$; since our option is the adjustment of locations of the UAV and we choose proper options to maximize ($\tilde{P}2$), the policy function can be described as the adjustment of the beamforming matrix, the sensing signal of the UAV, and the passive beamforming matrix of the RIS under the option $o_n$, which are actions required to complete an option and can be described as:

$${\mathit{\pi }}_o=\{\mathit{W}\left[n\right],\Theta \left[n\right],{\mathit{R}}_s\left[n\right]\}.$$

(22)

(3)

Termination probability $\beta$: $\beta \left(s_n^H\right)$ denotes the termination probability at state $s_n^H\in \mathcal{I}$. Each option $o_n$ denotes the location adjustment of the UAV in time slot n, and the UAV performs the sensing and communication tasks for the later $\Delta$ time slots; hence, $\beta \left(s_{n+\Delta }^H\right)=1$ means the termination probability of the option $o_n$ in the time slot $n+\Delta$ is 1, where $s_{n+\Delta }^H\in \mathcal{I}$ denotes the state in time slot $n+\Delta$.

Then, for a location chosen by the option $o_n$, the UAV hovers to complete the sensing and communication in the later $\Delta$ time slots, which can be modeled as an MDP. The details of the MDP can be described as:

(1)

State space $\mathcal{S}$: The observed state consists of the location of the UAV, which is determined by the option $o_n$, the estimated CSI between the UAV and the RIS, and between the UAV and the UEs, and the distance between the UAV and the targets; thus, the state space can be expressed as

$$\mathcal{S}=\{{\mathit{q}}_u\left[n\right],d_{U,j}\left[n\right],{\mathit{h}}_{U,R}\left[n\right],{\mathit{h}}_{U,k}\left[n\right]\},\forall j\in \mathcal{J},\forall k\in \mathcal{K}.$$

(23)

(2)

Action space $\mathcal{A}$: The action space contains the actions to perform the sensing and communication during the $\Delta$ time slots, which is consistent with the policy set ${\mathit{\pi }}_o$, (i.e., Equation (22)).

(3)

Reward $\mathcal{R}$: Since our object is to maximize the sum-rate throughput during the whole N process, the sum-rate throughput within time slot n can be directly used as the reward, which can be expressed as

$$\begin{array}{c}\hfill r_{s_n^L}^{a_n}=\left\{\begin{array}{c}{P}_e,constraintisviolated\hfill \\ \sum _{k=1}^K{r}_k\left[n\right],else,\hfill \end{array}\right.\end{array}$$

(24)

where $r_{s_n^L}^{a_n}$ denotes the reward obtained by executing action $a_n\in \mathcal{A}$ at the state $s_n^L\in \mathcal{S}$, and $P_e$ denotes the penalty coefficient, which is set to 0 (i.e., no data transmission).

In summary, the process of the SMDP is shown in Figure 2. In time slot n, the state $s_n^U$ is observed. An option $o_n$ is chosen to adjust the UAV’s location. The later $\Delta$ time slots can be regarded as an MDP. In each time slot, the UAV chooses an action $a_n$ to perform sensing and communication at the fixed location determined by the option $o_n$. At the end of the process, we observe the state $s_{n+\Delta }^U$ at time slot $n+\Delta$, obtain the reward $r_{s_n^U}^{o_n}$ for the option $o_n$, and then start the next option selection.

3.2. HDRL Framework

The aforementioned SMDP can be solved with HDRL; hence, we design a two-layer HDRL to perform continuous action control, where the lower layer network is adopted to maximize the sum-rate throughput and the upper layer network is adopted to maximize the average reward of the lower layer network.

Similar to reward $r_{s_n^L}^{a_n}$, the reward of executing option $o_n$ at state $s_n^H$ can be expressed as

$$\begin{array}{cc}\hfill & r_{s_n^U}^{o_n}=E[r_{s_{n+1}^L}^{a_{n+1}}+\gamma r_{s_{n+2}^L}^{a_{n+2}}+\cdots +{\gamma }^{\Delta -1}{r}_{s_{n+\Delta }^L}^{a_{n+\Delta }}|\epsilon (o_n,s_n^U,n)],\hfill \\ \hfill & s_n^U\in \mathcal{I},s_{n+1}^L,\dots ,s_{n+\Delta }^L\in \mathcal{S},\hfill \end{array}$$

(25)

where $\epsilon (o_n,s_n^U,n)$ denotes that option $o_n$ begins to be executed at time slot n and in state $s_n^U$. Thus, for option set $\mathcal{O}$, the $Bellman$ equation is given by

$$\begin{array}{cc}\hfill & Q_O^{*}(s_n^U,o_n)=E[r_{s_n^U}^{o_n}+{\gamma }^{\Delta }\underset{o^{\prime }\in \mathcal{O}}{max}{Q}_O^{*}(s_{n+\Delta }^U,o^{\prime })|\epsilon (o_n,s_n^U)],\hfill \\ \hfill & s_n^U,s_{n+\Delta }^U\in \mathcal{I},o_n\in \mathcal{O},\hfill \end{array}$$

(26)

where $Q_O^{*}(s_n^U,o_n)$ denotes the optimal option-value function, and $\gamma \in (0,1)$ denotes the discount factor, which represents the importance of the long-term return. And for MDP, the $Bellman$ equation can be given by

$$\begin{array}{c}\hfill Q^{*}(s_n^L,a_n)=E[r_{s_n^L}^{a_n}+{\gamma }^{\Delta }\underset{a^{\prime }\in \mathcal{A}}{max}{Q}^{*}(s_{n+1}^L,a^{\prime })],s_n^L,s_{n+1}^L\in \mathcal{S},a_n,a_{n+1}\in \mathcal{A};\end{array}$$

(27)

if we use $Q_O(s_n^U,o_n)$ and $Q(s_n^L,a_n)$ to represent the estimated optimal action-value function, then they can be updated by

$$\begin{array}{c}\hfill Q_O(s_n^U,o_n)\leftarrow Q_O(s_n^U,o_n)+\alpha [r_{s_n^U}^{o_n}+{\gamma }^{\Delta }\underset{o^{\prime }\in \mathcal{O}}{max}{Q}_O^{*}(s_{n+\Delta }^U,o^{\prime })-Q_O(s_n^U,o_n)],\end{array}$$

(28)

and

$$\begin{array}{c}\hfill Q(s_n^L,a_n)\leftarrow Q(s_n^L,a_n)+\alpha [r_{s_n^L}^{a_n}+{\gamma }^{\Delta }\underset{a^{\prime }\in \mathcal{A}}{max}Q(s_{n+1}^L,a^{\prime })-Q(s_n^L,a_n)],\end{array}$$

(29)

respectively, where $\alpha \in (0,1)$ denotes the learning rate.

The structure of the HDRL is an HTD3 structure, which is shown in Figure 3, where the upper network is adopted to solve (26), the lower network is adopted to solve (27), and both networks adopt a TD3 structure for continuous action control. The critic networks in the upper network can be expressed as $Q(s_n^U,o_n,{\vartheta }_{c,1}^U)$, $Q(s_n^U,o_n,{\vartheta }_{c,2}^U)$, which are adopted to estimate $Q_O^{*}(s_n^U,o_n)$, where ${\vartheta }_{c,1}^U$, ${\vartheta }_{c,2}^U$ are network parameters. The actor network in the upper network can be expressed as $\pi (s_n,{\vartheta }_a^U)$, which is adopted to estimate $argma{x}_{o^{\prime }\in \mathcal{O}}{Q}_O^{*}(s_{n+\Delta }^U,o^{\prime })$. Similarly, the networks $Q(s_n^L,a_n,{\vartheta }_{c,1}^L)$, $Q(s_n^L,a_n,{\vartheta }_{c,2}^L)$ and $\pi (s_n,{\vartheta }_a^L)$ in the lower network are adopted to estimate $Q^{*}(s_n^L,a_n)$ and $argma{x}_{a^{\prime }\in \mathcal{A}}{Q}^{*}(s_{n+1}^L,$$a^{\prime })$, respectively.

The update process of HTD3 is divided into two parts, which are the storing experience and network updating. Let ${\mathcal{B}}_U$ and ${\mathcal{B}}_L$ denote the replay buffers of the upper network and the lower network, respectively. ${\mathcal{B}}_U$ stores transitions $(s_n^U,o_n,r_{s_n^U}^{o_n},s_{n+\Delta }^U)$ while ${\mathcal{B}}_L$ stores transitions $(s_n^L,a_n,r_{s_n^L}^{a_n},s_{n+1}^L)$, where $o_n$ and $a_n$ are, respectively, output by $\pi (s_n,{\vartheta }_a^U)$ and $\pi (s_n,{\vartheta }_a^L)$. Then, the networks extract transitions from the replay buffer for training.

For the lower network, the objective of the critic networks is to estimate the action value as accurately as possible, and hence the TD target is adopted to represent the true value of the action value; with the transition $(s_n^L,a_n,r_{s_n^L}^{a_n},s_{n+1}^L)$, the TD target can be calculated as

$$y_n^{TD,L}=r_{s_n^L}+\gamma \underset{i=1,2}{min}Q(s_{n+1}^L,{\tilde{a}}_n,{\widehat{\vartheta }}_{c,i}^L),$$

(30)

where $Q(s_n^L,a_n,{\widehat{\vartheta }}_{c,i}^L)$ denotes the target network of the $Q(s_n^L,a_n,{\vartheta }_{c,i}^L)$. The minimum value of two target networks is adopted to avoid overestimation. Action ${\tilde{a}}_n$ denotes the action of the agent at state $s_{n+1}^L$, which can be calculated as

$${\tilde{a}}_n=\pi (s_{n+1}^L,{\widehat{\vartheta }}_a^L)+{\mathit{\mu }}_a,$$

(31)

where ${\mathit{\mu }}_a$ denotes the action noise with zero mean and variance ${\sigma }_{action}^2$. Overall, the accuracy of the critic networks can be measured by the mean squared error (MSE) between the target value and the value estimated by the critic network, which can be expressed as

$$\mathbb{L}\left({\vartheta }_{c,i}^L\right)=(y_n^{TD,L}-Q{(s_n^L,a_n,{\vartheta }_{c,i}^L)}^2),i=1,2,$$

(32)

and the critic networks can be updated by the gradient descent which can be expressed as

$${\vartheta }_{c,i}^L\leftarrow {\vartheta }_{c,i}^L-{\alpha }_c{\nabla }_{{\vartheta }_{c,i}^L}\mathbb{L}\left({\vartheta }_{c,i}^L\right),i=1,2,$$

(33)

where ${\alpha }_c$ denotes the learning rate of the critic networks. The objective of the actor network is to make the output value of the critic networks as large as possible, and hence the loss function of the actor network can be expressed as

$$\mathbb{L}\left({\vartheta }_a^L\right)={\displaystyle \frac{1}{2}}\sum _{i}Q(s_n^L,\pi (s_n^L,{\vartheta }_a^L),{\vartheta }_{c,i}^L),i=1,2.$$

(34)

Then, the actor network can be updated by the gradient ascent, which can be expressed as

$${\vartheta }_a^L\leftarrow {\vartheta }_a^L+{\alpha }_a{\nabla }_{{\vartheta }_a^L}\mathbb{L}\left({\vartheta }_a^L\right).$$

(35)

In particular, the target networks of the critic networks and the actor networks can be soft-updated, respectively, by

$$\begin{array}{}\mathrm{(36a)}& \hfill & \widehat{{\vartheta }_{c,i}^L}\leftarrow (1-\tau )\widehat{{\vartheta }_{c,i}^L}+\tau {\vartheta }_{c,i}^L,i=1,2,\hfill \mathrm{(36b)}& \hfill & \widehat{{\vartheta }_a^L}\leftarrow (1-\tau )\widehat{{\vartheta }_a^L}+\tau {\vartheta }_a^L,\hfill \end{array}$$

where $\tau \in (0,1)$ denotes the ratio of the parameters in soft updating.

By the same token, for the upper network with the transition $(s_n^U,o_n,r_{s_n^U}^{o_n},s_{n+\Delta }^U)$, reward $r_{s_n^U}^{o_n}$ can be calculated by

$$r_{s_n^U}^{o_n}={\displaystyle \frac{1}{\Delta }}\sum _{i=n+1}^{n+\Delta }{r}_{s_i^L}^{a_i};$$

(37)

with the TD target

$$y_n^{TD,U}=r_{s_n^U}+\gamma \underset{i=1,2}{min}Q(s_{n+\Delta }^U,{\tilde{o}}_n,{\widehat{\vartheta }}_{c,i}^U),$$

(38)

the critic networks and actor network can be updated by

$${\vartheta }_{c,i}^U\leftarrow {\vartheta }_{c,i}^U-{\alpha }_c{\nabla }_{{\vartheta }_{c,i}^U}\mathbb{L}\left({\vartheta }_{c,i}^U\right),i=1,2,$$

(39)

and

$${\vartheta }_a^U\leftarrow {\vartheta }_a^U+{\alpha }_a{\nabla }_{{\vartheta }_a^U}\mathbb{L}\left({\vartheta }_a^U\right),$$

(40)

respectively, where

$$\begin{array}{}\mathrm{(41a)}& \hfill & \mathbb{L}\left({\vartheta }_{c,i}^U\right)=(y_n^{TD,U}-Q{(s_n^U,o_n,{\vartheta }_{c,i}^U)}^2,i=1,2,\hfill \mathrm{(41b)}& \hfill & \mathbb{L}\left({\vartheta }_a^U\right)={\displaystyle \frac{1}{2}}\sum _{i}Q(s_n^U,\pi (s_n^U,{\vartheta }_a^U),{\vartheta }_{c,i}^U),i=1,2.\hfill \end{array}$$

Finally, their target networks can be updated by

$$\begin{array}{}\mathrm{(42a)}& \hfill & \widehat{{\vartheta }_{c,i}^U}\leftarrow (1-\tau )\widehat{{\vartheta }_{c,i}^U}+\tau {\vartheta }_{c,i}^U,i=1,2,\hfill \mathrm{(42b)}& \hfill & \widehat{{\vartheta }_a^U}\leftarrow (1-\tau )\widehat{{\vartheta }_a^U}+\tau {\vartheta }_a^U.\hfill \end{array}$$

The details of the proposed HTD3 are summarized in Algorithm 1, where the upper TD3 chooses the option $o_n$ to adjust the location of the UAV. Then for the later $\Delta$ time slots, the UAV performs sensing and communication in the fixed location. During these $\Delta$ time slots, the UAV stores the transitions of the lower TD3 and updates the lower TD3 which is adopted for sensing and communication. After the termination of option $o_n$, the UAV stores the transition of the upper network and updates the upper network which is adopted for location adjustment.

Algorithm 1 The HDRL Algorithm

1: Initialize the upper network and lower network parameters: ${\vartheta }_{c,1}^U$,${\vartheta }_{c,2}^U$,${\vartheta }_a^U$,${\vartheta }_{c,1}^L$,${\vartheta }_{c,2}^L$,${\vartheta }_a^L$; 2: Initialize the upper network and lower network’s target networks: ${\widehat{\vartheta }}_{c,1}^U$,${\widehat{\vartheta }}_{c,2}^U$,${\widehat{\vartheta }}_a^U$,${\widehat{\vartheta }}_{c,1}^L$,${\widehat{\vartheta }}_{c,2}^L$,${\widehat{\vartheta }}_a^L$; 3: Initialize the replay buffers ${\mathcal{B}}_U$ and ${\mathcal{B}}_L$; 4: for Episode $n_{ep}\in [1,\dots ,N_{ep}]$ do 5:    Reset the state of the environment, 6:    while Time slot $n<N$ do 7:      For state $s_n^U\in \mathcal{I}$, select an option $o_n\in \mathcal{O}$ with $o_n=\pi (s_n^U,{\vartheta }_a^U)$; 8:      for Time slot $n_l\in [n+1,\dots ,n+\Delta ]$ do 9:          For state $s_{n_l}^L\in \mathcal{S}$ Execute action $a_{n_l}\in \mathcal{A}$ and observe reward $r_{s_{n_l}^L}^{a_{n_l}}$ and next state $s_{n_l+1}^L$; 10:         Store transitions $(s_{n_l}^L,a_{n_l},r_{s_{n_l}^L}^{a_n},s_{n_l+1}^L,done)$ into replay Buffer ${\mathit{B}}_L$; 11:         Sample a random batch of transitions ${\mathit{B}}^{\prime }$ from ${\mathit{B}}_L$; 12:         Compute target actions ${\tilde{a}}_n$ according to Equation (31); 13:         Compute TD targets according to Equation (30); 14:         Update the critic networks and actor networks according to Equations (32)–(35) 15:         Update the target networks according to Equations (36a) and (36b); 16:     end for 17:     Calculate the $r_{s_n^U}^{o_n}={\displaystyle \frac{1}{\Delta }}{\sum }_{i=n+1}^{n+\Delta }{r}_{s_i^L}^{a_i}$; 18:     Store the transition $(s_n^U,o_n,r_{s_n^U}^{o_n},s_{n+\Delta }^U,done)$ into replay buffer ${\mathit{B}}_U$; 19:     Update the upper network according to Equations (39), (40), (41a) and (41b); 20:     Update the target networks of upper network according to Equations (42a) and (42b); 21:     Time slot $n\leftarrow n+\Delta$; 22:   end while 23: end for

Generally speaking, the complexity of the proposed HTD3 structure is composed of the following parts, assuming that the lower layer network and upper layer network have the same structure. Let $u_0^L$, $u_0^U$, J, $u_j$, and $|{\mathit{B}}^{\prime }|$ denote the input layer size of the lower layer network, the input size of the upper layer network, the deep neural network (DNN) layer numbers, the $j_{th}$ layer size, and the mini-batch size, respectively. In each episode, the lower layer network takes N steps for training, and the upper layer network takes $N/\Delta$ steps for training. Hence, the overall computational complexity is $O\left(N_{ep}\right|{\mathit{B}}^{\prime }\left|N((u_0^L{u}_1+{\sum }_{j=1}^J{u}_j{u}_{j+1})+(u_0^U{u}_1+{\sum }_{j=1}^J{u}_j{u}_{j+1}))\right)$.

4. Simulations

In this section, our proposed HDRL is compared with other benchmarks, and numerous simulation results are presented to demonstrate the superiority of our proposed algorithm.

4.1. Simulation Settings

We consider an area of 200 m × 200 m, with two UEs and four targets under default conditions, where the initial position of the UAV is (150 m, 100 m, 40 m), the position of the RIS is fixed as (100 m, 200 m, 15 m), the positions of the UEs are (0 m, 50 m, 0 m) and (50 m, 0 m, 0 m), respectively, and the positions of the targets are (120 m, 80 m, 0 m), (180 m, 80 m, 0 m), (120 m, 60 m, 0 m), and (180 m, 60 m, 0 m), respectively. All neural networks adopt the hidden layer structure of 256 × 128 × 128, and all learning rates are fixed as 0.001. The simulation runs with Python 3.7 and pytorch 2.0.1. The remaining settings of the parameters in the simulations are shown in

Table 1. Parameter table.

Notation	Definition	Value
H	The fixed altitude of the UAV	40 m
$z_R$	The fixed altitude of the RIS	15 m
A	The number of antennas of the UAV	4
M	The number of antennas of the RIS	4
${\delta }_{fly}$	The flight time in each time slot	0.1 s
${\rho }_0$	The channel power gain at reference distance $d_0=1$ m	−30 dB
${ϵ}_{LoS},{ϵ}_{NLoS}$	The NLOS coefficients	[2.3, 34] [29]
$a_{pl},b_{pl}$	The NLOS coefficients	[27.23, 0.097] [29]
$f_c$	The carrier frequency	2 GHz
$v_c$	The speed of light	$3\times {10}^8$ m/s
$s_e$	The radar cross-section of targets	1 m2
${\sigma }_k^2$	The noise power	−114 dBm
$P_{max}$	The maximum transmission power of the UAV	1 W
$v_{max}$	The maximum speed of the UAV	20 m/s
$N_{ep}$	The training rounds	1000
N	The number of time slots of each round	100
$\Delta$	The steps of the lower layer network	10

.

4.2. The Impact of Environmental Parameters on HTD3

Figure 4 presents the convergence performance of the proposed HTD3 via different environment parameters. For the environment with two UEs, we set the thresholds ${\Gamma }_{th}$ to 0, −17 dBm, and −13 dBm, respectively. As shown in Figure 4a, it can be noticed that with a higher sensing performance threshold, the curves converge more slowly; this is because for a higher sensing performance threshold, the agents need more rounds to learn how to ensure successful sensing while maximizing communication performance. For ${\Gamma }_{th}=0$, the curve converges to the highest accumulated sum-rate throughput; the obvious reason is that the UAV does not need to allocate the power resources to sensing in this case. Correspondingly, for ${\Gamma }_{th}=-13$ dBm, the curve converges to the lowest value since the high demand for sensing performance leads to a decrease in communication performance, as more power resources are allocated to sensing to ensure the successful sensing of the targets. Then, we set different number of UEs with a sensing performance threshold ${\Gamma }_{th}=-17$ dBm. As shown in Figure 4b, at the beginning, all curves keep fluctuating, and then maintain convergence after a sharp rise in a certain round. It can be noticed that, as the number of UEs increases, the curve needs more rounds to converge; this is because the complexity of the environment increases. It can also be observed that the highest sum-rate throughput is achieved when the number of UEs is six, while the achieved sum-rate throughput decreases when the number of UEs is eight. This is because limited power resources are insufficient to support communication for more UEs; the system cannot improve communication performance with multiple UEs while ensuring successful sensing in this case.

As shown in Figure 5, we observe the accumulated sum-rate throughput via different maximum transmission power ($P_{max}$) and transmission scenarios with or without RIS. First, as the maximum transmission power increases, the accumulated sum-rate throughput increases; this is because higher power leads to a higher transmission rate within the given power range. Next, the higher the sensing performance threshold ${\Gamma }_{th}$, the lower the accumulated sum-rate throughput; the reason is that a higher sensing demand results in more power resources being allocated to sensing, which leads to a reduction in resources used for communication. Finally, the achieved accumulated sum-rate throughput of scenarios with the RIS is significantly higher than the scenarios without the RIS, which indicates that the deployment of the RIS effectively enhances the communication performance of the system.

4.3. Benchmark Schemes

To verify the effectiveness and superiority of our proposed algorithm, we compare the following benchmark algorithms with the algorithm we proposed:

(1)

TD3: We use TD3 to jointly control the trajectory of the UAV, the beamforming matrix of the UAV, and the passive beamforming matrix of the RIS. Differently from HTD3, all actions are output by one actor network. We use this benchmark to verify the effectiveness of decomposing the original problems.

(2)

TTD3: TTD3 was proposed in [25] and uses two agents to output two different sets of actions. In this article, in the TTD3 case, two agents are adopted to adjust the location of the UAV and handle the sensing and communication tasks. Differently from HTD3, which is a hierarchical structure, although both TTD3 and HTD3 contain two agents, TTD3 adopts a parallel structure, which outputs all actions simultaneously. We use this benchmark to verify the effectiveness of our hierarchical structure design.

(3)

TD3 with Hovering Only (TD3-HO): We use TD3 to jointly control the beamforming matrix of the UAV and the passive beamforming matrix of the RIS, while the UAV hovers at a fixed position. We use this benchmark to verify the effectiveness of the optimization of the trajectory.

(4)

Random action: The UAV randomly executes actions. We use this benchmark to verify the effectiveness of DRL.

Figure 6 shows the accumulated sum-rate throughput of the different algorithms when there are two UEs in the scenario, ${\Gamma }_{th}=-17$ dBm and $P_{max}=1$ W. Since all these algorithms adopt the TD3 structure, all learning rates are set to 0.001. The solid lines in the figure represent the mean values obtained by each algorithm under multiple experiments, while the shaded areas represent the fluctuation range of the curve. It can be observed that our proposed HDRL with an HTD3 structure achieves the highest accumulated sum-rate throughput and the smallest fluctuation. The convergence values obtained by the proposed algorithm are 33%, 50%, and 10% higher than TD3, TTD3, and TD3HO, respectively. All DRL-based schemes achieve a significantly higher accumulated sum-rate throughput to random action schemes due to the learning of agents. All curves with DRL schemes converge between 200 and 250 episodes, and stabilize at the convergence value with some fluctuations. The performance of HTD3 is superior to that of TD3 and TD3HO; the apparent reason is that each layer of HTD3 has lower action space and state space, which significantly improves the performance of learning. Conversely, it can be noticed that the curve of TD3 fluctuates greatly, and TD3HO achieves smaller fluctuations and higher convergence values. Although the TD3 scheme considers trajectory optimization, the increase in the dimensions of action space and state space leads to a decrease in algorithm performance. Furthermore, through comparing HTD3 with TTD3, we conclude that a hierarchical structure is superior to a parallel structure since HTD3 uses an upper network agent to optimize the lower network agent whereas TTD3 uses two agents to simultaneously maximize rewards. Even though they both split the action space and state space and adopt multiple agents, the hierarchical structure takes into account the relevance between agents while the parallel structure ignores it. Above all, the proposed HDRL with an HTD3 stucture has the best learning performance in the RIS-aided UAV-ISAC network.

Figure 7 shows the obtained trajectory with different algorithms. For HTD3 with sensing only (i.e., the UAV only performs sensing tasks to maximize the sensing beampattern gain [2]), it can be noticed that, in this case, the UAV flies directly towards the sensing area when adopting HTD3. This is because the UAV does not have to focus on communication and it will maximize sensing beampattern gain by approaching the sensing targets. We can observe that TD3 adopts a significant range of movement, while the moving distance of HTD3 is obviously shorter than that of TD3; this is because with the decomposing of the problem, the UAV provides clearer control over movement distance due to the lower action space. However, TD3 optimizes the trajectory with other actions, which leads to a greater probability of falling into local optima. Moreover, since the UAV first chooses a location and then performs sensing and communication with a fixed location for $\Delta$ time slots in the HTD3 case, the agent tends to reduce the movement distance to promote network convergence. When adopting TTD3, the movement of the UAV is extremely limited; this is because in the parallel structure, agents in TTD3 cannot observe global information, and hence the agent is unable to optimize the trajectory.

Figure 8 represents the communication and sensing performance via different sensing thresholds ${\Gamma }_{th}$. As shown in Figure 8a, a higher sensing performance threshold ${\Gamma }_{th}$ leads to a lower sum-rate throughput for all algorithms; the obvious reason is that more power resources will be used for sensing when the sensing performance threshold is high, and, correspondingly, the burden of communication will increase. It can be observed that our proposed HTD3 always achieves the best performance with different ${\gamma }_{th}$. The achieved accumulated sum-rate throughput of random action is significantly low; this is because random beamforming matrix and trajectory cannot guarantee effective sensing and communication. TD3HO always outperforms TD3 and TTD3; the reason is that TD3HO has smaller and more stable state spaces due to the lack of trajectory optimization, and, moreover, TD3 and TTD3 are unable to effectively integrate trajectory optimization with the optimization beamforming matrix of the UAV and the passive beamforming matrix of the RIS. As shown in Figure 8b, when using random action, the obtained average sensing SNR remains almost unchanged; this is because the untrained UAV cannot allocate resources reasonably. It can be seen that after the training of DRL, the higher the sensing threshold ${\gamma }_{th}$, the higher the achieved average sensing SNR; the apparent reason is that more power resources need to be allocated to sensing to ensure successful transmission. Notably, our proposed algorithm does not show significant improvement in sensing performance compared to other DRL approaches; this is because our goal is to optimize communication performance while meeting sensing requirements. In other words, all DRL approaches only learn how to allocate power resources to meet sensing requirements, and sensing performance itself is not an optimization goal.

Figure 9 illustrates the accumulated sum-rate throughput performance of various algorithms in scenarios with different parameters. As shown in Figure 9a, an increase in the number of UEs within a certain range (i.e., from two UEs to six UEs) leads to an increase in sum-rate throughput. When there are eight UEs in the scenario, the performance of all algorithms significantly decreases since a large number of UEs causes insufficient communication and sensing resources. The HTD3 we proposed still achieves the best performance under this trend. Figure 9b demonstrates the accumulated sum-rate throughput with different maximum flight speeds $v_{max}$; since TD3HO has no trajectory optimization, we do not consider TD3HO with different $v_{max}$. When $v_{max}=10$ m/s, $v_{max}=20$ m/s, and $v_{max}=30$ m/s, the proposed HTD3 significantly outperforms other benchmarks, which illustrates that our proposed algorithm could better optimize the trajectory. In particular, it can be noticed that TD3 outperforms TTD3 when $v_{max}=20$ m/s and TTD3 outperforms TD3 when $v_{max}=30$ m/s. The performance of all algorithms is almost identical when $v_{max}=40$ m/s; this is because the excessive maneuverability of the UAV leads to the inability of intelligent agents to adapt to changes of state.

5. Conclusions

This paper considers a RIS-aided UAV-ISAC scenario, where the UAV communicates with multiple UEs and performs sensing towards targets simultaneously. Subject to the constraints on the maximum transmission power, maximum flight speed, and minimum sensing beampattern gain requirements, we jointly optimize the trajectory of the UAV, the beamforming matrix of the UAV, and the passive beamforming matrix of the RIS to maximize the sum-rate throughput of the system. We decompose the original problem into a trajectory optimization problem and a sum-rate throughput maximizing problem, and a novel HDRL framework is proposed to alternately solve them. Differently from the existing DRL approaches, our proposed method obtains higher performance by solving two MDPs with smaller state space and action space. The simulation results with different parameters show the effectiveness of the proposed scheme in maximizing the sum-rate throughput, which is significantly better than TD3, TTD3, and TD3HO. Future works will consider the multi-UAV scenario, and we will consider how to extend HDRL to multi-agent reinforcement learning or federated learning in multi-UAV scenarios. Moreover, more precise measurement standards will be adopted to measure the performance of the system.