Adaptive Resource Allocation for Emergency Communications with Unmanned Aerial Vehicle-Assisted Free Space Optical/Radio Frequency Relay System

Abstract

This paper investigates the problem of coordinated resource allocation for multiple unmanned aerial vehicles (UAVs) to address the scarcity of communication resources in disaster-affected areas. UAVs carrying modules of free space optical (FSO) and radio frequency (RF) serve as relay nodes and edge offloading nodes, presenting an FSO/RF dual-hop framework. Considering the varying urgency levels of tasks, we assign task priorities and transform the proposed problem into distributed collaborative optimization problem. Based on the K-means algorithm and the multi-agent deep deterministic policy gradient (MADDPG) algorithm, we propose a UAV-coordinated K-means MADDPG (KMADDPG) to maximize the number of completed tasks while prioritizing high-priority tasks. Simulation results show that KMADDPG is 5% to 10% better than the benchmark DRL methods in convergence performance.

1. Introduction

Natural disasters often result in severe damage to ground communication infrastructure, such as base stations, leading to significant communication challenges and hindering emergency response efforts. These events create a critical need for efficient and rapid deployment of temporary communication networks to facilitate rescue operations and mitigate losses [1]. Infrastructures in affected areas may be destroyed, causing traditional communication networks to malfunction. A primary issue caused by natural disasters is resource scarcity, which also applies to communication networks. The efficient utilization of network resources is top priority. To prevent the losses from network collapse, researchers are focusing on rapidly establishing temporary networks [2,3,4].

Due to their flexibility, low cost, and independence of ground conditions, UAVs are widely used in smart city applications such as urban sensing and delivery [5,6]. UAV-assisted communication networks do not rely on ground infrastructure and can be quickly established as temporary networks using UAVs and FSO communication. FSO communication offers high bandwidth, high speed, and the advantage of no spectrum licensing by transmitting data through the air using visible or infrared light [7]. Its immunity to electromagnetic interference and highly focused beam transmission ensures communication security [8]. The rapid deployment capability as well as low installation and maintenance costs of FSO systems make them perform excellently in emergency communication recovery and complex geographical environments, particularly for urban network interconnection, temporary communication facilities, military communication, and post-disaster emergency communication. Therefore, under bandwidth constraints, adopting FSO technology significantly improves the resource utilization of communication systems [9]. It gradually becomes indispensable in network construction.

However, under adverse weather conditions, the FSO backhaul will be severely affected, reducing the reliability of FSO links [10]. In contrast, RF technology can be used in various environments, unaffected by weather, and is suitable for both indoor and outdoor communication scenarios [11]. To handle rare foggy conditions, it is imperative to quickly build emergency temporary networks instead of constructing expensive permanent parallel RF links [12]. Thus, UAVs can be employed to serve some users, reducing the load on GBSs. We therefore considered a dual-hop FSO/RF model, constructing two channels: from mobile users (MUs) to UAVs and from UAVs to GBSs.

When UAV-assisted communication networks operate in disaster areas, providing equal service to all user devices is unreasonable [13]. When users make communication requests and emergency rescue requests, their urgency levels are different. A more reasonable strategy is for UAVs to prioritize higher-urgency tasks, so we need to incorporate priority levels to distinguish different tasks [14].

To further address the limited terminal computing power and meet MU’s Quality-of-Service (QoS) requirements, we consider using Mobile Edge Computing (MEC) technology in emergency communication. In this mode, MUs can offload computationally intensive tasks to nearby GBSs, reducing processing delays and saving the energy consumption of user devices [15,16]. Specifically, to prevent the base station from being overloaded with tasks, in proposed application scenarios, MUs can offload tasks to UAVs for computation, leveraging UAVs’ mobility to mitigate GBS load [17].

Considering the implementation of UAVs autonomously performing edge offloading and resource allocation, we introduce deep reinforcement learning (DRL) technology. Deep multi-agent reinforcement learning has become a key role in the field of multi-UAV-assisted communications [18,19]. With the increase in the number of UAV and task complexity, traditional control and optimization methods struggle to cope with complex environments and dynamic demands. DRL enables multiple UAVs to act as agents for collaborative learning and decision-making, achieving adaptive and efficient communication network management in dynamic and uncertain environments [20,21].

This study explores the challenges of edge offloading and resource allocation within UAV-supported emergency communication systems. To enhance convergence speed, we incorporate the K-means algorithm into multi-agent deep reinforcement learning. This approach allows each agent to share locally observed user data during the training phase of action neural networks, jointly processing it and moving to a better positiaon. The key contributions of this research are outlined as follows:

(1)

A mathematical model for emergency communication scenarios is formulated, where several UAVs were used as offloading nodes for MEC or as access relays, facilitating the connection between MUs and GBSs. Specifically, we described the scenario, RF model, FSO model, and delay model of emergency communication.

(2)

We propose a UAV-assisted resource allocation method known as K-means MADDPS (KMADDPG), which aims to maximize the number of successful tasks while prioritizing high-priority ones. The proposed algorithm builds on the MADDPG algorithm, integrating it with the K-means algorithm to handle the high data dimensionality in the above-mentioned scenarios. For different tasks with different urgency levels, we incorporate a priority mechanism for mobile users.

(3)

We examine the time complexity of the proposed algorithm and assess its performance in emergency communication settings through simulation studies. The results indicate that the proposed KMADDPG effectively optimizes communication resource allocation in regions affected by disasters with compromised communication infrastructures. Additionally, extensive simulations reveal that KMADDPG surpasses several baseline methods regarding convergence speed and the number of successful tasks.

The structure of the rest of this paper is as follows: Section 2 covers a review of related work. Section 3 describes the system model. In Section 4, we introduce the KMADDPG for resource allocation and edge offloading. Section 5 provides and discusses the simulation results. Finally, conclusions are drawn in Section 6.

2. Related Work

The FSO/RF dual-hop model is a feasible approach to establishing a stable, cost-effective, and rapid heterogeneous network. Known for its extensive transmission range and high bandwidth, the FSO/RF dual-hop model has garnered significant research attention in recent years. For instance, Pang et al. [22] initially proposed an optical IRS-assisted dual-hop hybrid FSO and RF system for cloud radio access networks (C-RANs). They introduced polarization codes in the FSO link to mitigate signal fading and achieve optimal data rates for the RF link. Wang et al. [23] employed gamma–gamma distribution to characterize the atmospheric turbulence effects on the FSO link from UAV to HAP and proposed a novel RIS-assisted UAV secure multi-user FSO/RF system. Lee et al. [24] explored the forwarding of packets between ground terminals and backhaul through multi-HAPS relays in a dual-hop FSO/RF network.

The allocation of scarce communication resources is a critical issue that merits discussion. Che et al. [25] enhanced system-level energy efficiency by jointly designing FSO and RF links and optimizing UAV altitude using power allocation techniques. Qi et al. [26] proposed a UAV-assisted vehicular communication network, utilizing DRL for resource allocation to enhance UAV energy efficiency and ensure QoS. MEC significantly enhances the computational performance of MUs with constrained resources. In the study by Jiang et al.  [27], a new distributed DRL framework utilizing multi-agent systems is introduced to reduce latency and energy usage in solving optimization challenges within large-scale MEC systems.

Most studies have explored mechanisms for task prioritization. For instance, Qin et al. [28] addressed the issue of time-varying priorities for reconnaissance tasks, examining task selection and scheduling within UAV-enabled multi-access edge computing for reconnaissance. Liu et al.  [29] focused on prioritizing industrial equipment and developed a dynamic priority multi-channel access algorithm using DRL. However, they have yet to apply the integration of prioritization and DRL within the domain of UAV.

Researchers have introduced multi-agent reinforcement learning and integrated it with UAV-assisted networks to enable UAV autonomous decision-making [30]. For instance, a UAV-assisted communication network with edge offloading capabilities was developed, and the MADDPG algorithm was proposed to minimize service delay in internet of vehicle (IoV) task processing [31]. In contrast to previous optimization goals, a similar model was constructed, and the MADDPG algorithm was utilized to optimize UAV trajectory design and offloading strategy in a 3D environment [32]. Lee et al. [33] investigated the use of deep reinforcement learning to maximize communication efficiency in the context of hybrid FSO/RF communication models. Guan et al. [34] studied the cooperative trajectory optimization of multiple UAVs in the hybrid FSO/RF communication model to maximize the achievable rate for mobile users.

The above research has made significant contributions to the field of UAV-assisted communication networks. Studies such as  [22,23,24] primarily focus on constructing FSO/RF models. We have incorporated reinforcement learning to train UAVs for autonomous decision-making within the FSO/RF system model. In contrast to  [25,26], our focus is on resource allocation for bandwidth and CPU frequency. Our objective, distinct from the aforementioned studies  [31,32,33,34], is to maximize both the number of successful tasks and the number of high-priority tasks. In this article, we apply a multi-agent reinforcement learning scheme to achieve resource allocation, offloading control, and task prioritization in MEC, enabling UAVs to learn cooperative behaviors.

3. System Model

3.1. Scenario

This paper proposes an application scenario of a multi-UAV-assisted heterogeneous wireless network. As shown in Figure 1, we define that there are J MUs scattered on the plane, following a Poisson distribution, denoted by a set $\mathcal{J}=\{1,\cdots ,J\}$. A fleet of UAVs hover above the MUs, serving as aerial base stations for relay or as nodes for edge computing, receiving data transmitted by MUs. The height of the UAVs is denoted by H and the UAVs are represented by a set $\mathcal{I}=\{1,\cdots ,I\}$. GBSs are located at the edges of the plane, represented by a set $\mathcal{G}=\{1,\cdots ,G\}$. The main notations are listed in

Table 1. List of main notations.

Notation	Definition
$i/\mathcal{I}$	The index/set of UAVs
$j/\mathcal{J}$	The index/set of MUs
$g/\mathcal{G}$	The index/set of GBSs
$d_{ij}$	Distance between UAV i and MU j
$Q_{u_i}/Q_{m_j}$	The position of UAV i/MU j
${\mathsf{\Gamma }}_{ij}^{\mathrm{avg}}$	Average path loss between UAV i and MU j
${\theta }_{ij}^{\mathrm{RF}}$	Transmission rate between UAV i and MU j
${\theta }_i^{\mathrm{FSO}}$	Backhaul rate between UAV i and its GBS
$d_{ig}$	Distance between UAV i and GBS g
$R_{ij}$	Load rate between UAV i and MU j
$B_{ij}$	Bandwidth allocated to MU j by UAV i
$f_{ij}$	CPU frequency allocated to MU j by UAV i
$T_{ij}^{RF}/T_{ig}^{FSO}$	The RF/FSO transmit time between UAV i and GBS g
$\sigma$	Additive white Gaussian noise
$p{r}_i$	The weight of priority
S	State space
A	Action space
$r\left(t\right)$	Reward
$J\left({\theta }_i\right)$	Obeject function
$Q^{\mu }(s,a)$	Value function

. This model can be considered as a two-layer model, since we considered the transmission channel between the UAV and the MU is different from the transmission channel to the GBSs. Communication between UAVs and MUs uses traditional RF communication, while communication between UAVs and GBSs uses FSO communication. The positions of MUs and UAVs are represented by $Q_j^{MU}={\left[x_j,y_j,0\right]}^T$ and $Q_i^{UAV}={\left[x_j,y_j,H\right]}^T$, respectively.

3.2. Air-to-Ground Channel

In the proposed channel model, due to the high complexity of urban environments and dynamic environmental changes, UAVs cannot obtain sufficient information. Therefore, directly applying the ideal free space model is impractical. We introduce a probabilistic model for light-of-sight (LoS) and non-light-of-sight (NLoS), which depends on environmental parameters, elevation angle, and the positions of MUs and UAVs.

The 3D distance between the MU and UAV can be expressed as

$$d_{ij}=\sqrt{{(x_i-x_j)}^2+{(y_i-y_j)}^2+H^2}.$$

(1)

Similarly, according to  [35], the probability of the channel being LoS can be considered as

$$p_{ij}^{LoS}\left({\phi }_{ij}\right)={\displaystyle \frac{1}{1+Cexp(-D({\phi }_{ij}-C))}},$$

(2)

where C and D are coefficients depending on environmental factors such as terrain, atmospheric conditions, lighting conditions, etc. The elevation angle ${\phi }_{ij}$ between UAV i and MU j is given by

$${\phi }_{ij}={\displaystyle \frac{{180}^{\circ }}{2\pi }}arcsin(H/d_{ij}).$$

(3)

According to  [36], the LoS and NLoS path losses can be defined as

$$\begin{array}{cc}\hfill & {\mathsf{\Gamma }}_{ij}^{LoS}={\eta }^{LoS}+{\gamma }^{LoS}log{d}_{ij}+G,\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & {\mathsf{\Gamma }}_{ij}^{NLoS}={\eta }^{NLoS}+{\gamma }^{NLoS}log{d}_{ij}+G,\hfill \end{array}$$

(5)

where ${\eta }^{LoS}$ and ${\eta }^{NLoS}$ present the path losses when the reference distance $d_{ij}$ is equal to 1 m. ${\gamma }^{LoS}$ and ${\gamma }^{NLoS}$ indicate the path loss exponents of the LoS and NLoS transmissions, respectively; G represents a Gaussian random variable with standard deviation. The average value of the path loss can be determined by

$${\mathsf{\Gamma }}_{ij}^{avg}=p_{ij}^{LoS}{\mathsf{\Gamma }}_{ij}^{LoS}+(1-p_{ij}^{LoS}){\mathsf{\Gamma }}_{ij}^{NLoS}.$$

(6)

Between the UAV and the MU, the Signal-to-Interference-plus-Noise Ratio (SINR) of the RF channel can be expressed as

$$\begin{array}{cc}\hfill SINR& ={\displaystyle \frac{P_{\mathrm{signal}}}{P_{\mathrm{noise}}+P_{\mathrm{intf}}}},\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{p_i^{RF}{10}^{\frac{-{\mathsf{\Gamma }}_{ij}^{avg}}{10}}}{{\sigma }^2+\left({\displaystyle \sum _{a=j+1}^{{\mathsf{\Phi }}_i}}{p}_a^{RF}{10}^{\frac{-{\mathsf{\Gamma }}_{ia}^{avg}}{10}}\right)}},\hfill \end{array}$$

(8)

where $P_{\mathrm{signal}}$ is the transmission power; $P_{\mathrm{intf}}$ and $P_{\mathrm{noise}}$ are the co-channel interference and noise, respectively. The latter is modeled as additive white Gaussian noise (AWGN) $\sigma$. To improve the communication quality, we employ Non-Orthogonal Multiple Access (NOMA) technology to reduce co-channel interference among multiple devices served by the same UAV. The set of tasks received by UAV ${\mathsf{\Phi }}_i$ is defined and sorted in descending order of signal strength. When a UAV processes offloading tasks, it prioritizes tasks with stronger signals, reducing interference for subsequent tasks.

The achievable rate can be defined by the SINR as

$${\theta }^{RF}=B_{ij}\ast {log}_2\left(1+{\displaystyle \frac{p_i^{RF}{10}^{\frac{-{\mathsf{\Gamma }}_{ij}^{avg}}{10}}}{{\sigma }^2+{\sum }_{a=j+1}^{{\mathsf{\Phi }}_i}{p}_a^{RF}{10}^{\frac{-{\mathsf{\Gamma }}_{ia}^{avg}}{10}}}}\right),$$

(9)

where $B_{ij}$ is the bandwidth allocated by UAV i to MU j.

3.3. Air-to-Air Channel

We consider that the computing ability of the GBS, which is the endpoint of the FSO channel for task offloading, is sufficiently abundant. Therefore, the computing time for edge offloading tasks at the GBS can be neglected. We only need to focus on optimizing the transmission rate of the FSO channel between the UAV and the GBS. Due to the typical high-altitude operation range of UAVs, atmospheric conditions affecting the FSO channel must be considered. According to  [37], we introduce a size distribution of scattering particles based on weather conditions, defined as

$$\xi =\left\{\begin{array}{cc}1.6,\hfill & \nu >50\mathrm{km},\hfill \\ 1.3,\hfill & 6\mathrm{km}<\nu <50\mathrm{km},\hfill \\ 0.16\nu +0.34,\hfill & 1\mathrm{km}<\nu <6\mathrm{km},\hfill \\ \nu -0.5,\hfill & 0.5\mathrm{km}<\nu <1\mathrm{km},\hfill \\ 0,\hfill & \nu <0.5\mathrm{km},\hfill \end{array}\right.$$

(10)

where $\nu$ is the atmospheric visibility per kilometer. The atmospheric attenuation coefficient can be represented as  [38]

$$\rho ={\displaystyle \frac{3.91}{\nu }}{\left({\displaystyle \frac{\lambda }{550\times {10}^{-9}}}\right)}^{-\zeta },$$

(11)

where $\lambda$ is the transmission wavelength. Utilizing the atmospheric attenuation coefficient $\rho$ and the distance between the UAV and the GBS $d_{ig}$, the atmospheric transmittance at the laser transmitter wavelength ${\varkappa }_{atm}$ can be expressed as [34]

$${\varkappa }_{atm}={10}^{{\displaystyle \frac{-\rho d_{ig}}{10}}}.$$

(12)

The achievable rate of the FSO channel between the UAV and the GBS can be represented as  [34]

$${\theta }_i^{FSO}={\displaystyle \frac{p^{FSO}{\varkappa }_t{\varkappa }_{atm}{\zeta }^2}{\pi {({\upsilon }_t/2)}^2{d}_{ig}{E}_p{N}_b}},$$

(13)

where $p^{FSO}$ is the transmission power of the UAV on the FSO channel; ${\varkappa }_t$ is the optical efficiency of the receiver and transmitter. $\zeta$ represents the free-space beam diameter at the UAV receiver. ${\upsilon }_t$ denotes the divergence angle of the transmitter. $E_p=hc/{\lambda }_c$ represents the photon energy, where h denotes the constant of Planck, c is the speed of light, and ${\lambda }_{\mathrm{c}}$ is the wavelength. $N_{\mathrm{b}}$ represents the average receiver sensitivity. Summarizing these considerations, we establish the high-altitude FSO channel model.

3.4. Computing Delay Model

In the proposed system model, computational tasks of user devices can be processed in three ways: local computing, partial offloading to UAVs via RF channels, and further offloading from UAVs to the GBSs via FSO channels. These methods are detailed as follows:

(1)

Local Computing: When the computing power of the device of an MU can satisfy the task requirements, the MU can choose local processing. The processing time of a task can be expressed as $T_{ij}^{total}=T_{ij}^{local}$, where $T_{ij}^{local}$ can be expressed as

$$T_j^{local}\left[l\right]={\displaystyle \frac{(1-R_{ij}\left[l\right])D_j\left[l\right]s}{f_{m_i}\left[l\right]}},$$

(14)

where l indicates the time slot, $R_{ij}\left[l\right]$ represents the pre-assigned offload ratio of UAVs for the task of MU j. $D_j$ is the size of the task submitted by user j. $f_{m_j}$ is the CPU frequency of the user’s device. s represents the CPU cycles required per bit.

(2)

Offloading to UAVs: When the computing power of the device of an MU cannot meet the task requirements, the MU can choose to offload part of the task to the UAVs for concurrent execution. $T_{ij}^{total}$ can be represented as

$$T_{ij}^{total}=max\left(T_{ij}^{local},T_{ij}^{RF}+T_{ij}^{comp}\right),$$

(15)

where $T_{ij}^{RF}$ and $T_{ij}^{comp}$ represent the time of transmission and computing when the user transmits data to UAV i by the RF channel, respectively. $T_{ij}^{RF}$ and $T_{ij}^{comp}$ can be expressed as

$$T_{ij}^{RF}\left[l\right]={\displaystyle \frac{R_{ij}\left[l\right]D_j\left[l\right]}{{\theta }^{RF}}},$$

(16)

   and

$$T_{ij}^{comp}\left[l\right]={\displaystyle \frac{R_{ij}\left[l\right]D_j\left[l\right]s}{f_{u_i}\left[l\right]}}.$$

(17)

(3)

Offloading to BS via UAV: When the computing power of both the user’s device and the UAV cannot meet the task requirements, the UAV will forward the received task data to the GBS via the FSO channel. $T_{ij}^{total}$ can be represented as

$$T_j^{total}=max(T_j^{local},T_{ij}^{RF}+T_{ig}^{FSO}),$$

(18)

where $T_{ig}^{FSO}$ is the transmission time from UAV i to the GBS g via the FSO channel, $T_{ig}^{FSO}$ can be expressed as

$$T_{ig}^{FSO}\left[l\right]={\displaystyle \frac{R_{ij}\left[l\right]D_j\left[l\right]}{{\theta }^{FSO}}}.$$

(19)

3.5. Problem Formulation

In scenarios with limited system resources and environmental obstructions, our objective is to maximize the completion rate of tasks, with a particular emphasis on prioritizing high importance tasks. We introduce a binary parameter ${\lambda }_{ij}$ to represent the matching relationship between user j and UAV i. If ${\lambda }_{ij}=1$, this indicates that user J is served by UAV i, and vice versa. We can formalize the system optimization problem as

$$\begin{array}{cc}\hfill & P1:\underset{x_i\left[l\right],y_i\left[l\right],z_i\left[l\right]}{max}\sum _j^J\mathbb{I}(T_{ij}^{total}\le t_j^{delay})·p{r}_j,\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\left\{\begin{array}{c}C1:\sum _i{\lambda }_{ij}\le 1,\forall i\in \mathcal{I},\lambda \in [0,1]\hfill \\ C2:SINR-\delta \le 0,\hfill \\ C3:\sum _i{\lambda }_{ij}{B}_{ij}<B,\forall i\in \mathcal{I},\forall j\in \mathcal{J},\hfill \\ C4:\sum _i{\lambda }_{ij}{f}_{ij}<F,\forall i\in \mathcal{I},\forall j\in \mathcal{J},\hfill \\ C5:0\le R_{ij}\le 1,\forall i\in \mathcal{I},\forall j\in \mathcal{J},\hfill \\ C6:x_i\left[l\right]\in [X_{min},X_{max}],\forall i\in \mathcal{I},\hfill \\ C7:y_i\left[l\right]\in [Y_{min},Y_{max}],\forall i\in \mathcal{I},\hfill \\ C8:z_i\left[l\right]\in [Z_{min},Z_{max}],\forall i\in \mathcal{I}.\hfill \end{array}\right.\hfill \end{array}$$

(20)

In this context, P1 presents the goal of formulating the proposed system model problem $p{r}_j$, indicating the priority weights of tasks. $\mathbb{I}(T_{ij}^{total}\le t_j^{delay})$ is an indicator function, and we have $\mathbb{I}(T_{ij}^{total}\le t_j^{delay})=1$ if $T_{ij}^{total}\le t_j^{delay}$, otherwise, $\mathbb{I}(T_{ij}^{total}\le t_j^{delay})=0$. C1 represents that for user device j, no more than one UAV can serve it. C2 specifies that when users transmit data through the channel, if the SINR is below a certain threshold, it will be deemed a failure, where $\delta$ is the minimum SINR threshold. C3 and C4 indicate that the bandwidth and CPU frequency allocated by UAV i to user device j cannot exceed the total resource of the UAV. C5 states that the offloading ratio cannot be negative and cannot exceed 1. C6–8 specify that the coordinates of the UAV cannot exceed the scope of the application scenario.

4. KMADDPG-Based UAV Resource Allocation

4.1. Overview

In multi-agent scenarios, traditional reinforcement learning algorithms such as Deep Q-Network (DQN) face significant challenges. The primary issue is that during training, each agent treats other agents as part of the environment. As other agents change, the environment becomes unstable, violating the Markov assumption essential for Q-learning algorithms. Consequently, DQN struggles to converge in multi-agent environments.

Conversely, the MADDPG algorithm employs centralized training and decentralized execution strategies. This approach allows each agent to optimize its actions by considering the strategies of other agents, leading to more stable convergence and significantly enhancing the overall performance of the multi-agent system. Thus, MADDPG exhibits greater adaptability and efficiency in multi-agent contexts. Given the limited system resources, UAVs functioning as edge offloading nodes must carefully allocate resources to users.

4.2. Markov Decision Analysis

The proposed multi-UAV environment is typically considered a discrete partially observable Markov decision process (POMDP), as there are multiple interacting agents, each with its local observations and decision space, and the agents’ partial observations of the environment may be incomplete. The future state of a UAV depends only on the current state and the actions of other UAVs, not on past states. They are regarded as distributed agents capable of making resource allocation and task offloading decisions. The decisions made by UAVs are decentralized, but their actions affect other agents, necessitating coordination with other UAVs to achieve the objectives. The observations, states, actions, and rewards of distributed UAVs at time t are defined as $o\left(t\right), s\left(t\right), a\left(t\right)$ and $r\left(t\right)$:

(1)

State $s\left(t\right)$: In the proposed system model, the information obtained by agents through observations at time slot t is defined as $s\left(t\right)$. Due to each agent having its local observation and local decision space, $s\left(t\right)$ is typically incomplete. It can be composed of $\{s_i\left(t\right),s_{m_0}\left(t\right),\cdots ,s_{m_J}\left(t\right),s_g\left(t\right)\}$, where $s_g\left(t\right)$ represents the state information of the GBS. $s_i\left(t\right)$ denotes the state information of UAV i at time slot t, which can be expressed as

$$s_i\left(t\right)=\{x_i,y_i,z_i,B_i,f_i\},$$

(21)

where $B_i$ and $f_i$ represent the bandwidth and CPU computing power held by the UAV, respectively. $s_{m_J}\left(t\right)$ represents the state information of user equipment, and its components are as

$$s_{m_j}\left(t\right)=\{x_j,y_j,t_j^{delay},f_j,D_j,p{r}_j\},$$

(22)

where $t_j^{delay}$ represents the delay requirement for the task delivered by user j; $p{r}_j$ indicates the priority weight of the current task.

(2)

Observation $o\left(t\right)$: At time slot t, all agents’ local observation information $s\left(t\right)$ can be combined to form a global observation $o\left(t\right)$. It can be expressed as

$$o\left(t\right)=\{s_0\left(t\right),\cdots ,s_I\left(t\right),s_{m_0}\left(t\right),\cdots ,s_{m_J}\left(t\right),s_g\left(t\right)\}.$$

(23)

(3)

Action Space $a\left(t\right)$: The action space of KMADDPG mainly consists of three parts: decisions of task matching, edge offloading, and resource allocation. The components of $a\left(t\right)$ are as

$$a_i\left(t\right)=\{a_i^{mat}\left(t\right),a_i^{load}\left(t\right),a_i^{res}\left(t\right)\},$$

(24)

where $a_i^{mat}\left(t\right)$ denotes the decision of UAV matching with users, and $a_i^{mat}\left(t\right)=\{{\lambda }_{i0},\cdots ,{\lambda }_{iJ}\}$. $a_i^{of fload}\left(t\right)$ denotes the decision of task offloading rate, and $a_i^{res}\left(t\right)$ indicates the actions of resource allocation.

(4)

Reward Function $r\left(t\right)$: $r\left(t\right)$ is the sum of rewards completed by all UAVs. In the proposed scenario, the priority weights of user requests differ; it should focus on handling high-priority urgent requests. Therefore, the reward function is related to priority weights and completion status, which can be expressed as

$$R\left(t\right)=\sum _i^I\{{\gamma }_{ij}p{r}_j\omega +(1-{\gamma }_{ij})\xi \},$$

(25)

where ${\gamma }_{ij}$ is a binary coefficient indicating the completion status of the task. If the UAV can satisfy the user’s delay requirement, then ${\gamma }_{ij}$ = 1 and the task is considered completed. The task reward can be expressed as $R\left(t\right)={\gamma }_{ij}\times p{r}_j\times \omega$. Otherwise, ${\gamma }_{ij}$ = 0 and the task reward can be expressed as $R\left(t\right)=\xi$, where $\omega$ is the reward coefficient for task completion, and $\xi$ is the penalty coefficient for task failure.

4.3. MADDPG-Based Resource Allocation and Task Offloading

In the proposed multi-agent algorithm, we adopt a centralized training and decentralized execution (CTDE) strategy. This approach maximizes rewards while minimizing interactions between agents to enhance convergence speed. As show in Figure 2, we propose to use reinforcement learning combined with the K-means algorithm to optimize the initial positions of the agents, to achieve better optimization results.

(1)

Joint K-Means Algorithm: We propose a joint K-means algorithm for initializing the positions of UAVs. According to the path loss formula, path loss is positively correlated with the distance between UAVs and MUs. During centralized training, the system obtains the positions of all agents. We define sets ${\widehat{Q}}^{UAV}$ and ${\widehat{Q}}^{MU}$ to represent the positions of UAVs and users, respectively. Here, $\widehat{Q}$ is different from the Q mentioned earlier; it only represents the 2D coordinates of UAVs and users. The purpose of the K-means algorithm is to set I cluster centers $\{C_0,\cdots ,C_I\}$, assign J user devices to the nearest cluster centers $C_i$, and finally, the positions of the cluster centers will be the target positions for the UAVs. The optimization objective function of the K-means algorithm can be expressed as

$$arg\underset{C}{min}\sum _{i=1}^I\sum _{j\in C_i}{\parallel {\widehat{Q}}_j^{mu}-{\widehat{Q}}_i^{UAV}\parallel }^2.$$

(26)

The updating process of K-means can be expressed as

$${\widehat{Q}}_i^{UAV}(t+1)={\displaystyle \frac{1}{|C_j\left(t\right)|}}\sum _{i\in C_I\left(t\right)}{\widehat{Q}}_j^{mu}.$$

(27)

(2)

Deep Reinforcement Learning Algorithm: To avoid the additional load and delay caused by information exchange between the central controller and UAVs, this paper proposes a distributed joint resource management algorithm based on MADDPG. We consider a centralized training decentralized execution DRL mechanism. In the CTDE strategy, the collaborative operation of agents can be viewed as an optimization problem that maximizes expected rewards. This optimization problem can be expressed as

$$J=arg\underset{\mu }{max}\mathbb{E}\left[{\gamma }^{t-1}r\left(t\right)\right],$$

(28)

where $\mu =\{{\mu }_0,\cdots ,{\mu }_I\}$ represents the policy network of the agent. ${\gamma }^{t-1}$ denotes the discounted return at time slot t. We set the parameters of the policy network as $\theta =\{{\theta }_0,\cdots ,{\theta }_I\}$, then the expected return gradient of the UAV can be written as

$${\nabla }_{{\theta }_i}J\left({\theta }_i\right)={\mathbb{E}}_{s,a\sim D}\left[{\nabla }_{{\theta }_i}log{\mu }_i\left(a_i\mid o_i\right)Q_i^{\mu }(\ast )\mid a_i={\mu }_i\left(o_i\right)\right],$$

(29)

where D is the experience replay buffer that stores experience tuples $\left\{S,a,r,S^{\prime }\right\}$. $Q_i^{\mu }\left(*\right)$ is the value function. The loss function of the Critic network for UAV i can be expressed as

$$L\left({\theta }_i\right)={\mathbb{E}}_{S,a,r,S}\left[{\left(Q_i^{\mu }\left(S,a_1,\cdots ,a_I\right)-\widehat{y}\right)}^2\right],$$

(30)

where $\widehat{y}$ is the TD error, which can be expressed as

$$\widehat{y}=r_i+{\left.\gamma Q_i^{{\mu }^{-}}\left(S,\dot{a_1},\cdots ,a_I\right)\right|}_{a_i={\mu }^{-}\left(o_i^{-}\right)},$$

(31)

where $Q_i^{{\mu }^{-}}$ and ${\mu }_i^{-}(*)$ are the target Critic network and target Actor network, respectively. The specific neural network update process is shown in Algorithm 1.

(3)

Complexity Analysis: The complexity analysis of the KMADDPG algorithm can be divided into two main components: the K-means algorithm and the neural network. For the K-means algorithm, each iteration requires the position coordinates of each UAV and MU, resulting in a time complexity of $O(I\times J)$. The number of iteration is denoted as $N_k$. After iterating, the total time complexity for the K-means algorithm is given by

$$O(2\times N_k\times I\times J).$$

(32)

   Since the dimensionality of the 2D coordinates and the number of iterations are generally constants, this complexity simplifies to $O(I\times J)$. For the neural network part, the efficiency of the proposed algorithm is further evaluated through its time complexity. Both offline training and online execution involve mapping states to actions using deep neural networks. The computational complexity of the deep neural network, denoted as $O\left(NN\right)$, is expressed as follows [39]:

$$O(NN)=O(\sum _n^{N^{layer}}{l}_n{l}_{n+1}),$$

(33)

where $N^{layer}$ is the number of layers, and $l_n$ represents the number of neurons in the n-th layer. The dimensions of input and output layers of the deep neural network depends on the size of state space and action space. For $N_{ep}$ episodes, S steps, and $N_{exp}$ experiences, the algorithm’s time complexity is

$$O(\sum _n^{N^{layer}}{l}_n{l}_{n+1}\times N^{exp}\times N_{ep}^S).$$

(34)

Algorithm 1 KMADDPG Algorithm
Initialize: Max_episode, Max_ep_len;
1:	for episode from 1 to Max_episode do
2:	$\mathbf{Initialize}\mathbf{state}:S;\mathrm{local}\mathrm{observations}:o_i\mathrm{for}\mathrm{any}\mathrm{agent}$
3:	$i\in \mathcal{I};\mathrm{cluster}\{C_0,\cdots ,C_I\}\mathrm{centroids}\mathrm{randomly};$
4:	for each MU ${\widehat{Q}}_j^{mu}$ do
5:	Calculate the distance between UAV and MU;
6:	Find the closest centroids ${\widehat{C}}_i=arg{min}_i{\widehat{d}}_{ij}$;
7:	Assign ${\widehat{Q}}_j^{mu}$ to closet centroids $C_i$;
8:	endfor
9:	Update centroids position:
10:	${\widehat{Q}}_i^{UAV}={\displaystyle \frac{1}{|C_j\left(t\right)|}}{\sum }_{i\in C_I\left(t\right)}{\widehat{Q}}_j^{mu}$;
11:	for t = 1 to Max_ep_len do
12:	$\mathrm{For}\mathrm{each}\mathrm{UAV},\mathrm{move}\mathrm{to}{Q}^{UAV}\mathrm{from}\mathrm{K}\text{-}\mathrm{Means};$
13:	$\mathrm{UAV}\mathrm{select}\mathrm{action}{a}_i;$
14:	$\mathrm{Get}\mathrm{the}\mathrm{reward}{r}_i\mathrm{by}\mathrm{environment};$
15:	$\mathrm{Store}\left(S,a,r,S^{\prime }\right)\mathrm{in}\mathrm{the}\mathrm{replay}\mathrm{buffer}\mathcal{D};$
16:	$S\leftarrow S^{\prime }$
17:	for agent i = 1 to I do
18:	$\mathrm{Sample}\mathrm{randomly}\mathrm{a}\mathrm{batch}\mathrm{of}$ N $\mathrm{samples}$
19:	$\left(S_j,a_j,r_j,S_j^{\prime }\right)\mathrm{sampled}\mathrm{from}\mathcal{D};$
20:	$\mathrm{Calculate}\mathrm{TD}\mathrm{error}\mathrm{according}\mathrm{to}\left(31\right);$
20:	$\mathrm{Update}\mathrm{critic}\mathrm{network}\mathrm{using}\mathrm{minimizing}\mathrm{loss}L\left({\vartheta }_i\right);$
22:	$\mathrm{Update}\mathrm{actor}\mathrm{network}\mathrm{through}\mathrm{gradient}{\nabla }_{{\vartheta }_i}J\left({\mu }_i\right);$
23:	endfor
24:	$\mathrm{Update}\mathrm{the}\mathrm{target}\mathrm{network}\mathrm{parameters}\mathrm{for}\mathrm{agent}i;$
25:	${\omega }_{new}^{i-}\leftarrow \tau {\omega }_{new}^i+(1-\tau ){\omega }_{now}^{i-};$
26:	${\vartheta }_{new}^{i-}\leftarrow \tau {\vartheta }_{new}^i+(1-\tau ){\vartheta }_{now}^{i-};$
27:	endfor
28:	endfor

5. Experimental Design and Analysis

5.1. Simulation Parameters

In this section, we evaluate the proposed KMADDPG-based UAV resource allocation and edge offloading optimization scheme. Specifically, we consider a disaster area of 200 m × 200 m. We consider a GBS set up at the center of the disaster area, establishing an FSO communication link with the UAVs. Meanwhile, the UAVs serve as relay BS or edge offloading nodes, providing radio communication services or computational resources to MUs. To simplify the system model, we set the penalty coefficient to −1 and the reward coefficient to 1. The reward is the cumulative reward of tasks completed by all UAVs in each time slot. We used 8 MUs, 3 UAVs, and 1 GBS as the experimental subjects. The detailed parameters are listed in

Table 2. List of main simulations parameters.

Parameters	Value
Learning rate of actor	2 $\times {10}^{-5}$
Learning rate of Critic	5 $\times {10}^{-4}$
The number of Episode	15,000
Episode length	10
Channel bandwidth $\left(B\right)$	50 $\mathrm{MHz}$
Cpu frequency of UAV ($f_{ij}$)	$1.5\times {10}^9$
Cpu frequency of MU ($f_{local}$)	${10}^9$
Size of task ($D_{ij}$)	0.1 M–1.5 M
Requested delay of task ($t_j^{delay}$)	12 ms–15 ms
LoS additional path loss $\left({\eta }^{LoS}\right)$	$1\mathrm{dB}$
NLoS additional path loss for $\left({\eta }^{NLoS}\right)$	$20\mathrm{dB}$
LoS path loss exponent $\left({\beta }^{\mathrm{LoS}}\right)$	2.09
NLoS path loss exponent $\left({\beta }^{\mathrm{NLoS}}\right)$	3.75
Noise power $\left(\sigma \right)$	−95 $\mathrm{dBm}$
Operating altitude of UAV $\left(h_t\right)$	$100\mathrm{m}$
Transmit power of UAV i $\left(p^{\mathrm{RF}}\right)$	30 $\mathrm{dBm}$
Transmission power of FSO $\left(p^{\mathrm{FSO}}\right)$	200 $\mathrm{mW}$
Efficiencies of optic $\left({\varkappa }_t\right)$	0.8
Receiver diameter of FSO channel $\left(\varsigma \right)$	$0.06\mathrm{m}$
Transmitter divergence of FSO channel (${\upsilon }_t$)	$2.07\times {10}^{-4}$ rad
Wavelength of FSO channel $\left(\lambda \right)$	$1550\mathrm{mm}$
Sensitivity of receiver $\left(N_b\right)$	100 photons/bit

. The algorithms to be evaluated are as follows:

•

MeanResource MADDPG (MeanRes): An algorithm where the neural network selects the matching objects, but the offloading ratio and resource allocation ratio are evenly distributed.

•

NearbyMatch MADDPG (NearbyMatch): An algorithm where the neural network selects the offloading and resource ratios, and the service objects are the nearest users.

•

Base MADDPG (Base): An algorithm where the neural network selects the matching objects, offloading, and resource ratios.

•

Proposed KMADDPG (Proposed): An algorithm that integrates K-means into the Base MADDPG algorithm.

5.2. Convergence Analysis

Figure 3 illustrates the trend of average system rewards obtained by each UAV per episode, and Figure 4 presents the tendency of total success rate obtained by each UAV per episode. This experiment evaluates the convergence of the proposed algorithm in comparison to other algorithms. It can be observed that the rewards for KMADDPG, Base, NearbyMatch, and MeanResource decrease in that order. The MeanRes algorithm exhibits significant fluctuations in performance, performs the worst, and fails to converge. This is due to the algorithm’s inability to customize based on user demand differences, resulting in a significant waste of computational resources.

The NearbyMatch algorithm performs somewhat better, with average rewards and average total success rates of 5.8 and 78% converges when the episode is equal to 8000, as it devises strategies based on user demand. However, the limitation of NearbyMatch algorithm often causes it to fall into local optima. This limitation is that during training, MUs may cluster around a single UAV and offload task to it, leading to an inability of that UAV to satisfy all demands of MUs while the remaining UAVs remain idle.

The Base algorithm demonstrates better performance, with average rewards and average total success rates of 6.8 and 86% converges when the episode is equal to 10,000. The proposed KMADDPG algorithm addresses these shortcomings by integrating the KMeans algorithm to optimize UAV positioning, reducing path loss, and achieving better convergence performance compared to Base MADDPG. Consequently, it achieved the best results, with average rewards and average total success rates of 7.8 and 90% converges when the episode is equal to 3000. The proposed algorithm is capable of converging under minor constraints and has a faster convergence rate than the other algorithm.

5.3. Priority Analysis

Figure 5 and Figure 6 depict the performance of various algorithms in terms of task completion within a single episode and their ability to handle tasks of different priority levels, respectively. Figure 5 shows the number of completed tasks within one episode. The KMADDPG algorithm completed 719 tasks, which is 8%, 12%, and 41% more than those completed by Base, NearbyMatch, and MeanResource, respectively. This indicates that the proposed algorithm exhibits superior task scheduling and processing capabilities. Figure 6 illustrates the performance of different algorithms in handling tasks with varying priorities, where P1, P2, and P3 represent low-, medium-, and high-priority tasks, respectively. It is evident from Figure 6 that all four algorithms perform better on higher-priority tasks, indicating that high-priority tasks receive more resource allocation. The proposed algorithm outperforms the other three algorithms across all three priority levels, with an increase in task completion by 12.5%, 5%, and 6% for low, medium, and high priorities, respectively, compared to Base. This demonstrates that the proposed algorithm not only maximizes the number of successful tasks but also exhibits superior handling capabilities for high-priority tasks.

5.4. Convergence under Different Proportions

Figure 7 and Figure 8 illustrate the trends of the average system reward and total success rate, respectively. In the proposed system model, as the number of UAVs and users increases, resources increase proportionally, making the environment more complex. Particularly, the increase in the number of agents means that each agent’s decision can more frequently affect the states of other agents, making it harder for the results to converge. We compare different combinations of ratios to demonstrate that the proposed algorithm can still perform well even as the number of agents increases, though inevitably at a slower convergence rate. The reason for increasing resources proportionally is to avoid task failures due to insufficient resources. This experiment evaluated the convergence speed of various algorithms under different ratios of UAV and mobile user (MU) combinations. With the increase in the number of UAVs and users, resources increase proportionally, and the environment becomes more complex. To verify that the proposed algorithm can achieve good convergence results in complex situations, we used a UAV ratio and MU of 3:8 (simplified to (3, 8)) as the baseline, adjusting their numbers proportionally. From Figure 7, it is evident that the convergence rewards vary with different UAV combinations: the (2, 4) combination converges around 4, the (6, 16) combination converges around 15, and the (9, 24) combination converges around 22. This variation is due to the different number of users each combination serves. The convergence speed decreases sequentially because, as the number of agents and users increases, the system environment becomes more complex, necessitating more training for the neural network to achieve optimal results. However, as shown in Figure 8, even as the environment complexity increases, the final convergence effect still reaches 90%, demonstrating that the proposed algorithm performs well in more complex environments.

6. Conclusions

To address the challenges faced by multi-UAV rescue systems utilizing a hybrid FSO/RF dual-hop model, we introduce a priority-driven resource allocation strategy. This innovative approach is designed to optimize the completion rate of tasks, with a particular emphasis on ensuring that high-priority tasks are efficiently managed and expedited. This paper transforms the problem into a discrete partially observable Markov decision process and develops a KMADDPG algorithm to improve convergence speed. Simulation results indicate that the KMADDPG algorithm outperforms other DRL algorithms in enhancing system performance. In future work, we will explore resource allocation strategies for UAVs in terms of energy efficiency and transmission power, and incorporate dynamic priority mechanisms.