Joint Optimization of Task Completion Time and Energy Consumption in UAV-Enabled Mobile Edge Computing

Abstract

Unmanned Aerial Vehicles (UAVs) hold great promise for Mobile Edge Computing (MEC) owing to their flexible mobility, rapid deployment, and low-cost characteristics. However, UAV-enabled MEC still faces challenges in terms of the real-time arrival of computational tasks, energy reservation, and the actual response efficiency of the system. In this study, we focus on a UAV-enabled MEC scenario, where multiple UAVs function as airborne edge servers, offering computation services to multiple ground-based user devices (UDs). We aim to minimize the cost of the MEC system by optimizing the computation offloading policy. Specifically, we take task latency into account to ensure the timeliness of real-time tasks. The Lyapunov optimization method is employed to maintain a uniform and stable queue for energy consumption. Additionally, we draw on the concept of maximum completion time in shop-floor scheduling to optimize the actual response latency. To this end, we propose a joint optimization algorithm. First, the joint optimization problem is transformed into a per-time-slot real-time optimization problem (PROP) using the Lyapunov optimization framework. Then, a reinforcement learning method, LyraRD, is proposed to solve the PROP. Experimental results verify that the proposed approach outperforms the benchmarks in terms of system performance.

1. Introduction

With the widespread adoption of the Internet of Things (IoT) and wireless communication technologies, numerous intelligent applications with strict computational latency requirements have emerged, such as real-time video, autonomous driving, and online gaming [1]. Nevertheless, the terminal devices for these applications are limited in computing power and battery capacity, making it arduous to meet the low-latency demands. Mobile Edge Computing (MEC) is regarded as a promising solution due to its real-time computing and energy-saving capabilities [2,3,4]. Based on MEC, user devices can delegate computationally intensive tasks to edge servers instead of relying on remote data centers. This not only compensates for the computing power shortage of user devices but also reduces the response time.

Initially, most MEC servers were deployed on the ground [5,6,7]. In fact, these ground-based MEC servers have provided convenient services to users. However, they typically cover only a portion of the user population [2,8,9]. Fortunately, the rapid development of Unmanned Aerial Vehicles (UAVs) offers new opportunities to address the above-mentioned issues. UAVs can carry computing devices and hover in the air, enabling MEC servers to be deployed flexibly to meet user needs. As depicted in Figure 1, two UAVs hover in air, with different levels of battery capacity, computing resources, and bandwidth resources. We use a three-tuple to represent the different normalized resource values of the UAVs, i.e., $UA{V}_1:(20, 2, 80\%)$ and $UA{V}_2:(20, 1.5, 60\%)$. Two user devices, $U{D}_1$ and $U{D}_2$, each with unit computing resources, generate computation tasks at 5-second intervals in different time slots. $U{D}_1$ generates tasks $\{a_1=5, a_2=3\}$, $\{a_3=6, a_4=4\}$, $\{a_5=2, a_6=3\}$ and $U{D}_2$ generates tasks $\{b_1=3, b_2=1\}$, $\{b_3=5, b_4=4\}$, and $\{b_5=6, b_6=3\}$, where the values represent the units required of computing resources in three slots, respectively. Without loss of generality, we assume that UAVs consume one unit of energy for executing one unit of a computation task.

Figure 2 presents two solutions to offload the computing tasks. In Figure 2a, $U{D}_1$ offloads tasks $\{a_1, a_3\}$ to $UA{V}_1$, and $\{a_4, a_6\}$ to $UA{V}_2$, while $U{D}_2$ offloads tasks $\{b_4, b_5\}$ to $UA{V}_1$, and $\left\{b_1\right\}$ to $UA{V}_2$. In the first slot (i.e., $T=1$), task $b_1$ is executed in $UA{V}_2$, taking up $3/1.5=2$ seconds and $3/1=3$ units of energy; task $a_1$ is executed in $UA{V}_1$, taking up $5/2=2.5$ seconds and $5/1=5$ units of energy; task $a_2$ is executed in $U{D}_1$, taking up $3/1=3$ seconds; and task $b_2$ is executed in $U{D}_2$, taking up $1/1=1$ seconds. Then, the makespan of all tasks of $U{D}_1$ is $max(2.5, 3)=3$ seconds, while the makespan of all tasks of $U{D}_2$ is $max(2, 1)=2$ seconds. Similarly, we can easily obtain the energy cost and makespan in the next two slots (see the detailed values in Figure 2a). Figure 2b presents the other solution. We can easily find that the total percentages of energy use of $UA{V}_1$ and $UA{V}_2$ in Figure 2a are ${\displaystyle \frac{5+10+6}{20}}=105\%$ and ${\displaystyle \frac{3+4+3}{20}}=50\%$, while in Figure 2b, they are ${\displaystyle \frac{5+8+6}{20}}=95\%$ and ${\displaystyle \frac{3+6+3}{20}}=60\%$. The average makespan in Figure 2a is ${\displaystyle \frac{3+2+3+5+2+3}{6}}=3$, and in Figure 2b, it is ${\displaystyle \frac{3+4+2+2+5+3}{6}}=3.16$. Considering that the scenario in Figure 1 is online, meaning that users will constantly generate tasks, to ensure that users have a good experience in offloading tasks to the UAVs, the system should have rapidity to finish tasks in each slot (i.e., minimize the average makespan) as soon as possible, and durability to serve (i.e., minimize the maximum energy cost) as long as possible.

In this paper, we are committed to exploring the optimization of both efficiency and durability to identify an optimal task offloading solution in UAV-enabled MEC. However, solving this problem poses three major challenges:

(i) The existing research [10,11,12] predominantly relies on offline task scheduling or prior knowledge of tasks. In contrast, practical applications such as autonomous driving and real-time video analytics typically involve the arrival of stochastic real-time task data. This implies that the current decision making must consider the potential influence of future tasks.

(ii) Most research [4,7,13] efforts concentrate on keeping energy consumption below a certain limit or minimizing the overall energy consumption throughout the process. However, effective energy management should not only aim at minimization but also strive for balance. This is crucial to ensure that UAVs equipped with high computational resources can operate for extended durations.

(iii) It is commonly assumed that task offloading occurs at the beginning of each time slot [3,5,6]. Given that transmitting the task to the UAV will inevitably consume the available time within the time slot, careful bandwidth allocation is essential.

To address these challenges, we model the joint optimization problem of task scheduling in an online fashion instead of an offline one. This approach optimizes the average values across all time slots. Next, to ensure the stable operation of the MEC system with randomly arriving tasks, we utilize Lyapunov optimization theory to introduce a virtual queue that can evenly distribute energy consumption. To solve the online model more accurately, we further decompose the multi-stage stochastic problem into a series of deterministic problems, which can be effectively solved by reinforcement learning. The main contributions of this study are as follows:

• We present a novel online optimization framework to help users complete tasks quickly and provide them with more durable services. The framework integrates Lyapunov optimization and Mixed Integer Nonlinear Programming (MINLP), which dynamically adapts to real-time task arrivals while jointly optimizing task response time (by minimizing makespan) and energy consumption. Through Lyapunov optimization, we ensure a uniform energy distribution across the UAV, thereby effectively mitigating energy consumption fluctuations caused by bursty tasks while responding in a timely manner in the absence of a priori knowledge of the data.
• We introduce Lyapunov virtual energy queues, decompose the problem into a real-time optimization problem (PROP) for each time slot via Lyapunov drift, and design the reinforcement learning-based algorithm LyraRD. LyraRD uniquely combines Lyapunov-induced stability constraints with adaptive decision making to reduce the computational complexity to millisecond response times.
• We investigate the performance of our method under different parameter settings. With extensive experiments, we demonstrate that our method performs better than the benchmarks.

The rest of the paper is organized as follows. Section 2 summarizes the related work. Section 3 introduces the related system model and problem formulation in detail. Then, we describe the Lyapunov-based problem transformation in detail in Section 4. In Section 5, we introduce the two-stage optimization algorithm and theoretical analysis. Our evaluation method and experimental results are shown and analyzed in Section 6. Finally, this paper is concluded in Section 7.

2. Related Work

Due to the high mobility and flexibility, UAV-enabled MEC systems have attracted widespread attention in recent years.

2.1. Real-Time Task Arrival

Most studies have based their strategies on user task requirements given as a priori knowledge [10,11]. However, in most MEC applications, such as autonomous driving and online games, computational tasks arrive randomly in real time [12], and there is a lack of access to the a priori knowledge of the tasks. Therefore, considering the random arrival of task data, Wan et al. [14] developed a path planning algorithm based on deep reinforcement learning by discretizing the action space into a finite set, including eight steering directions and hovering modes. Yang et al. [13] considered the stochastic nature of user tasks and developed a Lyapunov optimization based on perturbation-based algorithm based on an online algorithm to jointly optimize the energy consumption and task processing rate of UAVs while maintaining the stability of long-term data queues. When considering user mobility and random task arrivals, Yang et al. proposed an online control algorithm for UAV trajectory and resource scheduling to minimize a certain budgeted energy consumption of the UAV in a certain budgeted energy consumption by converting the long-term optimization problem into an optimization problem for the real-time control of a real-time control system in [15]. Yan et al. [16] proposed an artificial noise-based UAV-assisted non-orthogonal multiple access (NOMA) secure communication scheme, whose channel dynamics analysis and resource allocation mechanism provide a theoretical basis for future extension of security enhancement strategies. Ali et al. [17] proposed a novel multi-objective adaptive learning framework called MOALF-UAV-MEC, tailored for dynamic IoT environments, which provides a scalable and adaptive solution for deploying computational resources in infrastructure-limited regions, ad hoc events or emergencies. However, they simply add the time delays linearly as the completion delay of the task, which is biased compared to the actual response delay of the task. For this reason, in this paper, we introduce the maximum completion delay (i.e., makespan) in the scheduling problem to represent the actual response delay of the task and improve the response efficiency of MEC by reducing it.

Meanwhile, most of the existing studies consider that the UAV computes the offloaded tasks at the initial moment of each time slot. Zeng et al. [11] optimized resource allocation and UAV trajectory to reduce the waiting latency of the task queue. Sheng et al. [18] considered the simultaneous occurrence of the time-varying/random ground channel and the air-to-ground line-of-sight channel, and optimized the UAV trajectory and the task offload strategy to ensure the robustness of task processing and, at the same time, minimize energy consumption.

All of the above studies theoretically assume that the computational delay of the tasks offloaded to the UAV should be less than or equal to the time slot length. However, when the computational delay of the offloaded tasks happens to be equal to the time slot length, due to the effect of the task transmission delay, it will result in part of the tasks not being able to become computed in all time slots (e.g., task $b_2$ on the $UA{V}_1$ at $T=2$ in Figure 2). If this remaining task is ignored, it will lead to part of the task losing packets, so this paper considers deferring the remaining task to the next time slot, which creates a period of deferred delay, thus causing the UAV pair to start calculating the task not at the initial moment of the time slot. The above effects should be taken into account when calculating the task response delay.

2.2. UAV Energy Management

Traditional UAV energy control mainly considers the available energy consumption of a given UAV and ensures that the energy consumption of the entire process is less than this [11,19,20], and this approach of energy control is mostly applicable to task scheduling problems with given a priori knowledge, where the allocation of the energy consumption of the whole MEC process is adjusted with the number of tasks unloaded in the time slot. New studies on UAV energy management and control are also underway. Michailidis et al. [21] maximize the minimum security computational efficiency (SCE) in a UAV-RIS-assisted MEC-IoT network in the presence of airborne and ground eavesdroppers by jointly optimizing the transmission power allocation, time-slot scheduling, task assignment, and phase-shift of the RIS to ensure that the system’s energy consumption and security performance are optimized. Lakew et al. [22] proposed a MEC model assisted in LEO satellites, in which UAVs can acquire a series of energy packets at the beginning of each time slot by the energy harvesting (EH) technique, which in turn efficiently manages the energy harvesting and consumption of the EH-UAVs to optimize the performance of the whole system and the mission accomplishment rate. However, when there is a lack of a priori knowledge about the tasks, i.e., the tasks arrive randomly, this energy management approach will lead to the uneven distribution of energy consumption and thus insufficient reservation, which will reduce the computational efficiency in the face of unexpected large-scale tasks and thus increase the response latency of the whole MEC system. For this reason, Lyapunov optimization methods are commonly used to stabilize the energy use of each time slot.

Wang et al. [23] considered the joint optimization problem of UAV trajectories and offloading decisions and used Lyapunov optimization methods to satisfy the long-time energy constraints of UAVs while making the energy allocation uniform to minimize the sum of the service latency linearity of each time slot. Qin et al. [24] considered task cache policies, UAV trajectories, and resource allocation policies to reduce the sum of service delay linearity.

In recent years, reinforcement learning has shown significant potential in UAV energy optimization. For example, Li et al. [25] balanced the multidimensional constraints of ground–air cooperative communication using multi-objective deep reinforcement learning, and proposed a ground–air cooperative communication framework based on Evolutionary Multi-objective Deep Reinforcement Learning (EMODRL), which balanced the multidimensional objectives of latency, energy consumption, and coverage through Pareto frontier search. Prakhar et al. [26] presented the SFRL method for the purpose of distributed ML model learning in wireless UAV-assisted MEC AI model networks. Compared to existing distributed learning algorithms, the proposed method yields comparable high test accuracy performance while consuming less energy. And Wu et al. [27] designed an evolutionary reinforcement learning framework for incomplete information scenarios to improve the strategy robustness through action sequence search, and its evolutionary exploration mechanism can provide a reference for the improvement of strategy generalizability under stochastic task arrival.

2.3. Differences of Our Work

Sequential Task Execution with Dynamic Deferral. Unlike existing approaches that either ignore deferred tasks or assume instantaneous computation starts at slot boundaries, we explicitly model the inter-slot dependency of task execution. By deferring unfinished tasks to subsequent slots and accounting for their cascading delays, our framework minimizes the actual response latency (via makespan optimization) while maintaining queue stability. This contrasts with methods like  [13,15], which linearly aggregate delays and fail to capture the nonlinear impact of deferred computations.

Non-Uniform Energy Balancing via Lyapunov-Guided RL. While previous studies can address problems such as UAV task offloading based on Lyapunov optimization [23,24] stabilize energy queues, they do not integrate reinforcement learning (RL) to adaptively balance energy consumption across heterogeneous UAVs. Our LyraRD algorithm uniquely combines Lyapunov optimization with RL to dynamically adjust offloading decisions, ensuring that UAVs with higher resource capacities sustain longer operational durations. This prevents energy “hotspots” and improves the resilience of the system, addressing a critical gap in works such as  [17,25], which treat energy allocation statically.

Inspired by these gaps, we consider a UAV-enabled MEC system with randomly arriving tasks. Task timeliness is ensured by deferring unprocessed tasks from the previous moment to the next moment, while UAV energy usage is controlled throughout the process to minimize the UAV energy consumption and actual response time of the system.

3. System Model and Problem Formulation

In this section, we first introduce a UAV-enabled MEC framework that is designed to offload computational tasks from UDs. Next, we model the latency and energy consumption of task offloading, transmission, and computation. We then present a joint optimization problem with the goal of minimizing the weighted sum of the actual task response latency and energy usage of UAVs across the MEC system.

3.1. System Overview

As illustrated in Figure 1, a MEC system enabled by UAVs consists of I UDs and J rotation UAVs, where the set $\mathcal{I}=\{1,2,\cdots ,I\}$, and $\mathcal{J}=\{1,2,\cdots ,J\}$. The system timeline is discretized into T equal time slots [28], i.e., $t\in \mathcal{T}=\{1,2,\cdots ,T\}$, where the duration of each time slot is denoted as $T_s$. The UAV acts as an aerial edge server of MEC, providing computation offloading services for UDs in each time slot.

UD Model. We assume that there are several computation tasks that arrive at each UD in each time slot [29]. For $U{D}_i(i\in \mathcal{I})$, the properties of this UD in time slot t can be expressed as $M_i\left(t\right)=(f_i,{\mathrm{\Phi }}_i\left(t\right),{\mathrm{P}}_i\left(t\right))$, where $f_i$ represents the local computation capability of $U{D}_i$, and ${\mathrm{P}}_i\left(t\right)$ represents the coordinates of $U{D}_i$. The task that reaches $U{D}_i$ at time t is ${\mathrm{\Phi }}_i\left(t\right)$, which consists of k tasks: ${\mathrm{\Phi }}_i\left(t\right)=({\alpha }_i^1\left(t\right),{\alpha }_i^2\left(t\right),{\alpha }_i^3\left(t\right),\dots ,{\alpha }_i^k\left(t\right))$. These tasks are expressed in the form ${\alpha }_i^k\left(t\right)=\{D_i^k\left(t\right),{\eta }_i^k\left(t\right)\}$, where $D_i^k\left(t\right)$ and ${\eta }_i^k\left(t\right)$ represent the amount of data and computing intensity of task ${\alpha }_i^k\left(t\right)$ generated at time t.

UAV Model. The attributes of $UA{V}_j(j\in \mathcal{J})$ are represented as $U_j\left(t\right)=(r_j,{\omega }_j,{\mathrm{P}}_j\left(t\right),H)$, where $r_j$ and ${\omega }_j$ represent the proportion of computing resources and bandwidth, respectively. ${\mathrm{P}}_j\left(t\right)$ represents the coordinates of the $UA{V}_j$ at time t, and H represents the altitude [30].

Decision Variables. Decision variables include the task offloading strategy and the UAV resource allocation strategy. (i) Task Offloading Decision. For task ${\mathrm{\Phi }}_i\left(t\right)$, define a binary variable $a_{i,j}^k\left(t\right)$ to represent the decision to load $U{D}_i$ in the time period t, where $a_{i,j}^k\left(t\right)=0$ means that the $k_{th}$ task of $U{D}_i$ in the time period t is processed locally, and $a_{i,j}^k\left(t\right)=1$ means that the $k_{th}$ task of $U{D}_i$ in time period t is loaded into $UA{V}_j$ for processing. (ii) Resource Allocation Decision. For UAVs, the communication resources and bandwidth resources allocated to the task k unloaded at time t are $r_j^{i,k}\left(t\right)$ and ${\omega }_j^{i,k}\left(t\right)$.

These factors affect each other, so joint decision making is required.

3.2. Communication Model

The probabilistic line-of-sight (LoS) channel model is used to simulate the communication between $U{D}_i$ and $UA{V}_j$ [19]. First, the probability of LoS and NLoS communication between $U{D}_i$ and $UA{V}_j$ in time period t is shown in (1) and (2) [31]:

$$P_{i,j}^{\mathrm{los}}\left(t\right)={\displaystyle \frac{1}{1+{\xi }_{1}exp\left(-{\xi }_2\left({\theta }_{i,j}\left(t\right)-{\xi }_1\right)\right)}}$$

(1)

and

$${\overline{P}}_{i,j}^{los}\left(t\right)=P_{i,j}^{\mathrm{los}}\left(t\right)+\left(1-P_{i,j}^{\mathrm{los}}\left(t\right)\right)\kappa$$

(2)

where ${\xi }_1$ and ${\xi }_2$ are constants depending on the propagation environment, ${\theta }_{i,j}\left(t\right)={\displaystyle \frac{180}{\pi }}arcsin{\displaystyle \frac{H}{d_{i,j}\left(t\right)}}$ represents the elevation angle, $d_{i,j}\left(t\right)$ represents the straight-line distance between $U{D}_i$ and $UA{V}_j$, and $\kappa$ represents the additional attenuation factor.

According to  [32], the communication rate of $U{D}_i$ in time slot t can be expressed as Equation (3):

$$R_{i,j}^k\left(t\right)={\omega }_j^{i,k}\left(t\right){\phi }_{i,j}\left(t\right)$$

(3)

where ${\phi }_{i,j}\left(t\right)$ is the spectrum efficiency of $U{D}_i$, which can be calculated as (4)–(5):

$$\begin{array}{c}\hfill {\phi }_{i,j}\left(t\right)={log}_2\left(1+{\displaystyle \frac{{\theta }_i\left(t\right)}{\left(\right||{\mathrm{P}}_j\left(t\right)-{\mathrm{P}}_i\left(t\right)|{|}^2+H^2{)}^{\mu }}}\right)\end{array}$$

(4)

$$\begin{array}{c}\hfill {\theta }_i\left(t\right)={\displaystyle \frac{P_i{\beta }_0{\overline{P}}_{i,j}^{los}\left(t\right)}{N_0}},\mu ={\displaystyle \frac{\overline{\mu }}{2}}\end{array}$$

(5)

where ${\beta }_0$ represents the channel gain at the reference distance of $1m$, $N0$ represents the noise power, $P_i$ is the transmission power of $U{D}_i$, and $\overline{\mu }$ represents the path loss exponent.

3.3. Computation Model

Local Computation. The local completion delay of the task ${\alpha }_i^k\left(t\right)$ arriving at $U{D}_i$ at time slot t can be calculated as Equation  (6):

$$T_i^k{\left(t\right)}^{loc}={\displaystyle \frac{{\eta }_i^k\left(t\right)D_i^k\left(t\right)}{f_i}}$$

(6)

Correspondingly, its energy consumption can be calculated as (7) [33]

$$E_i^k{\left(t\right)}^{loc}=k_0{\left(f_i\right)}^3{T}_i^k{\left(t\right)}^{loc}$$

(7)

where $k_0$ represents the hardware architecture parameter.

Edge Computation. The transmission delay of the task ${\alpha }_i^k\left(t\right)$ arriving at $U{D}_i$ at time slot t to be offloaded to $UA{V}_j$ can be calculated as Equation  (8):

$$T_{i,j}^k{\left(t\right)}^P={\displaystyle \frac{D_i^k\left(t\right)}{R_{i,j}^k\left(t\right)}}$$

(8)

And the edge computing delay of this process can be calculated as Equation  (9):

$$T_{i,j}^k{\left(t\right)}^C={\displaystyle \frac{{\eta }_i^k\left(t\right)D_i^k\left(t\right)}{r_j^{i,k}\left(t\right)}}$$

(9)

Accordingly, the energy consumption of transmission can be expressed as (10):

$$E_{i,j}^k{\left(t\right)}^P=P_i{\displaystyle \frac{D_i^k\left(t\right)}{R_{i,j}^k\left(t\right)}}$$

(10)

where $P_i$ represents the transmission power.

The energy consumption of edge computing can be expressed as Equation  (11):

$$E_{i,j}^k{\left(t\right)}^C=\varpi {\eta }_i^k\left(t\right)D_i^k\left(t\right)$$

(11)

where $\varpi$ represents the energy consumption of the UAV per unit CPU cycle.

3.4. Cost Model

UD Cost. Similar to [34,35], from the user’s perspective, in each time period, the task completion delay $T_i\left(t\right)$ and UD energy consumption $E_i\left(t\right)$ are considered the cost of UD, which reflects the quality of experience of the UD.

In fact, the task calculations between each UD and UAV are parallel within a certain period of time, so simply optimizing the linear sum of all delays is meaningless to reduce the delay of the entire MEC process. Therefore, this paper borrows the concept of the minimum maximum completion time in the flexible workshop scheduling problem.

$T_i\left(t\right)$ can be formulated as (12)

$$T_i\left(t\right)=\mathrm{Min}\left.\left\{\mathrm{Max}\left.\left\{T_i^{loc}\left(t\right),T_i^{mec}\left(t\right)\right.\right\},T_s\right.\right\}$$

(12)

where

$$T_i^{loc}\left(t\right)=\sum _{k=1}^K(1-a_{i,j}^k\left(t\right))T_i^k{\left(t\right)}^{loc}$$

(13)

and

$$T_i^{mec}\left(t\right)=\mathrm{Max}\{j\in J|{\Delta }_t^j+\sum _{i=1}^I\sum _{k=1}^K{a}_{i,j}^k\left(t\right)T_{i,j}^k{\left(t\right)}^C)\}$$

(14)

$T_i^{loc}\left(t\right)$ represents the sum of the latency of all local computing tasks of $U{D}_i$. $T_i^{mec}\left(t\right)$ represents the edge task completion delay of $U{D}_i$, that is, the time from the start of the time slot to the completion of the last task of $U{D}_i$ calculated by each UAV (makespan). ${\displaystyle \sum _{k=1}^K}(1-a_{i,j}^k\left(t\right))T_i^k{\left(t\right)}^{loc}$ is the total local processing time of $U{D}_i$. The maximum value of $T_i^{mec}\left(t\right)$ and ${\displaystyle \sum _{k=1}^K}(1-a_{i,j}^k\left(t\right))T_i^k{\left(t\right)}^{loc}$ is the task response delay of time slot t. Since the task response delay cannot exceed the time slot length, the smaller one of this value and the time slot length is taken.

$E_i\left(t\right)$ can be presented as (15)

$$E_i\left(t\right)=\sum _{k=1}^K(1-a_{i,j}^k\left(t\right))E_i^k{\left(t\right)}^{loc}+\sum _{k=1}^K{a}_{i,j}^k\left(t\right)E_{i,j}^k{\left(t\right)}^P$$

(15)

Then, the cost of $U{D}_i$ at time slot t can be given as (16) [30,35]

$$C_i\left(t\right)={\gamma }_i{T}_i\left(t\right)+(1-{\gamma }_i)E_i\left(t\right)$$

(16)

The weighted parameters ${\gamma }_i$ of $T_i\left(t\right)$ and $(1-{\gamma }_i)$ of $E_i\left(t\right)$ can be flexibly set according to the preference of the UD for latency and energy consumption.

UAV Energy Cost. The cost $E_j\left(t\right)$ of the $UA{V}_j$ in time slot t includes the computing energy consumption and movement energy consumption (17):

$$E_j\left(t\right)=E_j^C\left(t\right)+E_j^P\left(t\right)$$

(17)

where the computing energy consumption can be represented as (18)

$$E_j^C\left(t\right)=\sum _{i=1}^I\sum _{k=1}^K{a}_{i,j}^k\left(t\right)E_{i,j}^k{\left(t\right)}^C$$

(18)

and the movement energy consumption can be represented as (19) [36]

$$E_j^P\left(t\right)={\displaystyle \frac{W_j}{2}}{\left(v_j\left(t\right)\right)}^2{T}_P$$

(19)

where $W_j$ is the weight of the $UA{V}_j$, $v_j\left(t\right)$ is the flight speed of the $UA{V}_j$, and $T_P$ is the travel time. Due to the randomness and uncertainty of the arrival of UD tasks, we consider that each UAV travels along a certain regular route, so this part of the energy consumption is fixed in [21,22].

3.5. Data Queue Model

$A_i\left(t\right)$ represents the task data that arrive at $U{D}_i$ at time t. In the actual process, they are random. Let $Q_i\left(t\right)$ denote the queue length of the $i_{th}$ UD at the beginning of the time frame. Then, the queue dynamics can be modeled as

$$Q_i(t+1)=max\left\{Q_i\left(t\right)-{\overline{D}}_i\left(t\right)+A_i\left(t\right),0\right\},\forall i\in \mathcal{I}$$

(20)

where ${\overline{D}}_i^{=}min(Q_i\left(t\right),D_i\left(t\right))$, $D_i\left(t\right)$ represents the amount of task data processed at time t, and $Q_i\left(1\right)=0$. In this paper, we consider the infinite queuing capacity for analytical tractability. We force $Q_i\left(t\right)\ge 0$ to always hold. Then, the queue dynamics are simplified to

$$Q_i(t+1)=Q_i\left(t\right)-D_i\left(t\right)+A_i\left(t\right),\forall i\in \mathcal{I}$$

(21)

Definition 1.

A discrete time queue $Q_i\left(t\right)$ is strongly stable if the time average queue length ${lim}_{K\to \infty }{\displaystyle \frac{1}{T}}{\sum }_{t=1}^T\mathbb{E}\left[Q_i\left(t\right)\right]\le \infty$, where the expectation is taken with respect to the system random events [30], i.e., task data arrivals in this paper.

By the Little’s law, the average delay is proportional to the average queue length. Therefore, a strongly stable data queue translates into a bounded processing delay for each task data bit.

3.6. Problem Formulation

Based on the above model, we minimize the average cost of the entire MEC system over time by jointly optimizing unloading decisions and resource allocation. The joint optimization problem can be expressed as (22) [30]:

$$P:\underset{\mathcal{A},\mathcal{F},\mathcal{W}}{min}{\displaystyle \frac{1}{T}}\sum _{t=1}^T\sum _{i=1}^I{C}_i\left(t\right)$$

(22)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.\underset{\mathrm{T}\to +\infty }{\lim }{\displaystyle \frac{1}{T}}\sum _{t=1}^T\mathrm{E}\left\{E_j\left(t\right)\right\}\le \overline{E_j}\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill & a_{i,j}^k\left(t\right)\in \{0,1\},\forall i\in \mathcal{I},k\in \mathcal{K},j\in \mathcal{J},t\in \mathcal{T}\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & 0\le r_j^{i,k}\left(t\right)\le r_j,\forall j\in \mathcal{J},t\in \mathcal{T}\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill & \sum _{i=1}^I\sum _{k=1}^K{a}_{i,j}^k\left(t\right)r_j^{i,k}\left(t\right)\le r_j,\forall i\in \mathcal{I},k\in \mathcal{K},j\in \mathcal{J},t\in \mathcal{T}\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill & \underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}·\sum _{t=1}^T\mathbb{E}\left[Q_i\left(t\right)\right]\le \infty ,\forall t\in \mathcal{T}\hfill \end{array}$$

(27)

where $\overline{E_j}$ is the energy upper limit of a single time slot, and $F_{\max }$ is the total amount of computing resources. Constraint (23) is to ensure that the energy consumption of each UAV in each time slot does not exceed the upper limit. The goal of (24) is to ensure that all computing tasks arriving at the UD can be processed. (25) and (26) are communication resource constraints. (27) is the data queue stability constraint.

The solution to problem P depends on long-term information, such as the arrival of tasks at different times. More seriously, problem P contains two decision variables, task offloading and resource allocation, and is a Mixed Integer Nonlinear Programming problem (MINLP). Even if the future information is known, it is challenging to solve the problem directly.

4. Lyapunov-Based Decoupling of the Multi-Slot MINLP

In this section, we apply Lyapunov optimization to decouple the problem P into a per-frame deterministic problem. Since the problem P is related to future information, we use Lyapunov optimization to decompose the conditions with long-term constraints (i.e., the long-term energy constraints of the UAV (23)) into each time slot.

First, in order to satisfy constraint (23), a virtual energy queue is introduced: UAV computing energy queue $Y_j^c\left(t\right)$. Assume that the queue is set to zero at the initial time slot, $Y_j^c\left(t\right)=0$. And it is updated as (28)

$$Y_j^c(t+1)=max\left\{Y_j^c\left(t\right)+E_j^c\left(t\right)-{\overline{E}}_j^c,0\right\},\forall t\in \mathcal{T}$$

(28)

where ${\overline{E}}_j^c$ represents the computational energy budget of $UA{V}_j$ in each time slot. The backlog state of the energy queue is represented by the vector ${\mathrm{\Theta }}_j\left(t\right)=\left\{Y_j^c\left(t\right)\right\}$, and the scalar measure of the backlog of the energy queue can be represented by the function $L\left({\mathrm{\Theta }}_j\left(t\right)\right)$ (29):

$$L\left({\mathrm{\Theta }}_j\left(t\right)\right)={\left(Y_j^c\left(t\right)\right)}^2$$

(29)

Secondly, we define the Lyapunov drift, which is used to calculate the increment $\delta {\mathrm{\Theta }}_j\left(t\right)$ of the energy queue backlog from time slot t to time slot $t+1$ (30):

$$\delta {\theta }_j\left(t\right)=L\left({\mathrm{\Theta }}_j(t+1)\right)-L\left({\mathrm{\Theta }}_j\left(t\right)\right)$$

(30)

The drift penalty expression for problem P can be given as (31)

$$\begin{array}{cc}\hfill P_1:& \underset{\mathcal{A},\mathcal{F},\mathcal{W}}{min}\delta {\theta }_j\left(t\right)+V\sum _{i=1}^I{C}_i\left(t\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left(24\right), \left(25\right), \left(26\right), \left(27\right)\hfill \end{array}$$

(31)

where V is the parameter that weighs the total cost against the stability of the queue. Then, the scaling transformation of problem P is as follows:

$$\begin{array}{cc}\hfill \delta {\theta }_j\left(t\right)+VP\left(t\right)& ={\displaystyle \frac{1}{2}}\sum _{j=1}^J({\left(Y_j(t+1)\right)}^2-{\left(Y_j\left(t\right)\right)}^2)+VP\left(t\right)\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill & \le {\displaystyle \frac{1}{2}}\sum _{j=1}^J{\left(y_j\left(t\right)\right)}^2+\sum _{j=1}^J{Y}_j\left(t\right)y_j\left(t\right)+VP\left(t\right)\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill & \le B_{max}+\sum _{j=1}^J{Y}_j\left(t\right)y_j\left(t\right)+VP\left(t\right)\hfill \end{array}$$

(34)

where $Y_j\left(t\right)$ represents the energy queue, $y_j\left(t\right)$ represents the optimization goal, $P\left(t\right)$ represents the constraint condition, and $B_{max}$ represents the positive constant of the constraint condition.

Finally, the drift penalty expression for problem P after scaling is shown as (35)

$$P_2:\underset{{\mathcal{A}}^t,{\mathcal{F}}^t,{\mathcal{W}}^t}{min}\sum _{j=1}^J{Y}_j^c\left(t\right)E_j^c\left(t\right)+V\sum _{i=1}^I{C}_i\left(t\right)$$

(35)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.a_{i,j}^k\left(t\right)\in \{0,1\},\forall i\in \mathcal{I},k\in \mathcal{K},j\in \mathcal{J},t\in \mathcal{T}\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill & 0\le r_j^{i,k}\left(t\right)\le r_j,\forall j\in \mathcal{J},t\in \mathcal{T}\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill & \sum _{i=1}^I\sum _{k=1}^K{a}_{i,j}^k\left(t\right)r_j^{i,k}\left(t\right)\le r_j,\forall i\in \mathcal{I},k\in \mathcal{K},j\in \mathcal{J},t\in \mathcal{T}\hfill \end{array}$$

(38)

and we remove the irrelevant constant term from $P_2$ and obtain the problem $P_2^{\prime }$ (39):

$$\begin{array}{cc}\hfill P_2^{\prime }:& \underset{{\mathcal{A}}^t,{\mathcal{F}}^t,{\mathcal{W}}^t}{min}\sum _{j=1}^J{\displaystyle \frac{Y_j^c\left(t\right)E_j^c\left(t\right)}{V}}+\sum _{i=1}^I{C}_i\left(t\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left(36\right), \left(37\right), \left(38\right)\hfill \end{array}$$

(39)

Since the Lyapunov drift relies on information in the $t+1$ time slot, the problem P is scaled to become a real-time optimization problem that only needs information in a certain time slot. Due to the limited space, please refer to this article [37] for the related reasoning and proof.

Nevertheless, the new problem $P_2^{\prime }$ is still an MINLP problem. The two decision variables, task offloading strategy and resource allocation strategy, still affect each other. In order to solve the optimal solution of the $P_2^{\prime }$ problem, we transform the complex $P_2^{\prime }$ problem into two sub-problems: $P_3$ and $P_4$. Then, we design a joint optimization method to solve it.

5. Joint Optimization Algorithm

5.1. Optimal Resource Allocation Algorithm

Although the problem $P_2^{\prime }$ is still a multi-slot MINLP optimization problem, when considering a certain slot, we are surprised to find a subtle relationship between the two decision variables. That is, given the slot task offloading strategy, there is and will be a unique set of resource allocation strategies corresponding to it so that the objective function value of the slot reaches the optimal value. The derivation of this subtle relationship is as follows:

When the task offload strategy ${\mathcal{A}}^{t_{\varphi }}$ is given, the UAV resource allocation strategy is adjusted to optimize the objective function $P_3$ (40):

$$\begin{array}{cc}\hfill P_3:& \underset{{\mathcal{F}}^{t_{\varphi }},{\mathcal{W}}^{t_{\varphi }}}{min}\sum _{i=1}^I{\gamma }_i{T}_i\left(t\right)+\sum _{i=1}^I\sum _{k=1}^K(1-{\gamma }_i)a_{i,j}^k\left(t\right)E_{i,j}^k{\left(t\right)}^P\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left(36\right), \left(37\right), \left(38\right)\hfill \end{array}$$

(40)

Substituting Formulas (8)–(10) into problem $P_3$, we can obtain problem $P_3^{\prime }$ after removing the fixed constant term and simplifying the coefficients (41)

$$P_3^{\prime }:min{\displaystyle \frac{k_1}{n_1}}+{\displaystyle \frac{k_2}{n_2}}+{\displaystyle \frac{k_3}{n_3}}+\dots +{\displaystyle \frac{k_{\zeta }}{n_{\zeta }}}+\dots$$

(41)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.n_1, n_2, n_3,\dots , n_{\zeta },\dots , n_{last}\ge 0\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill & n_1+n_2+n_3+\dots +n_{\zeta }+\dots +n_{last}\le N,\hfill \end{array}$$

(43)

where $n_{\zeta }$ represents the computing resources allocated to the task numbered $\zeta$ of all tasks unloaded to $UA{V}_j$, $k_{\zeta }$ (known quantity) is the simplified coefficient in the objective function, and N represents the total computing resources of $UA{V}_j$.

Taking computing resources as an example, problem $P_3$ becomes problem $P_3^{\prime }$, which can be processed using the Lagrange multiplier method. It is proved that problem $P_3^{\prime }$ has only one solution (Appendix A) [38]. Based on this, we firstly design Algorithm 1 to solve the optimal resource and bandwidth allocation strategy when a given task offloading strategy is given.   

Algorithm 1: Algorithm for optimal resource allocation of ($P_3^{\prime }$).

Now that we have the optimal resource and bandwidth allocation strategies ${\mathcal{F}}^t$ and ${\mathcal{W}}^t$, solving the problem $P_2^{\prime }$ is equivalent to finding the optimal task offloading strategy ${\mathcal{A}}^t$. To do this, we introduce a new subproblem $P_4$ of $P_2^{\prime }$ (44):

$$\begin{array}{cc}\hfill P_4:& \left({\mathcal{A}}_t^{\ast }\right)=\underset{{\mathcal{F}}_t^{\ast },{\mathcal{W}}_t^{\ast }}{arg\; min}\sum _{j=1}^J{\displaystyle \frac{Y_j^c\left(t\right)E_j^c\left(t\right)}{V}}+\sum _{i=1}^I{C}_i\left(t\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left(36\right)\hfill \end{array}$$

(44)

Considering the different ways of offloading tasks of $U{D}_i$, we can introduce the optimization cost function $O_i\left({\mathcal{A}}_t\right)$ (45) of the $U{D}_i$ task offloading strategy to simplify the problem $P_4$ into $P_4^{\prime }$:

$$O_i\left({\mathcal{A}}_t\right)=\left\{\begin{array}{ll}\begin{array}{c}\sum _{j=1}^J{\displaystyle \frac{Y_j^c\left(t\right)E_{i,j}^k{\left(t\right)}^C}{V}}+\hfill \\ {\gamma }_i{T}_i^{mec}\left(t\right)+(1-{\gamma }_i)E_i^{mec}\left(t\right)\hfill \end{array}& a_{i,j}^k\left(t\right)=1\\ {\gamma }_i{T}_i^{loc}\left(t\right)+(1-{\gamma }_i)E_i^{loc}\left(t\right)& a_{i,j}^k\left(t\right)=0\end{array}\right.$$

(45)

Then, we can obtain $P_4^{\prime }$ (46):

$$\begin{array}{cc}\hfill P_4^{\prime }:& \left({\mathcal{A}}_t^{\ast }\right)=\underset{{\mathcal{F}}_t^{\ast },{\mathcal{W}}_t^{\ast }}{arg\; min}{O}_i\left({\mathcal{A}}_t\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left(36\right)\hfill \end{array}$$

(46)

To tackle problem $P_4^{\prime }$, we propose a Lyapunov-guided reinforcement learning-based rapid and durable optimization algorithm (LyraRD) framework. Traditional branch and bound and block coordinate descent methods require enumerating $2^N$ unloading strategies, which brings great complexity even when N is a moderate value (for example, N = 10). Using RL techniques, we propose a LyraRD algorithm to construct a policy $\pi$ that maps from input $I\left(t\right)$ to optimal actions $x^{\ast }\left(t\right)$ with very low complexity, for example, requiring tens of milliseconds of computation time (that is, the duration from observing a state to producing a control action $x\left(t\right)$) when N = 10.

5.2. LyraRD Algorithm Description

As shown in Figure 3, LyraRD consists of four main modules: the actor module accepts input $I\left(t\right)$ and produces a set of candidate offloading actions $x\left(t\right)$, the critic module evaluates $x\left(t\right)$ and selects the best offloading action $x^{\ast }\left(t\right)$, the policy update module improves the actor module’s policy over time, and the queue module updates the state of the system queue $Y\left(t\right)$ after performing an offloading action. Through repeated interactions with the stochastic environment $A\left(t\right)$, the four modules operate sequentially and iteratively as described below.

(1) Actor module: The actor module consists of a DNN and an action quantizer. The observation value $I\left(t\right)$ is taken as input, where it includes the amount of task data $A\left(t\right)$ generated by each UD at the current time slot and the virtual energy queue value $Y\left(t\right)$ for each UAV that performs long-term energy constraints. The network architecture of DNN consists of an input layer, a convolutional layer, and a fully connected layer. The features are extracted by convolving the input $I\left(t\right)$ with three layers through the ReLU activation function. The output of the convolutional layer is flattened for input to the fully connected layer. The fully connected layer, containing 64 neurons with the ReLU activation function, is used to integrate the global features. The DNN outputs a relaxed unloading decision $x\left(t\right)\in {[0,1]}^N$. Then, a sigmoid activation function is used in the output layer to quantize the continuous $x\left(t\right)$ into feasible candidate binary offloading actions $M_t$, where $M_t$ is a time-dependent design parameter. The quantization function is expressed as

$$\begin{array}{c}\hfill x\left(t\right)\mapsto \mathcal{g}=\left\{x_i\left(t\right)|x_i\left(t\right)\in {\{0,1\}}^N,i=1,\dots ,M_t\right\}\end{array}$$

(47)

where $\mathcal{g}$ denotes the set of candidate offloading actions in the $t_{th}$ time frames.

(2) Critic module: Followed by the actor module, the critic module evaluates $x_i\left(t\right)$ and selects the best unloading action $x^{\ast }\left(t\right)$ according to formula (46):

$$\begin{array}{c}\hfill LS\left({\theta }_t\right)=-{\displaystyle \frac{1}{|S_t|}}\sum _{T\in S_t}\left[x^T\left(t\right)log{\Pi }_{{\theta }_t}\left(I\left(t\right)\right)\right.\\ \hfill \left.+{(1-x\left(t\right))}^{T}log(1-{\Pi }_{{\theta }_t}\left(I\left(t\right)\right))\right]\end{array}$$

(48)

(3) Policy Update Module: LyraRD uses $\left(I\right(t),x(t\left)\right)$ as the labeled input–output samples to update the DNN policy. In particular, we maintain a replay memory that only stores the most recent data samples q. In practice, with the memory initially empty, we start training the DNN after collecting more than $q/2$ data samples. The DNN then is periodically trained every ${\delta }_t$ time interval to avoid overfitting the model. We then use the Adam algorithm [39] to minimize the average cross entropy loss function $LS\left({\theta }_t\right)$ (48) on the data samples to update the parameters of the DNN.

(4) Queuing module: As a by-product of the critic module, we obtain the optimal allocation of resources associated with $x^{\ast }\left(t\right)$. Then, we update the energy queue based on them and the newly arrived task data and iterate again from the actor module as the next round of system input.

The entire process of LyraRD algorithm is shown in Algorithm 2. The algorithm has been shown to have lower complexity and better convergence (LyraRD algorithm has been proven to have extremely low complexity and excellent convergence. For details, refer to [40]).

Algorithm 2: The online LyraRD algorithm for solving (P4’).

6. Simulation Results

In this section, we perform simulations to verify the effectiveness of the proposed method.

6.1. Platforms and Tasks Data

We conduct simulation experiments on a computer with the following specifications: CPU: Intel i7-13650HX 3.2 GHz; GPU: NVIDIA RTX 4060; RAM: 64 GB; OS: 64-bit Ubuntu 16.04. All algorithms are developed using Python 3.7.

We consider a MEC system enabled by UAVs consisting of 4 UAVs and 20 UDs, where the initial horizontal positions of the UAVs are set to P1(0) = [10, 10], P2(0) = [10, 80], P3(0) = [90, 20], P4(0) = [75, 85], respectively, with a fixed height of $H=100$ m. The UDs appear randomly in an area of 100 × 100 ${\mathrm{m}}^2$, and UAVs fly along fixed routes. The system timeline is discretized into 8000 time slots, each of which is 1 s long [41]. The remaining parameter settings are shown in the table in Appendix B.

In order to simulate the random arrival of data, we assume that each user reaches no more than three tasks at the same time, and the arrival forms of these tasks obey exponential, Poisson, and uniform distributions, respectively. That is, the task queue $Q1$∼$E\left(\lambda \right)$, $Q2$∼$\pi \left({\lambda }^{\prime }\right)$, $Q3$∼$U(a,b)$. To compare the impact of different types of data arrival on the stability of the data queue, we make the expectation the same for all three distributions, i.e., $\lambda ={\lambda }^{\prime }=(a+b)/2$, where $\lambda ={\lambda }^{\prime }=3,a=0,b=6$. Next, we obtain the simulation results shown in Figure 4, Figure 5 and Figure 6.

6.2. Overall Performance Analysis

Considering the randomness of data arrival, we fully conduct several experiments and select the four most representative experimental results to verify the performance of the proposed method. Figure 4 shows the training loss of the algorithm, which fully illustrates the convergence of the LyraRD algorithm.

Further, we observe the use of energy as shown in Figure 5, where the energy cohort fluctuates over the initial time period and gradually stabilizes. In Figure 5b–d, the energy queue still has a second peak after the initial fluctuation, but it gradually plateaus after that.

Based on the fluctuations in the results, we further investigate the arrival and processing of the data. First, observe the data queue, as shown in Figure 6; the data queue basically reaches a stable state after training. However, the data cohorts in Figure 6b–d all exhibit secondary fluctuations to varying degrees. In response to these fluctuations, we recover the specific values of the data in the queue during that period and analyze them.

Single UD data accumulation. In Figure 6b, the energy queue of the second peak is given by Figure 7. We find that the data queue of the 20th user has an abnormal accumulation that is significantly higher than that of other users. This abnormal accumulation may be due to the excessive amount of random data arriving from this user in the previous moment, resulting in untimely data calculation. In the short period of time after that, our algorithm processes these backlogged data, thus ensuring that the data queue returns to stability in the subsequent moments. This also confirms the energy consumption in Figure 5b.

Task-oriented offloading. The second peak value in Figure 6c is lower, so we speculate that there is no backlog in the data queue, but it is caused by more data being offloaded to the UAV for calculation. For this reason, we also observe the user data queue at this time. We can see in Figure 8 that the amount of data that need to be processed in the user’s data queue is very small, indicating that most of the data are offloaded to the UAV for calculation. The peak of the test energy consumption comes from the large amount of calculation of the UAV.

Large-scale distributed tasks. Then, we retrieve the abnormal data queue value in Figure 6d, as shown in Figure 9. It can be found that the pending value of each user’s subtask queue data is between 20 and 45. However, the sum of the pending data of Q1, Q2, and Q3 is 560.52, 556, and 576.38 respectively, which exceeds the data processing limit of the user and UAV in a period of time, thus causing data accumulation. As the task is continuously unloaded, the data queue returns to stability, and the energy consumption also tends to be stable.

The above results fully demonstrate that our method maintains stable data processing capabilities when dealing with randomly arriving data, while ensuring uniform and stable energy usage throughout the entire process, and can effectively cope with the sudden arrival of large-scale data.

6.3. Baseline Methods

Moreover, we compare LyraRD with the following benchmark schemes:

• Average latency of tasks ($ALT$): The optimization goal is the linear sum of the average task completion delay.
• Equal resource allocation ($ERA$): The UAV allocates computing and communication resources equally.

6.4. Evaluation Result

Figure 10 shows the case of cost in the three scenarios. The scheme proposed in this paper is $OBJ$. First, when the value of V is not large (not more than 40), the cost case of $OBJ$ is not significantly lower than the other two comparative cases as expected, which is caused by the value of V being small and the cost being too focused on the stability of the energy-consuming queue. When the value of V is greater than or equal to 50, it can be seen that the cost of $OBJ$ is significantly lower than the cost case of $ALT$, and $OBJ$ shows good performance in combining the queue stability and the time overhead of the MEC system with the energy overhead of the UAV. Counterintuitively, $ERA$ has a lower cost than $OBJ$ and $ALT$ in some cases compared to $OBJ$ and $ALT$. This is due to the fact that computational task arrivals are randomized, and when generating offloading actions based on these task data, $ERA$ outperforms $OBJ$ and $ALT$ in terms of UAV resource allocation in the following scenario, i.e., a chaotic scenario where very few tasks are offloaded to the UAV. In spite of this, the $ERA$ scheme does not demonstrate a clear superiority at cost, especially in resource-constrained scenarios with large-scale task arrivals. Finally, it can be seen that the proposed $OBJ$ scheme is able to adapt to different task data sizes and has relatively superior performance in terms of cost for MEC systems.

7. Conclusions

In this paper, we study a UAV-enabled MEC system that satisfies long-term energy consumption constraints with high response efficiency. The joint optimization of the offloading decision and resource allocation strategy is investigated to minimize the system cost. To solve this problem, we propose a joint optimization algorithm. Specifically, the multi-stage MINLP problem is transformed into a MINLP problem within a single time period using Lyapunov optimization, which is later solved using a reinforcement learning approach. Simulation results show that the method significantly reduces the system cost compared to the benchmark solution. However, our work has some limitations that need to be addressed in future work. First, the current model does not explicitly consider the motion patterns of the models, which may limit the applicability of the results in highly dynamic scenarios. Second, adaptive collaboration mechanisms for multi-UAV systems have been understudied. In the future, we will integrate realistic UD motion models and explore the interactions between dynamic UD behaviors and UAV trajectory optimization, as well as further investigating the adaptive collaboration of UAVs to enhance the utility of the framework. These extensions aim to provide more comprehensive solutions for complex UAV-based MEC networks in the real world.