A Hybrid Decision-Making Framework for UAV-Assisted MEC Systems: Integrating a Dynamic Adaptive Genetic Optimization Algorithm and Soft Actor–Critic Algorithm with Hierarchical Action Decomposition and Uncertainty-Quantified Critic Ensemble

Abstract

With the continuous progress of UAV technology and the rapid development of mobile edge computing (MEC), the UAV-assisted MEC system has shown great application potential in special fields such as disaster rescue and emergency response. However, traditional deep reinforcement learning (DRL) decision-making methods suffer from limitations such as difficulty in balancing multiple objectives and training convergence when making mixed action space decisions for UAV path planning and task offloading. This article innovatively proposes a hybrid decision framework based on the improved Dynamic Adaptive Genetic Optimization Algorithm (DAGOA) and soft actor–critic with hierarchical action decomposition, an uncertainty-quantified critic ensemble, and adaptive entropy temperature, where DAGOA performs an effective search and optimization in discrete action space, while SAC can perform fine control and adjustment in continuous action space. By combining the above algorithms, the joint optimization of drone path planning and task offloading can be achieved, improving the overall performance of the system. The experimental results show that the framework offers significant advantages in improving system performance, reducing energy consumption, and enhancing task completion efficiency. When the system adopts a hybrid decision framework, the reward score increases by a maximum of 153.53% compared to pure deep reinforcement learning algorithms for decision-making. Moreover, it can achieve an average improvement of 61.09% on the basis of various reinforcement learning algorithms such as proposed SAC, proximal policy optimization (PPO), deep deterministic policy gradient (DDPG), and twin delayed deep deterministic policy gradient (TD3).

1. Introduction

1.1. Research Background

With its rapid development, UAV technology has played an irreplaceable role in considerable fields, especially in mobile edge computing (MEC) systems. Traditional ground MEC servers suffer from problems such as high deployment costs, poor adaptability to network dynamics, and fixed service scope. In recent years, the paradigm shift from ground-edge computing to air–ground joint-edge computing, UAV-assisted MEC, has provided a new method for addressing the above limitations. The UAV-assisted MEC system aims to provide low-latency, high-bandwidth computing services by deploying computing resources close to the user or edge platform, thereby meeting the growing demand for data processing. This technology serves as an essential foundation for the Internet of Things, end-to-end large model inference, and collaboration among multiple unmanned platforms.

In recent years, the research focus in the field of UAV-assisted MEC systems has mainly been on deploying, integrating, and optimizing the communication, perception, and computing capabilities of various nodes in the network through drone platforms to form intelligent edge networks, aiming to achieve an agile and ubiquitous intelligence of things. The high maneuverability of drones enables them to be rapidly and flexibly deployed as airborne MEC servers to assist ground MEC servers in providing temporary computing services anytime, anywhere. In addition, the high-probability line of sight (LOS) link of drones could also improve the communication reliability and network capacity of ground MEC networks [1]. As a mobile edge server, the UAV not only improves data transmission efficiency but also reduces data transmission latency, providing efficient and convenient services for terminal platforms and users.

The UAV-assisted MEC systems in Figure 1 have demonstrated great potential in several application scenarios and can play a significant role in post-disaster rescue and emergency response to accidents due to their combination of the high maneuverability of drones and the proximity computation capability of MEC. In the process of post-disaster rescue, the UAV serves as an airborne mobile edge computing server, providing computing resources close to the ground users. The ground users generate tasks that require computation, and these tasks can be offloaded to the UAV for processing. The UAV communicates with the ground users through data links, receiving tasks and transmitting processed results back to users. The UAV’s high mobility allows it to quickly move to areas where computing resources are needed, enhancing the system’s responsiveness and efficiency.

1.2. Research Gap

In practical applications, UAV-assisted MEC systems still face many challenges, such as the contradiction between supply and demand in resource allocation, the heterogeneity of supply and demand in computational offloading, energy saving and real-time trajectory control, and different time-scale dynamics, which put forward higher requirements for the effective allocation of computational resources, UAV trajectory control, and the design of task offloading strategies [2]. First, the endurance and energy consumption problems of UAVs need to be solved. Since UAVs are limited by their battery capacity, how to extend endurance while ensuring the efficiency of mission execution is a critical issue still to be solved. Secondly, path planning and mission offloading strategies for UAVs in complex environments are also a major difficulty. In addition, with the increase in the number of user devices, the MEC system’s demand for the effective management and scheduling of MEC resources to meet different application scenarios severely limits the stability of the system in complex environments and real-time data processing.

In order to cope with the above challenges, research on optimization for UAV-assisted MEC systems, especially in optimizing decision-making algorithms for UAVs, is particularly crucial. By introducing advanced decision-making algorithms, such as reinforcement learning and deep learning, the decision-making efficiency of UAVs in path planning and task offloading can be improved, thus further enhancing the overall performance of the MEC system. However, the traditional pure deep reinforcement learning (DRL) decision-making methods have certain limitations in practical applications [3]. For example, DRL algorithms often require a large amount of training data and computational resources when dealing with complex dynamic environments, resulting in a long training time and difficulties in convergence. Secondly, DRL usually faces difficulty in balancing the weights between different objectives when facing multi-objective optimization problems, resulting in ineffective decision-making. In addition, the DRL algorithm also suffers from certain accuracy degradation when dealing with mixed problems in discrete and continuous action spaces [4].

To overcome such limitations, this paper proposes an innovative hybrid decision-making framework that combines the Dynamic Adaptive Genetic Optimization Algorithm (DAGOA) with deep reinforcement learning for UAV-assisted MEC systems. The innovation of the framework is to make task offloading decisions via the improved genetic algorithm, which is able to locate the near-optimal solution in a complex search space by utilizing its global search capability and good convergence, whereas DRL focuses on path planning decisions, which is able to achieve efficient decision-making in dynamic environments by learning the optimal policy through interaction with the environment. The potential advantage of the hybrid decision-making framework is that it can fully utilize the respective advantages of DAGOA and DRL to compensate for the deficiencies of a single algorithm. Specifically, DAGOA is able to perform effective search and optimization in discrete action space, while DRL is able to perform fine control and adjustment in continuous action space. By combining the two, the joint optimization of UAV path planning and mission offloading can be achieved to improve the overall performance of the system.

In the next part of this paper, a hybrid decision-making framework based on DAGOA and the soft actor–critic (SAC) with multiple novel mechanisms is proposed as a primary solution for UAV control in the UAV-assisted MEC system, and simulation experiments are conducted to verify that DAGOA, when coupled with a variety of DRL algorithms, is able to help UAVs achieve efficient path planning and task offloading decisions in complex dynamic environments in UAV-assisted MEC systems. The proposed decision framework achieves the co-optimization of discrete task offloading decisions (via DAGOA) and continuous path planning (via enhanced SAC), overcoming the limitations of single-algorithm approaches in handling mixed action spaces.

The experimental results show that the framework achieves significant advantages in enhancing system performance, reducing energy consumption, and improving task completion efficiency. The system adopts a hybrid decision-making framework with a maximum increase of 153.53% in reward scores compared to pure deep reinforcement learning algorithms for decision-making. The framework achieves an average of 61.09% improvement in rewards on the basis of multiple reinforcement learning algorithms such as SAC, Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), and Twin Delayed Deep Deterministic (TD3) [5]. Meanwhile, the research in this paper provides new ideas and methods for the optimization of UAV-assisted MEC systems, which has high academic value and application prospects and is of great theoretical, as well as practical, significance for promoting the development and application of UAV-assisted MEC systems.

2. Related Works

2.1. State of the Art of UAV Path-Planning Algorithms

An algorithm based on the Kalman filter and the improved rapid exploration random tree (KF-RRT) was proposed by Yan and Fu [6] for the dynamic path-planning problem of UAVs. The algorithm improves the efficiency and adaptability of UAV path planning by introducing weighting coefficients for target region trends and predicting trajectories of dynamic obstacles using the Kalman filter. The feasibility of the path is further optimized via the B-spline curve smoothing technique. Compared with existing RRT algorithms, the KF-RRT algorithm shows superiority in simulation experiments when planning UAV paths in dynamic environments. However, the limitation of this study is that the algorithm may encounter high computational complexity and a lack of real-time performance in highly dynamic and complex environments.

Wang et al. [7] proposed an unmanned aerial vehicle (UAV) path-planning method based on modified particle swarm optimization (MPSO). By establishing a path-planning model, considering UAV performance limitations and spatial obstacle threats, the MPSO algo-rithm with dynamically adjusted inertia weights is used to plan the navigation path of the UAV, which effectively improves the efficiency and reliability of path planning. The simulation results show that the algorithm can effectively solve the path-planning problem of UAVs. However, the limitation of this study is that the algorithm may encounter high computational complexity and a lack of real-time performance in highly dynamic and complex environments.

An ant colony optimization (ACO)-based path-planning algorithm for unmanned aerial vehicles (UAVs) in dynamic environments was proposed by Baroomi et al. [8]. The algorithm determines the optimal path of the UAV using colony intelligence and ant behavior and dynamically adapts to environmental changes through pheromone trajectories and heuristic functions. Simulation results show that the algorithm is effective and efficient in dynamic environments. However, the research limitation is that the energy constraints of UAVs and the dynamic changes in real complex environments are not considered. Although Zhang et al. [9] optimized the efficiency and success rate of UAV path planning by improving the goal-biased RRT algorithm, the adaptability in highly dynamic and complex constraint environments still needs further research.

Zhou et al. [10] proposed a multi-UAV trajectory planning algorithm based on the multi-agent deep deterministic policy gradient (MADDPG) algorithm, which is named potential field dense MADDPG (PF-MADDPG). This algorithm employs a potential field-dense reward function to effectively improve the learning efficiency and path-planning performance in complex environments. Nevertheless, the study neglects the communication synergy between UAVs, and the scalability and robustness in terms of high dynamics have not been fully verified.

2.2. UAV-Assisted Mobile Edge Computing

Li et al. [11] proposed a cooperative computation offloading strategy in UAV-assisted mobile edge computing, which optimizes the initial number and 3D location by deploying multiple UAVs on demand. The rational planning of UAV trajectories during computation offloading is investigated to ensure the communication quality of mobile users and reduce energy consumption in order to accomplish the task with a limited battery capacity and computational power. The mixed-integer nonlinear planning problem is solved by the block coordinate descent method, and the results show that the algorithm can effectively reduce the path loss and total energy consumption. However, the study is limited to an idealized UAV energy and computational model, and real-world applications may face more complex environmental and equipment constraints.

Xiang et al. [12] proposed a UAV-assisted mobile edge computing system that aims to minimize the energy consumption and latency-weighted sum of the system by jointly optimizing the flight trajectory of the UAV and the user’s task offloading strategy. The study employs a differential evolutionary algorithm and an optimistic actor–critic algorithm and verifies the advantages of the algorithms in terms of energy consumption and convergence performance through simulations. However, the limitation of the study is that it does not take into account the energy constraints of UAVs or dynamics in real complex environments.

Gao et al. [13] proposed a secure data transmission scheme for UAV relay-assisted maritime mobile edge computing (MEC) systems. Interference signals are sent via ground jammers to reduce the risk of interception, and the amplify-and-forward (AF) protocol is utilized to jointly optimize the user’s transmission power, time-slot allocation factor, and UAV flight trajectory in order to maximize the user’s minimum secure computing power. The study employs block coordinate descent (BCD) and successive convex approximation (SCA) techniques to solve the optimization problem. Simulation results show that the proposed scheme is more effective in improving the safe computing power of the system compared to the four benchmark schemes. However, the limitations of this study are the assumption of static relative positions between UAVs and the neglect of the effects of variable weather conditions in marine environments on UAV communication and flight performance.

A deep reinforcement learning-based trajectory optimization and resource allocation scheme (DRTORA) is proposed by Gao et al. [14] to address the problem of improving the secure computing performance of UAV-assisted MEC networks. The scheme utilizes deep Q-learning (DQN) to optimize the flight trajectory, task offloading decision, and time allocation of UAVs to maximize the system’s secure computing capability. Simulation results show that DRTORA is effective in improving cybersecurity computing performance. However, the limitation of this study is that a simplified communication model is assumed, which does not adequately consider the dynamic changes and complexity in real-world environments.

Wang and Sun [15] proposed a UAV-assisted MEC system to reduce energy consumption by jointly optimizing the computation of offloading decisions and resource allocation, but they did not include consideration of signal coverage and energy dynamics. Zhang et al. [16] employed a PPO reinforcement learning algorithm to optimize the computational offloading strategy for UAV-assisted MEC, which improves the task processing efficiency but ignores the UAV energy consumption and potential communication changes. Zhou et al. [17] optimized a computational of-floading strategy for UAV-assisted MEC via the DDPG algorithm, which effectively reduces the latency but ignores the challenges of UAV energy consumption and the dynamic ocean environment.

Yin and Tian [18] explored how to reduce the energy consumption of user devices by optimizing user task offloading policies and UAV trajectories in UAV-assisted mobile edge computing (MEC) in the context of 5G. An iterative optimization algorithm based on the deep deterministic policy gradient (DDPG) (IOECA) was proposed to minimize energy consumption. The study considered the battery capacity limitations of UAVs and validated the effectiveness of the proposed algorithm through simulation experiments. However, the limitation of the study is that a fixed UAV altitude and an idealized communication environment are assumed, and the dynamic changes and interference factors in practical applications were not fully considered.

The study by Han et al. [19] proposed a joint user association and a UAV deployment optimization method in UAV-assisted MEC networks, which effectively reduces task latency through optimal transmission theory and particle swarm optimization algorithms. However, the study faced several limitations: first, the paper did not consider UAV energy management and fine-grained computing loads, which may affect the UAV’s sustained operation capability and communication quality; second, though the paper focused on the optimization of the task delay, it did not account for the algorithm execution efficiency, which leads to greater difficulty in practical application.

2.3. Application of Genetic Algorithms

The current genetic algorithms applied to UAV-related control and decision-making generally suffer from limitations in dynamic adaptation and performance. Yao et al. [20] presented an improved genetic algorithm and two related coding methods for the multi-UAV cooperative search problem. Despite their innovative coding strategies, the genetic algorithm still faces challenges in dealing with large-scale search spaces and dynamic environments, leading to a large scope for improving both the accuracy and speed of the algorithm. Wang et al. [21] proposed an improved genetic algorithm based on the beetle tentacle search algorithm for solving the multi-UAV cooperative multi-tasking problem. The algorithm improves the global and local search ability of the genetic algorithm by enhancing population diversity and dynamically adjusting the variance probability, but further research is still needed to overcome the precocity and parameter tuning difficulties. Chen and Qi [22] proposed a path-planning method for UAVs that incorporates a genetic algorithm and an improved ant colony algorithm, aiming to improve the efficiency and safety of path planning. The method utilizes a genetic algorithm to initialize the pheromone matrix of the ant colony algorithm to accelerate convergence and avoid local optima. However, the limitation of this fusion algorithm is that the iterative redundancy of the genetic algorithm in the later search may lead to a long planning time, while the improved ACO algorithm enhances the global search capability, but the response speed and path smoothing for unexpected threats in dynamic environments still need to be improved.

On the other hand, the genetic algorithm research field lacks research on joint applications with deep reinforcement learning in the field of UAV-assisted MEC systems, where UAV path planning and task execution cannot be balanced currently. Gao et al. [23] introduced a genetic algorithm combining oppositional and chaotic search for solving the path-planning problem of UAVs in a multi-obstacle environment. The algorithm improves the search capability and convergence speed of the genetic algorithm by improving the initial population generation and crossover strategies. Nevertheless, the genetic algorithm still faces the challenges of convergence speed and local optimization when dealing with complex constraints and high-dimensional problems, and future research should focus on improving the performance for dynamic path point processing and real-time path planning. Li et al. [24] investigated the multi-UAV air combat weapon-target assignment problem based on genetic algorithms and deep learning by constructing an optimization model in order to improve combat efficiency. Although genetic algorithms perform well in global search, their computational complexity and sensitivity to parameters when dealing with large-scale problems limit their application. An exploration for more efficient algorithms to meet the challenges of real-time dynamic battlefield environments is still needed.

The current improvements to genetic algorithms are still insufficient in terms of execution efficiency and dynamic environment adaptation. Su et al. [25] proposed an improved genetic algorithm that effectively improves the efficiency and effectiveness of multi-UAV task assignment by introducing the Metropolis criterion of simulated annealing and the variable step annealing cooling method. However, compared with the DAGOA adaptive genetic algorithm, the method faces limitations in terms of dynamic environment adaptation and early stopping mechanisms, and future research needs to be further optimized to improve the flexibility and efficiency of the algorithm. Li et al. [26] proposed a multi-UAV maritime target search path-planning algorithm based on GA, which achieves the target classification and path planning by combining K-means clustering and multi-chromosome GA path planning. However, compared with the DAGOA adaptive genetic algorithm, the algorithm faces limitations in dynamic adaptation and individual selection, resulting in limitations in the robustness and performance of the algorithm.

3. Framework Design

3.1. Design of DAGOA

Recent studies have demonstrated the complementary advantages of combining evolutionary algorithms with deep reinforcement learning. Liang et al. [27] proposed RL-RVEA that integrates reinforcement learning with reference vector adaptation, showing RL’s capability of dynamically adjusting search strategies while maintaining EA’s global exploration. In the PSO domain, Li et al. [28] developed NRLPSO using Q-learning to guide velocity vector generation, achieving a 61.09% performance improvement over standalone RL. For complex action spaces, Wang et al. [29] employed genetic programming to automatically design trigger conditions in multi-agent RL systems, leveraging EA’s structure search ability. These hybrid approaches inspire our framework design: DAGOA inherits GA’s global search capability through dynamic mutation rate adaptation and an early stopping mechanism, while SAC with hierarchical action decomposition enables fine-grained policy learning. Compared with existing GA-DRL hybrids [30], our innovation lies in the uncertainty-quantified critic ensemble that overcomes the challenge of dynamic and partial observability in the environment.

The proposed Dynamic Adaptive Genetic Optimization Algorithm (DAGOA) is an advanced evolutionary computation technique that enhances traditional genetic algorithms by integrating adaptivity and dynamic parameter tuning. This section provides its detailing operational principles, step-by-step procedural descriptions, mathematical foundations, and the innovative mechanisms that contribute to its superior performance in solving complex optimization problems. The rigorous examination of DAGOA’s methodology is shown in the next section.

Genetic algorithms (GAs) are stochastic search methods inspired by the principles of natural evolution and genetics. While effective in exploring large solution spaces, traditional GAs can suffer from premature convergence and inefficiency in dynamic environments. DAGOA addresses these limitations through adaptive mechanisms and early stopping criteria, optimizing the task offloading decisions in user-centric systems. The DAGOA algorithm dynamically adjusts parameters, such as mutation rate and population size, according to the real-time state of the optimization process, which increases the adaptability of the algorithm to complex tasks and dynamic environments, and this adaptability enables the algorithm to maintain the diversity of the population, thus avoiding local optima and increasing the convergence speed. The integration of early stopping criteria further enhances computational efficiency by terminating the process when negligible improvements are observed. The following formulation describes the main mechanism of DAGOA as implemented in the framework.

1.

Initialization: The DAGOA begins by initializing a population of candidate solutions. Each individual is represented as a vector of decision variables, populated with random values from a uniform distribution.

$$\mathbf{P}\left(0\right)=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N\},{\mathbf{x}}_i=(x_{i1},x_{i2},\dots ,x_{iM})\sim \mathrm{Uniform}(0,1)$$

(1)

where $\mathbf{N}$ is the number of individuals, and each ${\mathbf{x}}_i$ consists of decision variables for a specific number of users (user_num).

2.

Fitness evaluation: The fitness of each individual is assessed using a predefined fitness function, which evaluates the quality of the solution based on the state of the system and its parameters. An adaptation function, $f\left({\mathbf{x}}_i\right)$, is defined for each individual to evaluate its offloading decision performance under a specific environment state, s, and location, $(x,y)$.

$$f\left({\mathbf{x}}_i\right)=-\left(w_{\mathrm{energy}}·E\left({\mathbf{x}}_i\right)+w_{\mathrm{delay}}·D\left({\mathbf{x}}_i\right)-w_{\mathrm{coverage}}·C\right)$$

(2)

where $E\left({\mathbf{x}}_i\right)$, $D\left({\mathbf{x}}_i\right)$ and C denote the total energy consumption, the total latency, and the coverage, respectively, which are described in the experiment design section.

3.

Selection: DAGOA employs tournament selection to choose individuals for reproduction. This method selects a subset of individuals randomly, and the one with the highest fitness is chosen as a parent, promoting high-quality solutions while maintaining genetic diversity.

$$\mathrm{Select}\left(\mathbf{P}\right)=arg\underset{\mathbf{x}\in \mathrm{Tournament}}{max}f\left(\mathbf{x}\right)$$

(3)

4.

Crossover: genetic diversity is further enhanced through crossover, where pairs of individuals exchange segments of their decision variables, producing offspring that inherit characteristics from both parents.

$$\begin{array}{c}\hfill \mathrm{Given}{\mathbf{x}}_1=(x_{11},x_{12},\dots ,x_{1N})\\ \hfill \mathrm{and}{\mathbf{x}}_2=(x_{21},x_{22},\dots ,x_{2N}),\\ \hfill \mathrm{Choose}\mathrm{crossover}\mathrm{point}c\in \{1,2,\dots ,N-1\},\\ \hfill \mathrm{Generate}\mathrm{offspring}:\\ \hfill {\mathbf{x}}_{\mathrm{offspring}1}=(x_{11},\dots ,x_{1c},x_{2(c+1)},\dots ,x_{2N})\\ \hfill {\mathbf{x}}_{\mathrm{offspring}2}=(x_{21},\dots ,x_{2c},x_{1(c+1)},\dots ,x_{1N})\end{array}$$

5.

Mutation: The mutation operation introduces random variations in the offspring to explore new regions of the solution space. DAGOA adaptively adjusts the mutation rate, increasing it if the population shows signs of stagnation.

$$x_{ij}=\left\{\begin{array}{cc}\hfill \mathrm{Uniform}(0,1),\hfill & \hfill \mathrm{if}\mathrm{rand}<p_{\mathrm{mutate}}\left(t\right)\hfill \\ \hfill x_{ij},\hfill & \hfill \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

where the probability of variation, $p_{\mathrm{mutate}}\left(t\right)$, increases in the long run without improvement to increase population diversity.

$$p_{\mathrm{mutate}}\left(t\right)=\left\{\begin{array}{cc}\hfill 0.05,\hfill & \hfill \mathrm{if}\mathrm{no}\mathrm{improvement}\mathrm{count}\ge \mathrm{patience}/2\hfill \\ \hfill 0.01,\hfill & \hfill \mathrm{otherwise}\hfill \end{array}\right.$$

(5)

6.

Adaptation and early stopping: The algorithm proceeds through generations, tracking improvements in fitness scores. Early stopping is triggered if the best fitness improvement remains below a defined threshold over several generations, conserving computational resources.

$$\mathrm{if}max\left(f\right(\mathbf{P}\left(t\right)\left)\right)-\mathrm{best}\mathrm{fitness}\le ϵ,\mathrm{then}\mathrm{increment}\mathrm{no}\mathrm{improvement}\mathrm{count}$$

(6)

7.

Extracting the best solution: upon termination, the algorithm selects the individual with the highest fitness as the ultimate solution.

$${\mathbf{x}}^{*}=arg\underset{\mathbf{x}\in \mathbf{P}\left(t\right)}{max}f\left(\mathbf{x}\right)$$

(7)

DAGOA showcases several notable advantages, such as adaptive mutation, early stopping mechanism, and tournament selection, ensuring its robustness and flexibility. Through the above steps, the DAGOA achieves the optimization of the task offloading problem, dynamically adjusts the parameters of the genetic algorithm to adapt to the environmental changes, and improves the efficiency of the algorithm. The DAGOA represents a significant advancement in the field of evolutionary computation. Its adaptive mechanisms and efficient termination criteria make it a powerful tool for handling complex optimization challenges, providing an optimal balance between exploration and exploitation while ensuring computational efficiency.

3.2. Embedded SAC Algorithm

The soft actor–critic (SAC) algorithm is an advanced, model-free, off-policy reinforcement learning method [31]. It was designed to address the exploration–exploitation trade-off by combining the benefits of entropy maximization with policy learning. The principle behind SAC is to learn a stochastic policy that not only maximizes expected returns but also maximizes entropy, leading to more robust and exploratory behavior. In this section, we propose enhancements to the SAC algorithm within the hybrid decision-making framework for UAV-assisted mobile edge computing (MEC) systems. The goal is to improve the robustness and decision efficiency of the framework, addressing the challenges of dynamic environments and multi-objective optimization.

SAC, as a deep reinforcement learning algorithm based on the principle of maximum entropy, is able to learn efficiently in high-dimensional continuous action space. SAC optimizes the strategy learning process and enhances its adaptability to dynamic changes in the environment by introducing a dual structure of the value function and policy network. In this framework, the SAC is responsible for the path planning of the UAV, which directly interacts with the environment by obtaining an effective representation of the environment state, including user position, task type, and obstacle information, and it updates its strategy in real time to maximize the cumulative reward. In addition to SAC, this paper also investigates the way in which algorithms such as PPO, DDPG, and TD3 [32] are combined with the DAGOA algorithm and tests their performance in the same simulation environment, thus verifying the extensibility of the proposed framework and the superiority of the proposed SAC.

The proposed soft actor–critic (SAC) algorithm forms the core of UAV path planning in our hybrid decision framework. It operates in a continuous action space, $\mathcal{A}\subseteq {\mathbb{R}}^2$, where actions correspond to directional control, $a_{\theta }\in [0,\pi /2]$, and speed modulation, $a_v\in [1,10]$. The state space ${\mathbf{s}}_t$ integrates four critical components, including user context, ${\mathbf{s}}_{{\mathrm{user}}_t}$, environmental constraints, ${\mathbf{s}}_{\mathrm{obstacle}}$, UAV status, ${\mathbf{s}}_{{\mathrm{uav}}_t}$, and trajectory history, ${\mathbf{s}}_{{\mathrm{history}}_t}$, which are introduced in Section 4.1.

The policy network ${\pi }_{\varphi }$ and Q-function ensemble ${\left\{Q_{{\theta }_i}\right\}}_{i=1}^N$ jointly optimize the objective:

$$\underset{\pi }{max}{\mathbb{E}}_{\begin{array}{c}s\sim \mathcal{D}\\ a\sim \pi \end{array}}\left[\sum _{t=0}^{\infty }{\gamma }^t\left(r(s_t,a_t)+\alpha \mathcal{H}\left(\pi (·|s_t)\right)\right)\right]$$

(8)

where $\mathcal{H}\left(\pi \right)=-{\mathbb{E}}_{\pi }[log\pi \left(a\right|s)]$ is the policy entropy. On the basis of maximizing rewards in traditional reinforcement learning, this objective function introduces policy entropy, $\mathcal{H}\left(\pi \right)$, as a regularization term. Maximizing entropy encourages strategies to maintain randomness and avoid premature convergence to local optima.

3.2.1. Hierarchical Action Decomposition

Due to the difficulty of exploring the optimized overall control strategy, the action space is decomposed into two coupled sub-spaces to reduce exploration complexity, which can be denoted as a high-level planner and a low-level controller. The high-level planner generates target direction ${\tilde{a}}_{\theta }$ using attention-weighted state features:

$${\tilde{a}}_{\theta }=\mathrm{MLP}\left(\mathrm{Att}(W_q{s}_v,W_k{s}_u)\oplus s_e\right)$$

(9)

where $\mathrm{Att}(·)$ computes multi-head attention weights over user states. The low-level controller outputs speed modulation, $\mathrm{\Delta }{a}_v$, conditioned on obstacle proximity:

$$\mathrm{\Delta }{a}_v=\sigma \left(W_l·\left[s_v\oplus \underset{j}{min}{\parallel s_v-s_e^j\parallel }_2\right]\right)$$

(10)

where $\sigma$ is a sigmoid function constraining output to $[0,1]$.

The design of the hierarchical action decomposition of the proposed SAC could reduce gradient variance, which is similar to the framework proposed by M. Daniel et al. [33]. In our work, let ${\nabla }_{\varphi }J$ and ${\nabla }_{\varphi }{J}_{HAD}$ denote policy gradients under standard and hierarchical action spaces, respectively. Then,

$$\mathrm{Var}\left({\nabla }_{\varphi }{J}_{HAD}\right)\le \mathrm{Var}\left({\nabla }_{\varphi }J\right)$$

(11)

The complete policy gradient for standard SAC is as follows:

$${\nabla }_{\varphi }J={\mathbb{E}}_{\begin{array}{c}s\sim \mathcal{D}\\ a\sim \pi \end{array}}\left[{\nabla }_{\varphi }log\pi \left(a\right|s)·Q(s,a)\right]$$

(12)

Under HAD, the gradient decomposes into the following:

$${\nabla }_{\varphi }{J}_{HAD}=\mathbb{E}\left[{\nabla }_{{\varphi }_h}log{\pi }_h\left({\tilde{a}}_{\theta }\right|s)Q_h(s,{\tilde{a}}_{\theta })\right]+\mathbb{E}\left[{\nabla }_{{\varphi }_l}log{\pi }_l\left(a_v\right|s,{\tilde{a}}_{\theta })Q_l(s,a_v)\right]$$

(13)

Decompose the policy gradient using the law of total variance:

$$\begin{array}{cc}\hfill \mathrm{Var}(\nabla J)& \hfill =\mathrm{Var}\left(\mathbb{E}[\nabla J|a_0]\right)+\mathbb{E}\left[\mathrm{Var}(\nabla J|a_0)\right]\hfill \\ \hfill \mathrm{Var}(\nabla J_{HAD})& \hfill =\mathrm{Var}\left(\mathbb{E}[\nabla J_h|a_0]\right)+\mathbb{E}\left[\mathrm{Var}(\nabla J_l|a_0)\right]\hfill \end{array}$$

(14)

Hierarchical decomposition minimizes cross-term correlations in the second moment matrix, thereby reducing overall variance.

3.2.2. Uncertainty-Quantified Critic Ensemble

To overcome the challenge of dynamic and partial observability in the environment, the Uncertainty-Quantified Critic Ensemble (UQEC) mechanism is employed, introducing an ensemble of $N=5$ Q-networks with randomized initializations, which provides robust value estimation [34]. The Pessimistic Q-Learning target values that incorporate epistemic uncertainty are as follows:

$$y_t=r_t+\gamma \left(\underset{i\in \mathcal{K}}{min}{Q}_{{\theta }_i^{\prime }}(s^{\prime },a^{\prime })-\beta {\sigma }_Q\left(s^{\prime }\right)\right)$$

(15)

where $\mathcal{K}\subset \{1,\dots ,N\}$ is a random subset ($\left|\mathcal{K}\right|=3$) per update, and ${\sigma }_Q\left(s^{\prime }\right)$ represents the ensemble standard deviation. $\beta$ is dynamically adjusted to increase punishment in high-uncertainty environments (such as areas with dense obstacles) according to Formula (17) to avoid high-risk actions.

$${\sigma }_Q\left(s\right)=\sqrt{{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\left(Q_{{\theta }_i}(s,a)-\overline{Q}(s,a)\right)}^2}$$

(16)

The dynamic uncertainty penalty mechanism can adaptively adjust the penalty coefficient, $\beta$, balancing exploration and risk avoidance.

$${\beta }_{t+1}={\beta }_t+{\eta }_{\beta }\left({\mathbb{E}}_D\left[{\sigma }_Q\left(s\right)\right]-{\sigma }_{target}\right)$$

(17)

where ${\sigma }_{target}$ is set to 0.2 to maintain moderate conservatism. The Uncertainty-Quantified Critic Ensemble mechanism could address model bias issues in dynamic environments and enhance the robustness of strategies in partially observable scenarios. The integrated mean $\overline{Q}(s,a)={\displaystyle \frac{1}{N}}\sum Q_{{\theta }_i}(s,a)$ reduces the estimated variance, while the minimum operator constrains the maximum deviation.

$$\mathbb{E}\left[\underset{i}{min}{Q}_{{\theta }_i}\right]\le Q^{*}\le \mathbb{E}\left[\underset{i}{max}{Q}_{{\theta }_i}\right]$$

(18)

When the strategy is updated, the uncertainty penalty term $\beta {\sigma }_Q\left(s^{\prime }\right)$ establishes a probabilistic robust boundary. Assuming the true Q-value $Q^{*}$ and the estimation error ${ϵ}_i\sim \mathcal{N}(0,{\sigma }^2)$, there is the following:

$$\mathbb{P}\left(\underset{i}{min}{Q}_{{\theta }_i}\le Q^{*}-\beta \sigma \right)\ge 1-{\left[\mathrm{\Phi }\left(-{\displaystyle \frac{\beta }{\sqrt{N}}}\right)\right]}^N$$

(19)

When $N=5$, and $\beta =1.6$, the probability exceeds 95% ($\mathrm{\Phi }$ is the standard normal CDF), which can effectively avoid high-risk action choices.

3.2.3. Adaptive Entropy Temperature

The temperature parameter $\alpha$ in SAC is crucial for balancing exploration and exploitation. In dynamic environments, a fixed $\alpha$ may not be optimal. We employ an adaptive temperature tuning mechanism that adjusts $\alpha$ based on the system’s current state and the complexity of the environment. The adaptive entropy temperature mechanism can dynamically balance strategy exploration and utilization, solving the problems of over-exploration at an early stage and under-exploration at a later stage caused by traditional SAC-fixed temperature parameters [35]. In addition to the uncertainty penalty, the dynamic adjustment principle of the strategy entropy weight $\alpha$ is as follows:

$${\alpha }_{t+1}={\alpha }_t+{\eta }_{\alpha }\left({\mathbb{E}}_D\left[\mathcal{H}\left({\pi }_t\right)\right]-{\mathcal{H}}_{target}\right)$$

(20)

where ${\mathcal{H}}_{target}=-dim\left(\mathcal{A}\right)=-2$, corresponding to the two-dimensional action space. By coupling the adjustment strategy, the update of $\beta$ is associated with the uncertainty of the Q value, thereby enhancing the conservatism of the algorithm in high-uncertainty observation states. The coupling update of $\beta$ and $\alpha$ will form a dual closed-loop control, allowing for higher uncertainty penalties ($\beta$ increase) and suppressing risk exploration during the high-exploration period (where $\alpha$ is larger); during the high-utilization period (where $\alpha$ is small), the algorithm will reduce the penalty ($\beta$ is reduced) and fully utilize the known optimal strategy.

The exploratory nature of SAC complements the exploitation focus of genetic algorithms, creating a balanced and holistic decision-making approach. In the context of the hybrid decision framework, the SAC algorithm was chosen due to its unique ability to address the challenges of dynamic and uncertain environments. SAC’s robustness in various settings ensures that it can adapt to different scenarios within the hybrid framework, maintaining high performance despite environmental changes. The integration of the SAC algorithm into the hybrid decision framework represents a strategic innovation, driven by SAC’s strengths in efficient exploration, sample efficiency, and adaptability. These attributes make it an ideal candidate for addressing the complex challenges present in optimizing UAV path planning and task offloading. As a result, SAC forms a critical component of the framework, driving its effectiveness and success in achieving optimal UAV path planning outcomes.

The uncertainty-quantified critic ensemble and adaptive entropy temperature improve the decision performance of the SAC algorithm through dual aspects. At the level of value estimation, the ensemble critic reduces Q-value variance and adapts $\beta$-dynamic constraints to estimate bias; at the level of strategic optimization, entropy regulation maintains exploration, while $\beta$ regulation avoids high-risk areas. Therefore, the improvement in strategies can be decomposed into the following:

$$\mathbb{E}[Q_{new}-Q_{old}]=\underset{\mathrm{Optimize}\mathrm{Reward}}{\underbrace{\mathbb{E}\left[\mathrm{\Delta }{Q}_{optim}\right]}}-\underset{\mathrm{Risk}\mathrm{Penalty}}{\underbrace{\beta \mathbb{E}\left[\mathrm{\Delta }{\sigma }_Q\right]}}+\underset{\mathrm{Explore}\mathrm{Incentives}}{\underbrace{\alpha \mathbb{E}\left[\mathrm{\Delta }\mathcal{H}\right]}}$$

(21)

The dynamic balance of the above three factors ensures that strategy updates always move towards the Pareto-optimal direction of high returns, low risks, and moderate exploration.

By combining DAGOA with SAC, this study realizes an efficient decision-making mechanism for UAVs when performing MEC missions. In the hybrid decision-making framework, SAC is responsible for rapidly responding to environmental changes and optimizing path planning, while AGA improves the overall resource utilization efficiency of the system by evaluating and optimizing mission offloading decisions offline. In the experiments, the hybrid framework is found to outperform the decision-making strategy using SAC alone or the strategy combining a fixed-parameter genetic algorithm and SAC, especially in the face of dynamic user demands and complex environments, improving the performance of the system significantly.

3.3. Integration

Current UAV-assisted mobile edge computing systems usually demand high efficiency in the utilization of computational, communication, and energy resources, so the research on optimizing the task offloading decision and path planning for UAVs is of great significance in this field. An efficient decision-making framework is essential for optimizing resource allocation and improving system performance. In this paper, we propose an innovative hybrid decision-making framework combining deep reinforcement learning algorithms and DAGOA, which is designed to achieve the synergistic optimization of UAV path planning and ground-based mission offloading. By assigning the path planning task to deep reinforcement learning algorithms (e.g., SAC and PPO, etc.), while the ground task offloading decision is handled via DAGOA, the task execution efficiency and resource utilization of UAVs can be improved. The hybrid framework realizes efficient UAV path-planning and task-offloading decisions, demonstrating its innovation and technical superiority in complex environments.

The convergence of DAGOA is guaranteed by its adaptive mechanism, which prevents premature convergence. The convergence of SAC is ensured by its off-policy nature and regularization of entropies. In the aspect of stability, the novelty of the hybrid decision-making framework proposed in this paper is mainly reflected in the following three aspects.

• Improved end-to-end decision making: While traditional decision-making frameworks often require the collaborative work of multiple subsystems, this hybrid decision-making framework realizes end-to-end decision making without additional layers. By integrating two subtasks, path planning and task offloading, into a unified framework, the proposed framework is able to consider the constraints and objectives of these two subtasks simultaneously, leading to better global optimization. In this case, this end-to-end hybrid decision-making framework is highly innovative in current academic research.
• Task decomposition and co-optimization: The framework decomposes, for the first time, the UAV path planning and ground task offloading into two independent tasks for decision-making via the DAGOA algorithm and the DRL algorithm, and this decomposition not only reduces the complexity of the tasks but also achieves overall performance enhancement through the co-optimization.
• Adaptive mechanism: The adaptive mechanism in DAGOA is innovatively proposed to dynamically adjust the crossover and mutation probabilities according to the evolutionary state of the population, which improves the search efficiency. At the same time, the deep reinforcement learning algorithm realizes the steady updating of strategies through methods such as the strategy gradient, and the two complement each other to jointly improve the decision-making performance of the framework.

The core idea of the hybrid framework proposed in this paper is to make full use of the global search capability of genetic algorithms and the local optimization capability of deep reinforcement learning and to further improve the robustness and decision-making efficiency of the hybrid decision-making framework by realizing adaptive adjustment of the parameters of the DAGOA. DAGOA is an improved genetic algorithm that improves the search efficiency and convergence speed by dynamically adjusting the crossover and mutation probabilities. In this framework, DAGOA is used to solve the ground mission offloading tasks. DAGOA generates and optimizes the mission offloading strategy by simulating natural selection and genetic mechanisms. The innovation is the introduction of an adaptive mechanism, which dynamically adjusts the algorithm parameters according to the evolutionary state of the population, thus accelerating convergence and avoiding local optima. DAGOA dynamically adjusts the fitness of the individuals in the population by simulating the process of natural selection to ensure that suitable mission offloading strategies can be found in complex environments. Specifically, DAGOA evaluates the fitness of individuals in each generation and utilizes a tournament selection strategy to select the optimal individuals for crossover and mutation to generate a new population. In order to avoid converging too quickly early on and thus falling into a local optimum, DAGOA introduces the Early Stop mechanism, which ensures that the genetic operation can be terminated when the fitness has not improved significantly.

The integration of DAGOA and SAC not only enhances the autonomous decision-making ability of UAVs in MEC systems but also provides new ideas for research in related fields, demonstrating the potential of combining optimization algorithms with deep learning methods. Future research can further explore the synergistic mechanisms of other optimization algorithms with deep reinforcement learning to expand the applicability and effectiveness of this framework in a wider range of application scenarios.

4. Experimental Design

In this paper, we design a training environment for simulating UAV-assisted mobile edge computing (MEC) systems, especially for the application of the hybrid decision-making framework proposed in this paper for UAV path planning and mission offloading decisions. This environment is able to access deep reinforcement learning algorithms for UAV path planning and DAGOA for the optimization of ground-based task offloading decisions. The design of this environment takes into account a variety of practical factors, introducing users, obstacles, dynamic noise, diverse task types, and multi-dimensional reward mechanisms in the three-dimensional space to simulate the complexity of real application scenarios and to ensure that the trained model has a high practical application value.

The environment is implemented based on the Python 3.10.15 programming language and employs the Box space from the gym library to define the observation and action spaces. The main components of the environment include the UAV, the user, obstacles, and task types. The action state of the UAV is represented by position, velocity, and direction; the user’s position and task requirements change dynamically, and the position of obstacles is randomly generated at each reset.

4.1. Experimental Environment Components

The size of the environment was 5 km × 5 km × 1 km, and the ground user, the airborne UAV, and the airborne obstacle were the three main components of this environment. At each environment reset, the location of the user was randomly generated within a 2D plane of 5 km × 5 km and maintained random motion, with the UAV flying at a fixed altitude, H, of 1 km from the ground. The location of the obstacle was randomly generated within 5 km × 5 km at a height of 1 km, and the distance between the center of the obstacle and the UAV and the user was ensured to be greater than the size of the obstacle. The task load, $L_j$, for each user was randomly generated between $1\times {10}^6$ and $1\times {10}^7$, reallocated at each environmental reset. The task types were categorized into real-time tasks and batch tasks, which were randomly assigned at each environment reset. Detailed parameters of the proposed environment are shown in

Table 1. Detailed parameters of the proposed environment.

Environment Parameter	Value
Maximum user number	30
Maximum obstacle number	10
UAV number	1
Maximum UAV speed	10 m/s
Minimum UAV speed	1 m/s
Maximum UAV acceration	3 m/$s^2$
Size	5 km × 5 km × 1 km

:

The roles and main tasks of each entity in this system are shown in

Table 2. Entity roles and tasks.

Entity	Role	Major Tasks
UAV	Mobile edge server for dynamic resource allocation and path planning	Path planning, task offloading, and energy management
Ground Users	Generate computational tasks and request UAV services	Task generation, dynamic mobility, and task offloading
Obstacles	Physical constraints simulating dynamic environmental interference	Path blocking
MEC system	Global optimization of energy, latency, and coverage	Joint optimization and dynamic reward weight adjustment

.

The user’s initial position ${\mathbf{u}}_j=(u_{jx},u_{jy})$ is randomly distributed and may move at each time interval. The user wanders randomly on the ground with randomized motion step lengths and directions, subject to boundary conditions. Task size per user is noted as $L_j$ (in bits). The task type can be divided into real-time and batches (labeled 0 and 1, respectively). The SAC algorithm realizes the path planning for the UAV by controlling the speed, v, and direction, $\theta$, of the UAV. The state space defined in this environment is a multidimensional vector containing elements such as the UAV state, user information, obstacle location, and historical trajectory of the UAV, and the expression of the complete state space at time t is as follows:

$${\mathbf{s}}_t=({\mathbf{s}}_{{\mathrm{uav}}_t},{\mathbf{s}}_{{\mathrm{user}}_t},{\mathbf{s}}_{\mathrm{obstacle}},{\mathbf{s}}_{{\mathrm{history}}_t})$$

(22)

The UAV state can be represented as follows:

$${\mathbf{s}}_{{\mathrm{uav}}_t}=(p_{x_t},p_{y_t},v_t,{\theta }_t)$$

(23)

where ${\mathbf{p}}_t=(p_{x_t},p_{y_t})$ is the location of the UAV, $v_t$ represents the speed of the UAV, and ${\theta }_t$ represents the direction of the UAV.

The N user states can be expressed as follows:

$${\mathbf{s}}_{{\mathrm{user}}_t}=\{({\mathbf{u}}_{j_t},L_{j_t},T_{j_t})\mid j=1,2,\dots ,N\}$$

(24)

where ${\mathbf{u}}_{j_t}=(u_{j{x}_t},u_{j{y}_t})$ is the location, $L_{j_t}$ is the task size, and $T_{j_t}$ is the task type of each user.

The M obstacle states can be expressed as follows:

$${\mathbf{s}}_{\mathrm{obstacle}}=\{(o_{ix},o_{iy})\mid i=1,2,\dots ,M\}$$

(25)

where ${\mathbf{o}}_i=(o_{ix},o_{iy})$ is the location of each obstacle.

The historical trajectory of the UAV within k time intervals in the current episodes can be represented as follows:

$${\mathbf{s}}_{{\mathrm{history}}_t}=\{(p_{x_{t-k}},p_{y_{t-k}})\mid k=1,2,\dots ,K\}$$

(26)

The action space of the SAC algorithm can be represented as follows:

$$\mathbf{a}=(a_{\theta },a_v)$$

(27)

As a result, the update function of the UAV’s location is determined via the control of the SAC algorithm:

$$p_x^{\prime }=p_x+v·cos\left(\theta \right),p_y^{\prime }=p_y+v·sin\left(\theta \right),$$

(28)

$$v^{\prime }=\mathrm{clip}(v+a_v,v_{min},v_{max}),{\theta }^{\prime }=\theta +a_{\theta }.$$

(29)

The random wandering behavior of ground users can be represented in the following equation:

$$u_{jx}^{\prime }=u_{jx}+S_j·cos\left({\varphi }_j\right),u_{jy}^{\prime }=u_{jy}+S_j·sin\left({\varphi }_j\right),$$

(30)

where $S_j$ is the random speed, and ${\varphi }_j$ is the random direction.

The simulation environment proposed in this paper is set up with observation noise and reward noise, where observation noise refers to the introduction of Gaussian noise in acquiring state information to simulate sensor measurement errors, and reward noise refers to the introduction of random noise in the reward function to simulate the uncertainty of the environment.

The observation noise can be denoted as follows:

$${\tilde{p}}_x=p_x+\mathcal{N}(0,{\sigma }_p^2),{\tilde{p}}_y=p_y+\mathcal{N}(0,{\sigma }_p^2).$$

(31)

The reward noise can be denoted as follows:

$$N_{reward}\sim \mathcal{N}(0,{\sigma }_r^2)$$

(32)

4.2. System Model Design

The system model is a core component in this environment for the quantitative description of the task offloading and path-planning processes in UAV-assisted MEC systems. This component evaluates the total energy consumption and total delay of the system by calculating the communication and computation processes between the user device and the UAV, thus supporting the evaluation of the overall performance metrics of the system.

The channel gain, $g_j$, and transmission rate, $r_j$, between the UAV and the ground user are calculated as follows:

$$g_j={\displaystyle \frac{{\beta }_0}{H^2+{\parallel {\mathbf{u}}_j-\mathbf{p}\parallel }^2}}$$

(33)

$$r_j=B_0{log}_2\left(1+{\displaystyle \frac{P_{\mathrm{user}}·g_j}{{\delta }^2}}\right)$$

(34)

where ${\beta }_0$ is the path loss constant defined according to environment characteristics and carrier frequency, H is the altitude of the UAV, $P_{user}$ is the transmission power of users, and ${\delta }^2$ is the noise power at the UAV. The communication parameters are shown in

Table 3. Communication parameters between the UAV and users.

Communication Parameter	Value
Path loss constant ${\beta }_0$	1 × ${10}^{-5}$
Bandwidth	1 × ${10}^7$ Hz
$\delta$	1 × ${10}^{-5}$ W
$P_{user}$	0.5 W

.

Tasks for each user device can be partially processed locally and partially offloaded to the UAV. The task offloading decision, $o_j$, determines the number of tasks that are offloaded to the UAV and processed locally. The number of tasks processed via the UAV can be represented as follows:

$$L_{\mathrm{uav},j}=o_j·L_j$$

(35)

The amount of tasks processed by users can be represented as follows:

$$L_{\mathrm{user},j}=(1-o_j)·L_j$$

(36)

The time consumption, $t_{\mathrm{up},j}$, and energy consumption, $e_{\mathrm{up},j}$, of uploading tasks from users to the UAV can be expressed as follows:

$$t_{\mathrm{up},j}={\displaystyle \frac{L_{\mathrm{uav},j}}{r_j}},e_{\mathrm{up},j}=P_{\mathrm{user}}·t_{\mathrm{up},j}$$

(37)

The time consumption, $t_{\mathrm{local},j}$, and energy consumption, $e_{\mathrm{local},j}$, of processing tasks locally at users can be expressed as follows:

$$t_{\mathrm{local},j}={\displaystyle \frac{L_{\mathrm{user},j}·C_{\mathrm{user}}}{f_{\mathrm{user}}}},e_{\mathrm{local},j}=K_{\mathrm{user}}·\left(f_{\mathrm{user}}^3\right)·t_{\mathrm{local},j}$$

(38)

where $C_{user}$ is the number of CPU computation cycles required by a user to process 1 bit of data, $f_{user}$ is the local computation frequency of each user, and $K_{user}$ is the capacitance factor of the user CPU. The time consumption, $t_{\mathrm{UAV},j}$, and energy consumption, $e_{\mathrm{UAV},j}$, of processing tasks remotely at the UAV can be expressed as follows:

$$t_{\mathrm{uav},j}={\displaystyle \frac{L_{\mathrm{uav},j}·C_{\mathrm{uav}}}{f_{\mathrm{uav}}}},e_{\mathrm{uav},j}=K_{\mathrm{uav}}·\left(f_{\mathrm{uav}}^3\right)·t_{\mathrm{uav},j}$$

(39)

where $C_{uav}$ is the number of CPU computation cycles required by the UAV to process 1 bit of data, $f_{uav}$ is the local computation frequency of the UAV, and $K_{uav}$ is the capacitance factor of the UAV CPU.

Items in the above equations are shown in

Table 4. Users and the UAV attributes.

Attribute Parameter	Value
$C_{user}$	800
$C_{uav}$	1 × ${10}^3$
$f_{user}$	1 × ${10}^9$ Hz
$f_{uav}$	3 × ${10}^9$ Hz
$K_{user}$	1 × ${10}^{-27}$
$K_{uav}$	1 × ${10}^{-28}$

.

The total delay and total energy consumption of each user are denoted as follows:

$$t_{\mathrm{load},j}=max(t_{\mathrm{local},j},t_{\mathrm{up},j}+t_{\mathrm{uav},j})\times \mathrm{adjustment}\left(T_j\right)$$

(40)

$$e_{\mathrm{load},j}=e_{\mathrm{up},j}+e_{\mathrm{local},j}+e_{\mathrm{uav},j}$$

(41)

where $T_j$ is the task type adjustment factor, which is defined as follows:

$$\mathrm{adjustment}\left(T_j\right)=\left\{\begin{array}{cc}\hfill 0.8,\hfill & \hfill \mathrm{if}{T}_j=\mathrm{Real}\text{-}\mathrm{Time}\mathrm{Task}\hfill \\ \hfill 1.2,\hfill & \hfill \mathrm{if}{T}_j=\mathrm{Batch}\mathrm{Task}\hfill \end{array}\right.$$

(42)

The total delay and total energy consumption of the system are denoted as follows:

$$E_{total}=\sum _{j=1}^N{e}_{\mathrm{load},j},D_{total}=\sum _{j=1}^N{t}_{\mathrm{load},j}$$

(43)

In the environment proposed in this paper, the reward function not only includes a weighted sum of energy consumption, delay, and coverage but also introduces a dynamic weight-adjustment mechanism. The innovation of this mechanism lies in its ability to continuously and adaptively optimize weights in order to ensure that the system maintains optimal performance in changing environments. After a fixed time interval, the system re-evaluates the weights. After the weights are adjusted, the environment will renormalize the weights to ensure that the sum is 1. This normalization ensures the stability and consistency of the evaluation function of the system after the weights are adjusted.

The design of the reward function integrates the system energy consumption, delay, and coverage and is defined as follows:

$$\mathrm{reward}=-\left({\omega }_{\mathrm{energy}}{E}_{total}+{\omega }_{\mathrm{delay}}{D}_{total}-{\omega }_{\mathrm{coverage}}C\right)+N_{reward}$$

(44)

where the coverage reward C is defined as follows:

$$C={\displaystyle \frac{{\sum }_{j=1}^N\mathbb{I}(g_j\ge \mathrm{threshold})}{N}}$$

(45)

The adjustment mechanism of the reward factor for each item can be expressed as follows:

$${\omega }_{\mathrm{energy}}^{\prime }=\mathrm{clip}({\omega }_{\mathrm{energy}}+\mathrm{\Delta }{\omega }_{\mathrm{energy}},0,1),$$

(46)

$${\omega }_{\mathrm{delay}}^{\prime }=\mathrm{clip}({\omega }_{\mathrm{delay}}+\mathrm{\Delta }{\omega }_{\mathrm{delay}},0,1),$$

(47)

$${\omega }_{\mathrm{coverage}}^{\prime }=\mathrm{clip}({\omega }_{\mathrm{coverage}}+\mathrm{\Delta }{\omega }_{\mathrm{coverage}},0,1).$$

(48)

By regularly and dynamically adjusting the weights, the system is able to autonomously adjust the priority of optimization objectives according to real-time environmental changes and task requirements. The dynamic adjustment mechanism can guide the algorithm to focus on the system delay in the early stage to ensure the system performance and focus on the system energy consumption in the later stage to extend the mission timeframe, and thus, it can improve the practicality of the system, as well as the robustness of the system to environmental changes. This mechanism provides strong support for the optimization of UAV path planning and mission offloading strategies, enabling the system to always pursue the all-around optimization of energy consumption, delay, and coverage under extreme conditions, thus improving the overall performance and efficiency of the system.

5. Results

This paper has conducted a series of experiments based on the described environment, aiming to provide insights into the key metrics and their interrelationships with the hybrid decision-making framework in this environment. In this section, the experimental data are analyzed in detail to reveal the superiority demonstrated by the hybrid decision-making framework proposed in this paper.

The performance baselines selected for the experiment are four representative high-performance deep reinforcement learning algorithms, including PPO, SAC, DDPG, and TD3, and their performance is analyzed by comparing the reward scores of the deep reinforcement learning algorithms with the hybrid decision-making framework in the same environment. In order to reflect the superiority of DAGOA, a hybrid decision framework that combines a genetic algorithm without an adaptive mechanism with a deep reinforcement learning algorithm is also set as a control in the experiment.

The reward scores of each algorithm with the hybrid decision framework based on each algorithm during the same number of steps of training (one million steps) are shown in Figure 2. Each solid line in the figure represents the reward score after smoothing with a sliding window; each dashed line represents the original reward score, and each color represents an algorithm. Among them, the hybrid decision frameworks generally achieved higher scores compared to the deep reinforcement learning algorithms, with the best performance being the hybrid decision framework consisting of DAGOA and SAC (the green solid line in the figure), which had an average score of −1149.23.

The training effect of each baseline algorithm and its hybrid decision framework formed with GA and DAGOA according to the approach proposed in this paper are shown in Figure 3. After DAGOA and PPO formed the hybrid decision framework, the score of DAGOA improved by 153.53% compared to PPO in Figure 3c, which obtained the highest increase in all baseline algorithms, but it still failed to reach the highest score of the algorithm proposed in this paper. The hybrid decision-making framework consisting of DAGOA and SAC achieved the highest score among all algorithms by improving the score by 41.42% compared to SAC in Figure 3a. The DDPG and TD3 algorithms also gained 5.76% in Figure 3d and 43.63% in Figure 3b in improvement, respectively, after the hybrid decision framework was applied. The effectiveness of the hybrid decision framework and the superior performance of the hybrid decision framework of DAGOA and SAC are well proven through this set of experiments.

PPO stabilizes the training process by limiting the magnitude of policy updates. The significant improvement in the score of PPO in combination with DAGOA indicates that DAGOA is able to effectively explore the policy space and overcome the problem of convergence of PPO in the vicinity of the local optimum solution. The diversity provided via DAGOA allows PPO to jump out of the local optimum and explore a better solution, but due to the inherent limitation of the policy updates of PPO, it fails to reach the highest score of the proposed algorithm in this paper. SAC, with its entropy regularization mechanism, maintains a certain degree of randomness in strategy exploration to avoid premature convergence and combines with the global search capability of DAGOA to explore the strategy space more comprehensively. This combination makes full utilization of the exploration ability of the genetic algorithm and the randomness of SAC, resulting in a balance between the diversity of strategies and the degree of optimization, thus achieving the highest score. DDPG and TD3, as policy gradient-based methods, are susceptible to strategies falling into local optima. The addition of DAGOA introduces new policy variants through genetic manipulation, which enhances the diversity of policy search [36]. In particular, the introduction of double-delayed updating in TD3 further mitigates the policy estimation bias, which produces a more significant enhancement in combination with DAGOA.

In the complex dynamic environments of MEC-oriented systems, the action space dimension of reinforcement learning is high, path planning and task offloading face multiple uncertainties, and a single deep reinforcement learning algorithm may be struggling to optimize all the objectives at the same time. DAGOA provides a stronger ability to adapt to the environment through its adaptive genetic mechanism, which makes the decision-making process more flexible and robust. These results fully demonstrate the effectiveness of the hybrid decision-making framework composed of DAGOA and deep reinforcement learning algorithms, especially in combination with SAC, which exhibits significant performance advantages in complex dynamic environments.

In UAV-assisted MEC systems, as the number of ground users increases, the total energy consumption and total delay of the system increase accordingly, but the increase will be different for systems under the control of different algorithms. In order to verify the robustness of the hybrid decision-making framework proposed in this paper to the number of users in the MEC system, we conducted multiple sets of experiments when the number of users ranged from 3 to 30, we selected two series of algorithms based on SAC and PPO for comparison, and the results are shown in Figure 4c. The experimental results show that the hybrid decision framework of DAGOA and SAC achieves the best robustness to changes in the number of users, with higher scores than the PPO-based algorithms at all numbers of users. The SAC algorithm, with the introduction of the hybrid decision framework, improves its scores at all numbers of users tested except 10, with a maximum improvement of 146.26% in the three-user environment in Figure 4a. As a control, the PPO algorithm also achieved an improvement in its scores for all numbers of users after the introduction of the DAGOA hybrid decision framework, with the improvement ranging between 36.45% and 200.22% in Figure 4b.

Deep reinforcement learning algorithms such as SAC and PPO learn optimal strategies by continuously interacting with the environment when dealing with complex, dynamic environments. However, these algorithms may encounter the problem of insufficient generalization ability when facing large-scale changes in the number of users. The introduction of the DAGOA compensates for this deficiency through its global search capability and adaptive tuning mechanism, which improves the algorithm’s robustness in multi-user environments.

Based on the strong exploration capability inherent to the SAC algorithm, the introduction of DAGOA further optimizes its decision-making process, enabling the algorithm to find better path planning and task offloading strategies under more user environments. Especially in the three-user environment, the score is improved by as much as 146.26%, which demonstrates the effectiveness of the hybrid decision-making framework. The PPO algorithm, after the introduction of DAGOA, slightly lags behind SAC in terms of improvement, although the score is improved under each number of users. This may be related to the characteristics of the PPO algorithm itself; e.g., it focuses more on strategy optimization using known information and is relatively weak in global search and adaptive tuning [37]. Nevertheless, the addition of DAGOA still significantly improves the performance of PPO in a multi-user environment, suggesting that its enhancement of algorithm performance is of general interest.

In the simulation environment designed in this paper, the UAV performs autonomous maneuvering along with mission offloading, and the following Figure 5 represents the trajectory of the autonomous maneuvering of the UAV with mission offloading. In Figure 5, the red dot represents the UAV, the blue dot represents the ground user, the green dotted line represents the trajectory of the UAV, the red line represents the UAV mission offloading, and the gray area represents the obstacles. The trajectory in the figure reflects that the UAV not only optimizes the trajectory for the distribution of users in the mission area but also avoids obstacles. The hybrid decision framework of the DAGOA and SAC algorithms achieves the optimization of the overall energy consumption and the overall delay of the MEC system through the real-time control of the UAV and the real-time mission offloading and, at the same time, protects the flight safety and endurance of the UAV, providing solid application foundation for MEC systems.

Figure 5a demonstrates that the hybrid decision-making framework of DAGOA and SAC strives to cover more users while performing path planning for UAVs in a narrow space, which not only safeguards the overall latency and energy consumption of the system but also realizes safe and reliable services. Figure 5b demonstrates that the hybrid decision-making framework proposed in this paper is not only capable of long-distance maneuvering but also enables the UAV to hover according to the dynamic demand, and in scenarios where the users are relatively concentrated and safe, the UAV is able to select the relatively optimal position for hovering to maintain a low overall system latency. Figure 5c,d show the process of long-distance maneuvering via the UAV to find the real-time optimal location, respectively, in which the maneuvering process not only avoids obstacles but also takes into account the task offloading tasks in the vicinity of the waypoint. Therefore, the hybrid decision-making framework proposed in this paper has shown superior performance in dynamic scenarios.

In the SAC algorithm proposed in this article, the Uncertainty-Quantified Critic Ensemble and adaptive entropy temperature form a synergistic enhancement effect at the level of value estimation and strategy optimization through rigorous theoretical design. Experiments show that the two can reduce the collision rate by 15.6% in static user, dynamic user, and dynamic task scenarios and jointly improve user coverage by 21.1%, providing a robust and efficient decision-making basis for UAV edge computing scenarios.

Table 5. Comparison of collision rates under different environmental disturbances.

Tasks	Baseline SAC	Proposed SAC with UQCE
Static users	5.2%	1.1%
Dynamic users	15.7%	4.3%
Dynamic tasks	23.4%	7.8%

indicates the verification of the UQCE mechanism’s effectiveness, where the proposed SAC with UQCE performs better than the baseline SAC in multiple scenarios.

Table 6. Adaptive temperature ablation experiment.

Temperature Regulation Mechanism	User Coverage Rate	Energy Consumption
Static $\alpha$ = 0.2	73.5%	1.43 kj
Adaptive $\alpha$	89.2%	1.29 kj
Static $\alpha$ = 0.2 and $\beta$ = 1.0	68.1%	1.57 kj

indicates that the coupling update of $\alpha$ and $\beta$ can avoid the limitations of single parameter adjustment. The adaptive alpha mechanism exhibits significant advantages in coverage metrics in the experiment.

The hybrid framework’s training time (average 6.2 h for 1M steps on an NVIDIA RTX4090 GPU) is higher than simpler baselines like PPO (5.1 h) due to the dual-algorithm interaction. However, the inference phase computational overhead is comparable to that of standalone DRL algorithms (average 12 ms/decision), making it feasible for real-time deployment.

6. Conclusions

In summary, it is of great significance to validate the hybrid decision-making framework consisting of proposed DAGOA and deep reinforcement learning for the application of UAV-assisted MEC systems, which also demonstrates significant innovativeness with the proposed improved SAC algorithm. Through interdisciplinary algorithm fusion, dynamic task allocation, and optimization, the introduction of adaptive mechanisms, an end-to-end decision-making framework, and the combination of empirical research and theoretical analysis, the framework achieves an efficient, robust, and scalable decision-making process. By adopting the hybrid decision-making framework, the limitations of the pure DRL method in dealing with complex dynamic environments and multi-objective optimization problems can be effectively alleviated, and the contradiction between supply and demand in resource allocation and the heterogeneity of supply and demand in computational offloading can be resolved so as to improve the resource utilization rate and the efficiency. Through the combination of DAGOA and SAC or other DRL algorithms, the framework is able to realize the joint optimization of computational resource allocation, task offloading, and trajectory control on different time scales to improve the overall performance of the system and, at the same time, realize the energy saving and real-time trajectory control of UAVs in the MEC system to prolong the endurance time of the UAVs and reduce energy consumption, which provides a new idea and methodology for the research and application of related fields.

In future work, the proposed framework can be tested under varying channel models (e.g., urban vs. rural path loss and dynamic interference) and multi-UAV coordination scenarios. Energy-constrained UAVs with heterogeneous computational capacities in the dynamic environment can also be involved in extending the implementation of this framework. While the current framework supports up to 30 users, scaling to ultra-dense networks (100+ users) requires distributed training strategies, whose exploration in future work is also planned.