Task Offloading Strategy for UAV-Assisted Mobile Edge Computing with Covert Transmission

Abstract

Task offloading strategies for unmanned aerial vehicle (UAV) -assisted mobile edge computing (MEC) systems have emerged as a promising solution for computationally intensive applications. However, the broadcast and open nature of radio transmissions makes such systems vulnerable to eavesdropping threats. Therefore, developing strategies that can perform task offloading in a secure communication environment is critical for both ensuring the security and optimizing the performance of MEC systems. In this paper, we first design an architecture that utilizes covert communication techniques to guarantee that a UAV-assisted MEC system can securely offload highly confidential tasks from the relevant user equipment (UE) and calculations. Then, utilizing the Markov Decision Process (MDP) as a framework and incorporating the Prioritized Experience Replay (PER) mechanism into the Deep Deterministic Policy Gradient (DDPG) algorithm, a PER-DDPG algorithm is proposed, aiming to minimize the maximum processing delay of the system and the correct detection rate of the warden by jointly optimizing resource allocation, the movement of the UAV base station (UAV-BS), and the transmit power of the jammer. Simulation results demonstrate the convergence and effectiveness of the proposed approach. Compared to baseline algorithms such as Deep Q-Network (DQN) and DDPG, the PER-DDPG algorithm achieves significant performance improvements, with an average reward increase of over 16% compared to DDPG and over 53% compared to DQN. Furthermore, PER-DDPG exhibits the fastest convergence speed among the three algorithms, highlighting its efficiency in optimizing task offloading and communication security.

1. Introduction

With the advancement of 5G technology, CPU-bound applications on user equipment (UE) are increasingly prevalent and thriving, which has led to significant demand for computing resources, generating high levels of energy consumption. While Mobile Cloud Computing (MCC) can address this issue to some extent, the cloud server is generally physically far away from the UE, which causes high transmission latency and adversely affects user experience [1]. In this context, as a promising technology, Mobile Edge Computing (MEC) relocates cloud computation assets and facilities nearer to the UE and provides real-time processing to speed up data analysis and decision-making processes [2].

In recent years, with the advantages of reduced production cost, high adaptability in deployment, and the wide range of applications, the research on unmanned aerial vehicle (UAV) -assisted MEC has attracted extensive attention, especially in resource allocation and path planning [3]. Li et al. [4] constructed a Dinkelbach algorithm that aims to minimize UAV energy expenditure and maximize energy efficiency by optimizing computational offloading. Diao et al. [5] formulated an optimization algorithm that jointly optimizes task data distribution to conserve energy and UAV trajectories. Lin et al. [6] proposed a PDDQNLP algorithm to tackle the challenges of UAV trajectory design and terminal device decisions, optimizing the offloading decisions and time allocation. Selim et al. [7] also suggested incorporating Device-to-Device (D2D) communication as a supplementary means to facilitate both computation offloading and communication within the UAV-MEC system. However, in most research on UAV-assisted MEC systems, information is primarily conveyed via radio transmissions. Given the inherently broadcast and accessible characteristics of radio communications, UAV-assisted MEC faces a significant risk of eavesdropping [8,9].

Traditionally, cryptography technology can be used to share keys between nodes to enable secure communications [10], but the computational load and time increases, which is contrary to the goal of MEC to reduce latency [11]. In addition, physical layer security techniques also enhance UAV communication confidentiality. They maximize the disparity between legitimate and eavesdropping channels through resource allocation and noise generation, making it difficult for eavesdroppers to decode information accurately [12,13]. Yet, these methods alone cannot fully protect against eavesdropping. In certain critical scenarios, concealing transmissions is essential.

As a high-level security requirement, covert communication not only prevents the message content from being deciphered by adversaries, but also ensures that the existence of wireless transmission is undetected by keeping the detection probability of the warden below a set threshold [14]. Chen et al. [15] demonstrated that a maximum of $\mathcal{O}\left(\sqrt{n}\right)$ bits can be securely and secretly transmitted from Alice to Bob over AWGN channels in n transmissions. Following this principle of square root scaling, Yan et al. [16] investigate the optimality of Gaussian signaling for covert communications under Kullback–Leibler (KL) divergence constraints. They demonstrate that Gaussian signaling maximizes mutual information when constrained by an upper bound on $\mathcal{D}\left(p_1\right|\left|p_0\right)$. Using these principles, it was found that despite their high detectability due to dominant LoS links, the flexible 3-D mobility of UAV base stations (UAV-BSs) can improve system performance [17]. Consequently, studies on covert communications involving UAVs have been conducted in the past few years [18,19,20,21,22]. In [18], an integrated UAV positioning and transmit power optimization was aimed at maximizing communication capacity within a stealth-constrained three-node model. Chen et al. [19] propose an optimal design for transmit power and IRS reflection matrix in an IRS-assisted system to maximize detection at a legitimate receiver while ensuring low detection at a warden. Moreover, Mao et al. [20] present a framework for energy-efficient UAV-assisted short-packet transmission systems that addresses security against cooperative detection and eavesdropping by maximizing the Minimum Covert Secrecy Energy Efficiency (MCSEE) through a dual-loop iterative optimization algorithm. In the studies [21,22], a scenario involving covert transmissions from a UAV-BS to multiple users was examined, and the average covert performance was ultimately enhanced by co-optimizing the user association, the transmit power, and the UAV trajectory. In addition, artificial noise is also an effective strategy to impair the detection capabilities of wardens, while complementing the flexibility of UAV-BSs. By fine-tuning the transmit power of a cooperative jammer, it is possible to introduce uncertainty in the received power for wardens while keeping the negative effects on legitimate users to a minimum; therefore, the covert performance will be enhanced. In [23], Arzykulov et al. proposed a virtual partitioning of reconfigurable intelligent surface (RIS) elements to improve physical layer security by enhancing the signal for legitimate users and increasing artificial noise (AN) for illegitimate users. A UAV with two antennas, one for receiving covert data and the other for transmitting jamming signals, was studied by Zhou et al. [24]. Du et al. [25] formulate the problem of achieving covert communication and high data rates as a Nash bargaining game and propose two algorithms and demonstrate their effectiveness. However, the existing research above is still incomplete. For example, there has been insufficient exploration on covert transmission from one single UAV-BS to multiple end devices. Additionally, few research efforts have addressed task offloading in covert communication scenarios, and there is a scarcity of employing deep reinforcement learning to tackle these challenges.

It is worth noting that recent studies have further advanced the field of covert communication. For instance, Wang et al. [26] explored the trade-offs among reliability, covertness, latency, and transmission rate in wireless short-packet communications, proposing optimization frameworks to maximize throughput under proactive warden scenarios. Meanwhile, Ji et al. [27] proposed an OOK scheme with phase deflection for covert communication, leveraging Willie’s phase uncertainty to enhance transmission rates while ensuring covertness. Although these works provide valuable insights into the design and optimization of covert communication systems, their research focus is distinct from the scope of this study, which centers on the integration of covert communication into UAV-assisted MEC systems.

In this research, we consider an MEC system assisted by UAVs and covert communications, covering both air and ground network layers, in which the fixed-position warden and the slow-moving UEs are located on the ground, and the UAVs in the air play the roles of the jammer and the UAV-BS, respectively. The UEs offload excessive processing-demanding tasks to the UAV-BS to implement computation, and to prevent data transmission activities from being detected by the warden, the jammer transmits interference signals to the warden. We divide the system period equally into some time slots and assume orthogonal frequency division multiple access between the UAV-BS and the UEs. By collaboratively optimizing the movement of the UAV-BS, the resource allocation of the system and the transmit power of the jammer, the system delay, and the detection accuracy of the warden are minimized, which ensures the flexibility, real-time performance, and security of the system. Unlike previous studies, our work addresses the joint optimization of task offloading, resource allocation, and covert communication in UAV-assisted MEC systems, leveraging deep reinforcement learning to dynamically adapt to the changing environment. Specifically, we tackle the challenge of task offloading from UEs to a UAV-BS under covert communication constraints, a scenario that has been largely overlooked in existing research. This approach not only enhances the security and efficiency of MEC systems but also opens new avenues for research in UAV-assisted covert communication. The main contributions of this paper are summarized below.

• We establish a system that employs a UAV-BS and a UAV-Jammer to assist UEs with demanding computational tasks while remaining undetected by a warden. In this system, both the UEs and the UAV-BS can perform calculations while simultaneously transmitting and receiving information. After developing transmission and computation models for the UEs and UAV-BS, as well as formulating binary hypothesis testing to assess the surveillance mechanisms of the warden, we frame the overall task offloading strategy of the system as an optimization problem.
• The optimization problem is reformulated as a Markov decision process (MDP), and we employ a Deep Deterministic Policy Gradient (DDPG) algorithm to obtain the best policy. To improve the efficiency of experience replay in DDPG and accelerate the learning process, we integrate a prioritized experience replay (PER) mechanism into the DDPG framework and propose a novel PER-DDPG algorithm, which means we switch from the previous uniform sampling method to priority sampling. By adopting this approach, we accelerate the algorithm’s convergence and significantly enhance its performance.
• To evaluate the performance of our proposed method, extensive simulations are performed. Initially, we verify the impact of various parameters on the convergence of the proposed algorithm. Subsequently, ablation experiments are carried out to confirm the effectiveness of the proposed algorithm. Ultimately, simulation results from our comparisons demonstrate that our proposed solution outperforms existing methods regarding reductions in both system delay and the detection accuracy of the warden.

The remainder of the paper is organized as follows: Section 2 presents the system model and formulates the optimization problem of the secrecy transmission and task offloading. In Section 3, we construct the MDP and propose the PER-DDPG algorithm. Finally, performance simulations and conclusions are provided in Section 4 and Section 5.

2. System Model

A system model of UAV-assisted covert MEC is shown in Figure 1, composed of a UAV-BS, a UAV jammer, a warden, and K UEs. Let $\mathcal{K}=\{1,2,\dots ,K\}$ represent the set of UEs, and let the system duration be discretized into I equal time slots, denoted as $\mathcal{T}=\{1,2,\dots ,I\}$. At every time slot $i\in \mathcal{T}$, the UE $k\in \mathcal{K}$ generates a large number of computational tasks while moving at a low speed. Based on environmental factors such as distance, task size, and other relevant parameters, the UAV-BS flies to the vicinity of the selected UE and receives a portion of the task offloaded by the UE. During this time, a warden is stationed on the ground to monitor UEs while the jammer hovers at a fixed location and secures the communication link between UAV-BS and UEs by emitting interference to the warden. In the 3D Cartesian coordinate system, let $h_1$ and $h_2$ be the altitudes of the UAV-BS and jammer; then, the locations of the UAV-BS, jammer, warden, and UE k can be denoted by $L^{BS}\left(i\right)=(x^{BS}\left(i\right),y^{BS}\left(i\right),h_1)$, $L^J\left(i\right)=(x^J\left(i\right),y^J\left(i\right),h_2)$, $L^W\left(i\right)=(x^W\left(i\right),y^W\left(i\right),0)$ and $L_k^{UE}\left(i\right)=(x_k^{UE}\left(i\right),y_k^{UE}\left(i\right),0)$, respectively.

2.1. Communications Model

For convenience, $U=\left\{\mathrm{UAV}-\mathrm{BS},\mathrm{jammer}\right\}$ denotes the group of aerial nodes consisting of a UAV-BS and jammer, and $V=\left\{\mathrm{UEs},\mathrm{warden}\right\}$ denotes the group of ground nodes including UEs and the warden. Assuming that the air-to-ground link in this system is a combination of line-of-sight (LoS) and non-line-of-sight (NLoS) channels [28], in time slot i, the likelihood of LoS between nodes $u\in U$ and $v\in V$ can be approximated as follows:

$$p_{u.v}^{LoS}\left(i\right)={\displaystyle \frac{1}{1+aexp(-b({\theta }_{u,v}\left(i\right)-a))}},u\in U,v\in V,$$

(1)

where ${\theta }_{u,v}\left(i\right)=(180/\pi )arctan(h_u/r_{u,v}\left(i\right))$ is the elevation angle from u to v, with $r_{u,v}\left(i\right)=\sqrt{{(x_u-x_v)}^2+{(y_u-y_v)}^2}$, $h_u$ represents the elevation angle and horizontal distance from u to v, and a and b are environmental conditions associated with the concentration of buildings and the distribution of building heights, etc. Using $f_c$ to denote the carrier frequency, let ${\mu }_{LoS}$ and ${\mu }_{NLoS}$ denote the loss factors for LoS and NLoS links, and c denote the light speed; the average path loss between u and v can be calculated by the following:

$$L_{u,v}\left(i\right)=(p_{u,v}^{LoS}\left(i\right){\mu }_{LoS}+p_{u,v}^{NLoS}\left(i\right){\mu }_{NLoS}){\left({\displaystyle \frac{4\pi f_c{d}_{u,v}\left(i\right)}{c}}\right)}^2,$$

(2)

where $d_{u,v}\left(i\right)=\sqrt{{(x_u-x_v)}^2+{(y_u-y_v)}^2+h_u^2}$ is the Euclidean distance between u and v, and $p_{u,v}^{NLoS}\left(i\right)=1-p_{u,v}^{LoS}$ is the NLoS probability. Then, the reciprocal of the average path loss represents the average channel gain between u and v:

$$G_{u,v}\left(i\right)={\displaystyle \frac{1}{L_{u,v}\left(i\right)}}.$$

(3)

To prevent communication interference, the frequency bands allocated by the UAV-BS to each UE are orthogonal, as are those assigned by the warden. The total bandwidth B of the UAV-BS is equally distributed to K UEs, so that each UE has a communication bandwidth of $b_k=B/K$. Therefore, the wireless transmission rate from UE k to UAV-BS is

$$r_k\left(i\right)=b_k{log}_2(1+{\displaystyle \frac{P_k{G}_{u,v}\left(i\right)}{{\sigma }^2}}),$$

(4)

where the Gaussian white noise power in the communication link is given by ${\sigma }^2$, and $P_k$ takes the maximum value of the transmit power of UE k in order to reduce the system delay.

2.2. Computational Model

At each time slot i, the UE k generates tasks to be computed, denoted by a binary group $Tas{k}_k=\{D_k\left(i\right),s\}$, with $D_k\left(i\right)$ representing the size of the computation task and s the value of CPU cycles required per unit bit. Assuming that $R_k\left(i\right)\in \left[0,1\right]$ is the fraction of tasks delegated to the UAV-BS, the fraction retained in the UE k is $(1-R_k\left(i\right))$. Let $f_{UE}$ represent the computational power of the UE; consequently, the local processing latency of UE k is

$$t_k^{local}\left(i\right)={\displaystyle \frac{(1-R_k\left(i\right))D_k\left(i\right)s}{f_{UE}}}.$$

(5)

In time slot i, the UAV-BS transitions from $L^{BS}\left(i\right)$ to the new location $L^{BS}(i+1)=(x^{BS}\left(i\right)+v\left(i\right)t_{fly}cos\beta \left(i\right),y^{BS}\left(i\right)+v\left(i\right)t_{fly}sin\beta \left(i\right),h_1)$, with speed $v\left(i\right)\in \left[0,v_{max}\right]$ and angle $\beta \left(i\right)\in \left[0,2\pi \right]$, and the energy used to fly can be represented as follows:

$$E_{fly}\left(i\right)=0.5M_{BS}{t}_{fly}{∥v\left(i\right)∥}^2,$$

(6)

where $M_{BS}$ and $t_{fly}$ are the payload mass and fixed flight time of the UAV-BS, respectively. Although UAV-BS hovering also consumes energy, it is so tiny compared to flight energy and can be ignored. Similarly, the downlink transmission delay is disregarded due to the minimal size of computation results from the UAV-BS. In this case, the processing delay caused by a UAV-BS consists of two parts, i.e., transmission delay $t_k^{tr}\left(i\right)$ and computational delay $t_{BS}^{com}\left(i\right)$:

$$t_k^{tr}\left(i\right)={\displaystyle \frac{R_k\left(i\right)D_k\left(i\right)}{r_k\left(i\right)}},$$

(7)

$$t_{BS}^{com}\left(i\right)={\displaystyle \frac{R_k\left(i\right)D_k\left(i\right)s}{f_{BS}}},$$

(8)

where $f_{BS}$ denotes the computing frequency of the UAV-BS CPU. Correspondingly, the energy consumption for offloading tasks is also determined by both communication and computation. Ref. [29] provides a detailed computational model, but since the downlink energy consumption in this paper is negligible, we omit the communication energy consumption from the UAV-BS to the UEs and simplify the computational energy consumption as follows:

$$E_{BS}^{com}\left(i\right)=\eta f_{BS}^3{t}_{BS}^{com}\left(i\right)=\eta f_{BS}^2{R}_k\left(i\right)D_k\left(i\right)s,$$

(9)

where $\eta$ is the energy consumption factor related to the UAV-BS CPU model.

2.3. Binary Hypothesis Testing at Warden

The warden adopts binary hypothesis testing based on received signal observations to detect UE data transmission in covert communications. Assuming that the warden obtains the precise locations of all UEs and the jammer through radar or UAV position observation [22], the average receive power of the warden in time slot i is

$$T\left(i\right)=\left\{\begin{array}{cc}{P}_J\left(i\right)G_{J,W}\left(i\right)+{\sigma }^2,\hfill & {\mathcal{H}}_0\hfill \\ \sum _{k=1}^K{P}_k{G}_{k,W}\left(i\right)+P_J\left(i\right)G_{J,W}\left(i\right)+{\sigma }^2,\hfill & {\mathcal{H}}_1,\hfill \end{array}\right.$$

(10)

where $P_J\left(i\right)$ represents the transmit power of the jammer, while ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$ denote the null hypothesis (UE is silent) and the alternative hypothesis (UE is transmitting), respectively. Set the detection threshold to ${\tau }^{th}$, and use ${\mathcal{D}}_0$ and ${\mathcal{D}}_1$ to denote decisions in support of ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$, respectively; then, the false alarm rate $\phi \left(i\right)$ and miss detection rate $\psi \left(i\right)$ of the warden can be calculated as follows:

$$\begin{array}{c}\hfill \phi \left(i\right)=\mathrm{Pr}\left\{{\mathcal{D}}_1|{\mathcal{H}}_0\right\}=\mathrm{Pr}\left\{P_J\left(i\right)G_{J,W}\left(i\right)+{\sigma }^2\ge {\tau }^{th}\right\},\end{array}$$

(11)

$$\begin{array}{c}\hfill \psi \left(i\right)=\mathrm{Pr}\left\{{\mathcal{D}}_0|{\mathcal{H}}_1\right\}=\mathrm{Pr}\left\{\sum _{k=1}^K{P}_k{G}_{k,W}\left(i\right)+P_J\left(i\right)G_{J,W}\left(i\right)+{\sigma }^2\le {\tau }^{th}\right\}.\end{array}$$

(12)

Suppose $P_J\left(i\right)$ follows a uniform distribution within $\left[0,P_J^{max}\right]$; its probability density function (PDF) is

$$f_{P_J\left(i\right)}\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{P_J^{max}}},\hfill & 0\le x\le P_J^{max}\hfill \\ 0,\hfill & otherwise.\hfill \end{array}\right.$$

(13)

Applying (13) to Equations (11) and (12), we obtain

$$\begin{array}{cc}\hfill \phi \left(i\right)& =\mathrm{Pr}\left\{P_J\left(i\right)\ge {\displaystyle \frac{{\tau }^{th}-{\sigma }^2}{G_{J,W}\left(i\right)}}\right\}={\int }_{max(0,{\displaystyle \frac{{\tau }^{th}-{\sigma }^2}{G_{J,W}\left(i\right)}})}^{P_J\left(i\right)}{f}_{P_J\left(i\right)}\left(x\right)dx\hfill \\ \hfill & =\left\{\begin{array}{cc}1,\hfill & {\tau }^{th}\le {\sigma }^2\hfill \\ 1-{\displaystyle \frac{{\tau }^{th}-{\sigma }^2}{P_J\left(i\right)G_{J,W}\left(i\right)}},\hfill & {\sigma }^2<{\tau }^{th}\le {\mu }_1\hfill \\ 0,\hfill & {\mu }_1<{\tau }^{th},\hfill \end{array}\right.\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill \psi \left(i\right)& =\mathrm{Pr}\left\{P_J\left(i\right)\le {\displaystyle \frac{{\tau }^{th}-{\sum }_{k=1}^K{P}_k{G}_{k,W}\left(i\right)-{\sigma }^2}{G_{J,W}\left(i\right)}}\right\}={\int }_0^{min(P_J\left(i\right),{\displaystyle \frac{{\tau }^{th}-{\sum }_{k=1}^K{P}_k{G}_{k,W}\left(i\right)-{\sigma }^2}{G_{J,W}\left(i\right)}})}{f}_{P_J\left(i\right)}\left(x\right)dx\hfill \\ \hfill & =\left\{\begin{array}{cc}0,\hfill & {\tau }^{th}\le {\mu }_2\hfill \\ {\displaystyle \frac{{\tau }^{th}-{\sum }_{k=1}^K{P}_k{G}_{k,W}\left(i\right)-{\sigma }^2}{P_J\left(i\right)G_{J,W}\left(i\right)}},\hfill & {\mu }_2<{\tau }^{th}\le {\mu }_3\hfill \\ 1,\hfill & {\mu }_3<{\tau }^{th},\hfill \end{array}\right.\hfill \end{array}$$

(15)

where ${\mu }_1\triangleq P_J\left(i\right)G_{J,W}\left(i\right)+{\sigma }^2$, ${\mu }_2\triangleq {\sum }_{k=1}^K{P}_k{G}_{k,W}\left(i\right)+{\sigma }^2$, and ${\mu }_3\triangleq {\sum }_{k=1}^K{P}_k{G}_{k,W}\left(i\right)+P_J\left(i\right)G_{J,W}\left(i\right)+{\sigma }^2$.

The detection performance of the warden can be assessed by the error rate $\xi \left(i\right)=\phi \left(i\right)+\psi \left(i\right)$. If ${\mu }_1\le {\mu }_2$, i.e., $P_j\left(i\right)g_{j,w}\left(i\right)\le {\sum }_{k=1}^K{P}_k{g}_{k,w}\left(i\right)$, by setting ${\tau }^{th}={\mu }_2$, it is possible to achieve a detection error rate of zero at the warden. Consequently, it is imperative to examine the broader scenario where ${\mu }_2\le {\mu }_1\le {\mu }_3$, for which the detection error rate can be derived as follows:

$$\xi \left(i\right)=\left\{\begin{array}{cc}1,\hfill & {\tau }^{th}\le {\sigma }^2\hfill \\ 1-{\displaystyle \frac{{\tau }^{th}-{\sigma }^2}{P_J\left(i\right)G_{J,W}\left(i\right)}},\hfill & {\sigma }^2<{\tau }^{th}\le {\mu }_2\hfill \\ 1-{\displaystyle \frac{{\sum }_{k=1}^K{P}_k{G}_{k,W}\left(i\right)}{P_J\left(i\right)G_{J,W}\left(i\right)}},\hfill & {\mu }_2<{\tau }^{th}\le {\mu }_1\hfill \\ {\displaystyle \frac{{\tau }^{th}-{\sum }_{k=1}^K{P}_k{G}_{k,W}\left(i\right)-{\sigma }^2}{P_J\left(i\right)G_{J,W}\left(i\right)}},\hfill & {\mu }_1<{\tau }^{th}\le {\mu }_3\hfill \\ 1,\hfill & {\mu }_3<{\tau }^{th}.\hfill \end{array}\right.$$

(16)

With the goal of minimizing the total detection error rate, the warden sets an appropriate threshold. From (16), it can be seen that in order to ensure that $\xi \left(i\right)<1$, there must be ${\sigma }^2<{\tau }^{th}\le {\mu }_3$. Moreover, $\xi \left(i\right)$ monotonically decreases with ${\tau }^{th}$ for ${\sigma }^2<{\tau }^{th}\le {\mu }_2$ and monotonically increases for ${\mu }_1<{\tau }^{th}\le {\mu }_3$, which implies that $\xi \left(i\right)$ is a continuous function of ${\tau }^{th}$. Thus, by setting the threshold ${\mu }_2<{\tau }^{t{h}^{*}}\le {\mu }_1$, the warden can obtain the minimum detection error rate ${\xi }^{*}\left(i\right)$ as follows:

$${\xi }^{*}\left(i\right)=1-{\displaystyle \frac{{\sum }_{k=1}^K{P}_k{G}_{k,W}^{*}\left(i\right)}{P_J^{*}\left(i\right)G_{J,W}^{*}\left(i\right)}}.$$

(17)

2.4. Problem Formulation

By collaboratively optimizing resource allocation, UAV-BS movement, and jammer transmission power, the processing delay of the system and the correct detection rate of the warden can be minimized, thereby ensuring the effective utilization of limited computing resources. The specific problems are expressed as follows:

$$\begin{array}{cc}\hfill \underset{\left\{{\alpha }_k\left(i\right),L\left(i\right),R_k\left(i\right),{\xi }^{*}\left(i\right)\right\}}{min}(& \sum _{i=1}^I\sum _{k=1}^K{\alpha }_k\left(i\right)max(t_k^{local}\left(i\right),t_k^{tr}\left(i\right)+t_b^{com}\left(i\right))+(1-{\xi }^{*}\left(i\right)))\hfill \end{array}$$

(18a)

$$\begin{array}{cc}\hfill s.t.& {\alpha }_k\left(i\right)\in \left\{0,1\right\},\forall i\in \mathcal{T},k\in \mathcal{K},\hfill \end{array}$$

(18b)

$$\begin{array}{cc}\hfill & \sum _{k=1}^K{\alpha }_k\left(i\right)=1,\forall i,\hfill \end{array}$$

(18c)

$$\begin{array}{cc}\hfill & 0\le R_k\left(i\right)\le 1,\forall i,k,\hfill \end{array}$$

(18d)

$$\begin{array}{cc}\hfill & {\xi }^{*}\left(i\right)\ge 1-ϵ,\forall i,\hfill \end{array}$$

(18e)

$$\begin{array}{cc}\hfill & L\left(i\right)\in \mathbb{R},\forall i,\hfill \end{array}$$

(18f)

$$\begin{array}{cc}\hfill & 0\le v\left(i\right)\le v_{max},\forall i,\hfill \end{array}$$

(18g)

$$\begin{array}{cc}\hfill & \sum _{i=1}^I(E_{fly}\left(i\right)+E_{BS}^{com}\left(i\right))\le E_{BS},\forall k,\hfill \end{array}$$

(18h)

$$\begin{array}{cc}\hfill & 0\le P_J\left(i\right)\le P_J^{max},\forall i,\hfill \end{array}$$

(18i)

$$\begin{array}{cc}\hfill & \sum _{i=1}^I\sum _{k=1}^K{\alpha }_k\left(i\right)D_k\left(i\right)=D,\hfill \end{array}$$

(18j)

where constraints (18b) and (18c) make sure that the UAV-BS serves only one UE per time slot, and constraint (18d) specifies a scope of values for the offload portion of the computation task. In constraint (18e), $ϵ$ is the permitted correct detection rate of the warden and is typically set to a small value. In order to ensure covert transmission from UE to UAV-BS, the ${\xi }^{*}\left(i\right)$ must exceed $(1-ϵ)$. Furthermore, the movement area of the UEs, the flight speed of the UAV-BS, and the transmit power of the jammer are limited by constraints (18f), (18g), and (18i). Simultaneously, constraints (18h) and (18j) indicate that the energy consumption of the UAV-BS within the relevant duration does not surpass the battery limit and all computational tasks must be accomplished within the specified time frame.

3. PER-DDPG-Based Task Offloading Optimization

Equation (18) belongs to the class of non-convex problems of mixed-integer non-linear programs (MINLPs). Because of the complex and variable environment during the offloading decision process, it needs to support a continuous action space, resulting in an excessively large state space dimension, which means that the traditional optimization methods struggle to address these challenges. Therefore, we transform it into a Markov Decision Process (MDP) model and then solve it by applying DRL techniques, with the objective of lowering the total system delay and the correct detection rate of the warden through a dynamic decision-making approach. Moreover, the DDPG algorithm is refined for the purpose of cumulative reward maximization.

3.1. Construction of MDP

The MDP is represented by a quintuple $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma )$ consisting of the state space $\mathcal{S}$, the action space $\mathcal{A}$, the state transition probability $\mathcal{P}$, the reward function $\mathcal{R}$, and the discount factor $\gamma$. Continuous time is discretized into multiple time moments i, with each state s stored in the set of states $\mathcal{S}$ and each action a stored in the set of actions $\mathcal{A}$. The variable r denotes the reward at the current moment. By observing the environment and experimenting with different actions, the agent trains on maximizing the reward of the system until it identifies the action that yields the highest reward.

3.1.1. State Space

The state space of our UAV-assisted covert MEC system is co-determined by K UEs, the UAV-BS, and the jammer, denoted as follows:

$$\begin{array}{cc}\hfill {\mathcal{S}}_i=& (E_{BS}\left(i\right),L^{BS}\left(i\right),L^W\left(i\right),L_1^{UE}\left(i\right),\dots ,L_k^{UE}\left(i\right),E_J\left(i\right),D_{remain}\left(i\right),D_1\left(i\right),\dots ,D_k\left(i\right)),\hfill \end{array}$$

(19)

where $E_{BS}\left(i\right)$ and $E_J\left(i\right)$ represent the remaining UAV-BS and jammer batteries, respectively, and $D_{remain}\left(i\right)$ denotes the amount of computational tasks remaining for the entire period.

3.1.2. Action Space

The agent selects an action based on the detected environment and the system’s current state. The action space includes the selection of the UE k to be served, the determination of the flight speed and angle of the UAV-BS, the task offload rate, and the transmit power of the jammer, expressed as follows:

$${\mathcal{A}}_i=(k\left(i\right),\beta \left(i\right),v\left(i\right),R_k\left(i\right),P_J\left(i\right)),$$

(20)

where $k\left(i\right)$ denotes that the UAB-BS serves UE k at time slot i. $\beta \left(i\right)$ and $v\left(i\right)$ represent the angle of flight and the velocity of the UAV-BS, respectively. $R_k\left(i\right)$ indicates the task offloading ratio of UE k, while $P_J\left(i\right)$ represents the jamming power of jammer. It is worth noting that $k\left(i\right)$ is a discrete variable, whereas the other variables are continuous.

3.1.3. Reward Function

Since the behavior of the agent depends on the rewards it receives, selecting an appropriate reward function is essential. As defined in (18a), this paper aims to maximizing the reward while keeping the system’s processing delay and the warden’s detection accuracy at a minimum. Hence, the reward function is formulated as follows:

$$\begin{array}{cc}\hfill {\mathcal{R}}_i& =r(s_i,a_i)=-\sum _{i=1}^I\left[{\omega }_1(t_k^{tr}\left(i\right)+t_{BS}^{com}\left(i\right))+{\omega }_2(1-{\xi }^{*}\left(i\right))\right],\hfill \end{array}$$

(21)

where $(t_k^{tr}\left(i\right)+t_{BS}^{com}\left(i\right))$ is the processing delay of the system, and $(1-{\xi }^{*}\left(i\right))$ is the maximum correct detection rate of the warden.

Due to the highly continuous nature of the state and action space in the system, obtaining an accurate state transition probability matrix becomes challenging. Therefore, to explore the best policy in the presence of uncertain state transition probabilities, a DDPG-based algorithm is employed.

3.2. PER-DDPG-Based Solution

The DDPG algorithm is a type of deep reinforcement learning designed for the policy space in continuous time, based on the actor–critic (AC) framework [30]. DDPG employs two distinct Deep Neural Networks (DNNs): the actor network $\mu \left(s\right|{\theta }^{\mu })$, which describes the function of the policy, and the critic network $\mathcal{Q}(s,a|{\theta }^{\mathcal{Q}})$, which describes the function of the Q-value. Furthermore, to enhance learning stability, each network has a corresponding target network with the same structure: the actor target network ${\mu }^{\prime }$, parameterized by ${\theta }^{{\mu }^{\prime }}$; and the critic target network ${\mathcal{Q}}^{\prime }$, with parameter ${\theta }^{{\mathcal{Q}}^{\prime }}$.

The critic network $\mathcal{Q}(s,a|{\theta }^{\mathcal{Q}})$ is trained to minimize the loss function $L\left({\theta }^{\mathcal{Q}}\right)$, which quantifies the error between the predicted Q-value and the target value $y_i$, ensuring accurate value estimation for policy optimization:

$$\begin{array}{cc}\hfill L\left({\theta }^{\mathcal{Q}}\right)& ={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[{(y_i-\mathcal{Q}(s_i,a_i|{\theta }^{\mathcal{Q}}))}^2\right],\hfill \end{array}$$

(22a)

$$\begin{array}{cc}\hfill y_i& =r_i+\gamma {\mathcal{Q}}^{\prime }(s_{i+1},\mu \left(s_{i+1}\right)|{\theta }^{{\mathcal{Q}}^{\prime }}).\hfill \end{array}$$

(22b)

The policy gradient ${\nabla }_{{\theta }^{\mu }}J$ is updated using the chain rule [31], which adjusts the actor network parameters ${\theta }^{\mu }$ to maximize the expected Q-value:

$$\begin{array}{cc}\hfill {\nabla }_{{\theta }^{\mu }}J& \approx {\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[{\nabla }_{{\theta }^{\mu }}\mathcal{Q}(s,a|{\theta }^{\mathcal{Q}}){|}_{s=s_i,a=\mu \left(s_i\right|{\theta }^{\mu })}\right]\hfill \\ \hfill & ={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[{\nabla }_a\mathcal{Q}(s,a|{\theta }^{\mathcal{Q}}){{|}_{s=s_i,a=\mu \left(s_i\right|{\theta }^{\mu })}{\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu })|}_{s=s_i}\right].\hfill \end{array}$$

(23)

The whole training process of DDPG can be summarized as shown in Figure 2. Firstly, after the previous training step, the actor network $\mu$ outputs $\mu \left(s_i\right)$. Since DDPG is an off-policy algorithm, we can adequately explore the set of states by extending the action space. This can be achieved by including a behavioral bias $n_i$ to produce an action $a_i=\mu \left(s_i\right)+n_i$, where $n_i$ follows a Gaussian distribution $n_i\sim \mathbb{N}({\mu }_e,{\sigma }_{e,i}^2)$, with ${\mu }_e$ being the mean along with ${\sigma }_{e,i}$ being the standard deviation. The agent observes the subsequent state $s_{i+1}$ and the associated reward $r_i$ after executing $a_i$ in the environment. The experience replay buffer will then contain the transition $(s_i,a_i,r_i,s_{i+1})$. Secondly, the algorithm chooses N transitions $(s_j,a_j,r_j,s_{j+1})$ at random out of the buffer to construct a minibatch, used to update both critic and actor networks. By using the minibatch, the action ${\mu }^{\prime }\left(s_{j+1}\right)$ is generated by the actor target network ${\mu }^{\prime }$ and passed to the critic target network ${\mathcal{Q}}^{\prime }$. And the critic network can use minibatch and ${\mu }^{\prime }$ to compute the target value $y_j$ according to (22b). Thirdly, an optimizer will update the critic network $\mathcal{Q}$ to reduce the loss function. Then, actor network $\mu$ passes the action of the minibatch $a=\mu \left(s_j\right)$ to critic network to calculate the gradient of the action a using ${\nabla }_a\mathcal{Q}(s,a|{\theta }^{\mathcal{Q}}){|}_{s=s_j,a=\mu \left(s_j\right)}$. Parameter ${\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu }){|}_{s=s_j}$ can be calculated using its respective optimizer. These two gradients are then used to update the actor network:

$${\nabla }_{{\theta }^{\mu }}J\approx {\displaystyle \frac{1}{N}}\sum _{j=1}^N\left[{\nabla }_a\mathcal{Q}(s,a|{\theta }^{\mathcal{Q}}){{|}_{s=s_j,a=\mu \left(s_j\right|{\theta }^{\mu })}{\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu })|}_{s=s_j}\right].$$

(24)

At last, the target network of the actor and the target network of the critic will be updated softly by the agent with a small constant $\iota$:

$$\begin{array}{cc}\hfill & {\theta }^{{\mathcal{Q}}^{\prime }}\leftarrow {\theta }^{\mathcal{Q}}+(1-\iota ){\theta }^{{\mathcal{Q}}^{\prime }},\hfill \\ \hfill & {\theta }^{{\mu }^{\prime }}\leftarrow {\theta }^{\mu }+(1-\iota ){\theta }^{{\mu }^{\prime }}.\hfill \end{array}$$

(25)

The PER mechanism improves an agent’s learning efficiency by prioritizing important samples with larger TD-errors during the training phase [32]. A greater TD-error suggests that the sample is more significant for the agent’s learning. In the training process, however, samples with large TD-errors are seldom encountered. By elevating the likelihood of selecting these samples with high TD-errors, we can enhance the overall sample utilization, thereby refining the efficacy of the learning approach. The TD-error $T_x$ quantifies the difference between the predicted Q-value and the target value, serving as a measure of the sample’s importance for learning:

$$T_x\triangleq r_i+\gamma {\mathcal{Q}}^{\prime }(s_{i+1},\mu \left(s_{i+1}\right)|{\theta }^{{\mathcal{Q}}^{\prime }})-\mathcal{Q}(s_i,a_i|{\theta }^{\mathcal{Q}}).$$

(26)

To address the limitations of relying solely on TD-error, including the reduction in diversity and the issue of bias, we introduced two methods: stochastic sampling and importance sampling. Stochastic sampling guarantees that every sample from the experience replay buffer has a chance of being selected, which in turn improves the diversity of the training data. Specifically, the priority $p\left(x\right)$ of the sample x determines the sampling probability of each transition, with higher priorities assigned to transitions that have larger TD-errors:

$$\begin{array}{cc}\hfill p\left(x\right)& ={\displaystyle \frac{p_x^{\alpha }}{{\sum }_k{p}_k^{\alpha }}},\hfill \end{array}$$

(27a)

$$\begin{array}{cc}\hfill p_x& =|T_x|+\epsilon ,\hfill \end{array}$$

(27b)

where $\alpha$ determines the level of prioritization applied, $p_x$ represents the priority of sample x, and $\epsilon$ is a tiny positive value.

The bias problem refers to the tendency of the agent to update empirical samples with high temporal-difference (TD) error, which alters the original probability distribution and introduces errors into the model. As a result, this may prevent the model from converging during neural network training. Therefore, to correct the bias introduced by prioritized sampling, importance sampling weights $w_{IS\_x}$ are incorporated:

$$w_{IS\_x}={({\displaystyle \frac{1}{N}}·{\displaystyle \frac{1}{p\left(x\right)}})}^{\varrho },$$

(28)

with N denoting the overall number of samples, and the exponent $\varrho$ is typically modified to one from its starting point.

In traditional DDPG algorithms, uniform sampling from the experience replay buffer often results in inefficient use of samples, as all samples are treated equally regardless of their importance to the learning process. This can slow down convergence and reduce sample efficiency. To address this limitation, we propose an improved PER-DDPG algorithm that integrates the DDPG algorithm with the PER mechanism. The PER mechanism assigns higher sampling probabilities to transitions with larger TD errors, as these transitions are typically more informative for learning. By focusing on the most critical samples, PER-DDPG significantly improves sample efficiency and accelerates convergence [33,34]. The loss function (22a) is adjusted to incorporate importance sampling weights $w_{IS\_x}$, which balance the prioritization of high TD-error transitions and the stability of the learning process:

$$L\left({\theta }^{\mathcal{Q}}\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[w_{IS\_x}{(y_i-\mathcal{Q}(s_i,a_i|{\theta }^{\mathcal{Q}}))}^2\right].$$

(29)

The training process of the proposed PER-DDPG method is shown in Algorithm 1.

Algorithm 1 PER-DDPG-based task offloading algorithm.

1: Initialize $\mu \left(s\right|{\theta }^{\mu })$ and $Q(s,a|{\theta }^Q)$ with the parameters ${\theta }^{\mu }$ and ${\theta }^Q$, respectively. 2: Initialize ${\theta }^{{\mu }^{\prime }}={\theta }^{\mu }$, and ${\theta }^{Q^{\prime }}={\theta }^Q$, respectively. 3: Initialize the experience replay buffer M. 4: for each episode $e=1,2,\dots ,E$ do 5:    Reset system simulation parameters and obtain the first observation state $s_1$. 6:    for $i=1,2,\dots ,I$ do 7:      Normalizes $s_i$ to ${\widehat{s}}_i$. 8:      Obtain the action $a_i$ using the actor network ${\theta }^{\mu }$ with the noise $n_i$. 9:      Execute $a_i$, obtain the reward $r_i$ from Equation (21) and observe the next state $s_{i+1}$. 10:      Normalizes $s_{i+1}$ to ${\widehat{s}}_{i+1}$. 11:      if the buffer M is not full then 12:         Save the transition $(\widehat{s_i},a_i,r_i,{\widehat{s}}_{i+1})$ into M and calculate the priority $p_i$ of transition $(\widehat{s_i},a_i,r_i,{\widehat{s}}_{i+1})$ with (27b). 13:      else 14:         Replace a transition with the lowest priority in buffer M with $(\widehat{s_i},a_i,r_i,{\widehat{s}}_{i+1})$, and set $p_i={max}_{j<i}{p}_j$. 15:         for $j=1,2,\dots ,N$ do 16:           Sample the jth transition from M with probability $p\left(j\right)$ using (27a) and (27b). 17:           Update the priority $p_j$ of transition $(\widehat{s_j},a_j,r_j,{\widehat{s}}_{j+1})$. 18:         end for 19:         Update ${\theta }^{\mathcal{Q}}$ of critic net by minimizing the loss (29). 20:         Update ${\theta }^{\mu }$ of actor net with the sampled policy gradient (24). 21:         Soft update critic target net and actor target net (25). 22:      end if 23:    end for 24: end for

4. Simulation Results

This section uses numerical simulations to confirm the computational offloading performance of the UAV-assisted MEC system operating under covert communication conditions based on the improved PER-DDPG approach. The experiments were executed on a computer fitted with a 2.20 GHz Intel Core i9-13900HX CPU, a NVIDIA GeForce RTX 4060 graphics card, 16 GB RAM and a Windows operating system.

4.1. Parameters Setting

We set the ground area in the system as a two-dimensional square with a size of $100\times 100 {\mathrm{m}}^2$, where the UEs and a warden were randomly distributed. In addition, the

Table 1. Main parameters and assumptions.

Parameter	Descriptions	Value
K	Number of UEs	4
T	Entire time period	320 s
I	Time slots	40
$h_1$	UAV-BS flight height	100 m
$h_2$	Jammer flight height	110 m
$M_{UAV}$	UAV weight	9.65 kg
$v_{max}$	Maximum flight speed	50 m/s
$t_{fly}$	Flight time	1 s
$f_c$	Carrier frequency	2GHz
${\mu }_{LoS}$	Path loss factor for LoS links	3 dB
${\mu }_{NLoS}$	Path loss factor for NLoS links	23 dB
B	Communications bandwidth	1 MHz
${\sigma }^2$	Variance of AWGN	−100 dBm
$ϵ$	Allowed correct detection rate	0.05
$P_k$	UE transmit power	0.1 W
$E_{BS}$	UAV-BS battery	500 kJ
s	Required CPU cycles per bit	1000 cycles/bit
$f_{UE}$	UE computing capability	0.2 GHz
$f_{UAV}$	UAV-BS computing capability	1.2 GHz

shows the other parameters involved in the experiment. These values were incorporated into the algorithm for multiple runs, and performance comparisons were conducted using the obtained average values.

4.2. Convergence Analysis

To find the optimal values of the key hyperparameters involved in the algorithm comparison, a series of experiments were firstly carried out. Figure 3 shows the convergence performance of the PER-DDPG algorithm with various learning rates. Assuming the actor network and critic network have different learning rates, it is clear that PER-DDPG attains optimal performance with learning rates of ${\alpha }_{Actor}=0.001$ and ${\alpha }_{Critic}=0.002$. Conversely, due to their large update steps, a local optimal convergence is achieved with higher learning rates of ${\alpha }_{Actor}=0.1$ and ${\alpha }_{Critic}=0.2$. Furthermore, convergence slows when the learning rates decrease to ${\alpha }_{Actor}=0.00001$ and ${\alpha }_{Critic}=0.00002$ because DNN updates are more sluggish and require more iterations.

The discount factor $\gamma$ is another parameter that influences the convergence of the proposed PER-DDPG algorithm. When $\gamma$ is too large, the model may place excessive emphasis on long-term rewards while neglecting short-term gains, leading to slower convergence and diminished generalization ability. Conversely, if $\gamma$ is too small, the model may overly focus on short-term rewards, resulting in short-sighted behavior and unstable learning. As illustrated in Figure 4, the PER-DDPG algorithm achieves optimal performance when $\gamma =0.01$. Therefore, in subsequent experiments, the discount factor $\gamma$ will be kept at 0.01.

Figure 5 shows the effect of the exploration parameter ${\sigma }_e$ on the PER-DDPG algorithm. When ${\sigma }_e=0.001$ and ${\sigma }_e=0.01$, PER-DDPG falls into a local optimum. As ${\sigma }_e$ increases, the space of the random noise distribution generated and the action space that the agent can explore become larger, so the PER-DDPG algorithm performs best when ${\sigma }_e=0.1$.

Figure 6 evaluates how the PER-DDPG algorithm performs without the PER mechanism or state normalization. On the one hand, without the PER module, the convergence speed of the training algorithm is slowed down, which leads to an inability to find the best unloading policy. On the other hand, the absence of the deep neural network (DNN) produces large outputs due to large values in the state space that are not normalized, whereupon the algorithm requires more training iterations to converge and the training process becomes highly unstable.

4.3. Performance Comparison

In order to verify the superiority of the proposed PER-DDPG algorithm, three task offloading algorithms are selected for simulation and comparison under the same application scenarios. The first is the DQN algorithm, which addresses discrete action problems; the second is the DDPG algorithm, which handles a continuous action space; and the third is the PER-DDPG algorithm. Figure 7 illustrates the variation in delay and the correct detection rate of the warden over different algorithms with the number of training sessions, when the computational task size is 100 Mb. As shown in the figure, while all three algorithms are able to converge eventually, DQN cannot accurately determine the best offloading strategy. This limitation arises from DQN’s inability to effectively explore the space between discrete actions. In contrast, the other two algorithms, which are capable of exploring a continuous action space, can more effectively identify the optimal strategy. However, the PER-DDPG algorithm outperforms the DDPG algorithm when it comes to convergence. The addition of the PER mechanism in DDPG enables more frequent replay of both very successful and very unsuccessful experiences, and these experience samples carry higher learning value, resulting in faster convergence and improved performance. After convergence, the average reward obtained by PER-DDPG was −49.32, while the average reward for DDPG was −58.87, and for DQN, it was −106.1. Compared to DDPG, the average reward obtained by PER-DDPG increased by more than 16.22%, and when compared to DQN, the average reward obtained by PER-DDPG also increased by more than 53.52%. Additionally, the convergence speed of PER-DDPG was the fastest among the three algorithms.

The detection probabilities of the warden obtained by the three algorithms are illustrated in Figure 8. After convergence, we observe that the PER-DDPG and DDPG algorithms achieve warden average detection error rates of 98.24% and 96.04%, respectively, both exceeding the 95% threshold. In contrast, the DQN algorithm shows a lower average detection error rate of 93.55%, which does not meet the 95% threshold. Additionally, consistent with Figure 7, PER-DDPG converges the fastest among the three algorithms.

Figure 9 illustrates the changes in various evaluation indicators of the three algorithms as the $f_{UE}$ varies. As can be seen in Figure 9a, we can observe the differences in the convergence performance among the three algorithms at different values of $f_{UE}$. Previous experiments have demonstrated that the proposed PER-DDPG algorithm’s convergence performance is superior to DDPG and DQN at $f_{UE}$ = 0.2 GHz, while all three algorithms gradually degrade. Notably, as $f_{UE}$ increases, the reward values for all three algorithms also rise, although the overall downward trend remains unchanged. This phenomenon can be attributed to the ability of PER-DDPG and DDPG to explore continuous action sets, a capability that DQN lacks. However, it is evident that the performance gap between the three algorithms is narrowing, with DQN showing the largest increase. Figure 9b provides further insight into this trend. As $f_{UE}$ increases, the task offloading rate for all three algorithms decreases. This decline is attributed to the enhanced computing power of the UEs, which leads to a greater tendency to perform task calculations locally. Consequently, this reduces the delay difference between local task execution and offloading tasks to the UAV-BS, ultimately resulting in a decrease in the overall system delay. Among the three algorithms, DQN exhibits the smallest change in offloading rate. This is due to DQN’s difficulty in navigating discrete action spaces, preventing it from identifying better offloading solutions. The primary factor influencing DQN’s final convergence performance in Figure 9a is the increase in $f_{UE}$, which contributes to an overall increase in reward value, even if DQN’s performance is suboptimal. Additionally, Figure 9c shows that the detection error rate of the warden remains relatively stable as $f_{UE}$ increases. This stability is due to the fact that, although the proportion of tasks executed locally by the UE has risen, its transmission power has not changed, which does not significantly impact the warden’s detection behavior.

Figure 10 and Figure 11 show the performance changes of the three algorithms under different computational task sizes and the number of UEs, respectively. The results show that the proposed PER-DDPG algorithm consistently outperforms DDPG and DQN in different scenarios. As the state space becomes more complex with the increase in tasks or the increase in UEs, DQN has difficulty in exploring and optimizing effectively, resulting in either a significant drop in reward value (for task size increases) or strong instability (for UE number increases). In contrast, PER-DDPG and DDPG, with their continuous action space design and more efficient exploration strategies, can better adapt to the increasing complexity and maintain more stable performance. Both figures highlight a common limitation of DQN: it cannot effectively handle the growth of state dimension. Whether due to the increase of computational task size or the increase of UE number, DQN’s discrete action space and low sample efficiency make it difficult to explore and optimize in high-dimensional environments. Compared to the more stable performance of PER-DDPG and DDPG, DQN’s reward values fluctuate or drop significantly, which fully illustrates this limitation. In addition, the PER mechanism of PER-DDPG further enhances its ability to adapt to different task scales and UE numbers, demonstrating excellent performance and stability.

In summary, the proposed PER-DDPG algorithm outperforms the other two algorithms. Consequently, after integrating optimization of resource allocation, UAV-BS movement, and jammer transmit power, this algorithm achieves a relatively small maximum processing delay for the system and a low correct detection rate by the warden. At the same time, while deploying the proposed system with current UAV and MEC technologies is feasible, it still faces challenges such as limited UAV battery life, computational resource constraints, and the need for robust communication protocols in dynamic environments. Additionally, while covert communication enhances privacy and security by preventing eavesdropping, it also raises ethical concerns regarding potential misuse. Advances in energy-efficient hardware, lightweight algorithms, and adaptive communication strategies, along with strict regulatory frameworks, will be critical for the real-world deployment of such systems.

5. Conclusions

In this research, we investigated a strategy for task offloading in UAV-assisted covert edge computing. Firstly, we established a UAV-assisted MEC task offloading model that adheres to covert communication requirements. An optimization problem was formulated with the objective to minimize system delay and detection accuracy of the warden by jointly optimizing resource allocation, UAV motion, and jammer signal strength. Secondly, we introduced a novel PER-DDPG algorithm, which enables UAVs to efficiently perform task offloading calculations while ensuring communication security. Finally, we perform a set of experiments to validate the feasibility of the proposed algorithm and demonstrate its superior performance in comparison to several benchmark algorithms. For future work, we plan to extend our approach to more complex and dynamic scenarios, such as multi-UAV systems with collaborative task offloading, the incorporation of adversarial learning techniques to address intelligent adversaries, and the integration of advanced MEC frameworks, including federated learning and blockchain technology, to enhance security, privacy, and scalability in UAV-assisted covert edge computing.