Task Offloading Scheme Based on Proximal Policy Optimization Algorithm

Abstract

The rapid development of mobile Internet technology has made users’ requirements for quality of service (QoS) continuously improve. The task unloading process of mobile edge computing has the problem that it is impossible to balance delay and energy consumption for task unloading under the condition of fluctuating network bandwidth. To address this issue, this paper proposes a task offloading scheme based on the Proximal Policy Optimization (PPO) algorithm. On the basis of traditional cloud edge collaborative architecture, the collaborative computing mechanism between edge node devices is further integrated, and the concept of service caching is introduced to reduce duplicate data transmission, reduce communication latency and network load, and improve overall system performance. Firstly, this article constructs an energy efficiency function with a certain weight ratio of energy consumption and latency as the core optimization objective. Then, the task offloading process of mobile terminal devices is modeled as a Markov Decision Process (MDP). Finally, the deep reinforcement learning PPO algorithm is used for training and learning, and the model is solved. The simulation results show that the proposed scheme has significant advantages in reducing energy consumption and latency compared to the comparative scheme.

1. Introduction

With the rapid development of mobile Internet and intelligent devices, mobile communication technology has become a kind of infrastructure closely related to people’s daily life, which is widely used in social networking, online shopping, entertainment, and many other fields. At the same time, the data and computation required by mobile terminal devices are also showing an exponential growth trend, which has brought tremendous pressure to mobile networks. The limited computing and communication capabilities make it impossible for mobile terminal devices to meet the demands of massive computing tasks. Mobile Cloud Computing (MCC) [1], as an emerging computing model utilizing the high-performance computing and storage capabilities of the cloud to provide computer resources and storage services for mobile terminal devices, has become an effective solution to the problem. However, in recent years, applications such as augmented reality and autonomous driving that require a large amount of computation and bandwidth have emerged. The traditional mobile cloud computing model has significant transmission delays during task processing, which cannot meet the latency-sensitive requirements of real-time tasks. The main reason is that the distance between mobile terminal devices and central cloud and edge node devices is relatively long [2].

In order to solve the above problems, the new concept of mobile edge computing (MEC) was proposed [3,4]. This computing method reduces the distance between users and remote ends, reduces task processing delay, and improves the efficiency of task processing by distributing the original centralized computing resources and storage resources on edge node devices close to mobile terminal users. Through this approach, the problem of significant transmission latency in traditional cloud computing models is solved, which has a significant impact on improving user experience. MEC technology allocates computing resources reasonably based on dynamic changes in the network, ensuring service quality while meeting the different latency and energy consumption requirements of different user tasks [5].

Despite the rapid development of MEC technology in multiple fields, MEC still faces many challenges as network size and task complexity continue to increase. When faced with complex computing tasks or excessive workload, it is still necessary to offload the tasks to the central cloud for processing. Therefore, it is necessary to coordinate the processing of computing tasks among the central cloud, edge node devices, and mobile terminal devices [6]. The cloud-side collaboration mechanism can give full play to the powerful computing power of cloud computing and the low-latency advantage of edge computing, but it also needs to consider the limited resources and heterogeneous services of mobile edge servers [7]. Due to the limited computing and storage resources of edge node devices, only some services can be cached, and the caching strategy of the services can affect users’ decisions on computing offloading. In addition, uploading pending tasks to the cloud for processing carries the risk of private data leakage during channel transmission [8]. Therefore, optimizing the task offloading strategy under the cloud edge collaboration mechanism while ensuring user data privacy, as well as achieving the comprehensive optimization of latency and energy consumption as much as possible, is a challenging and significant problem.

The concept of task offloading is very important in mobile edge computing networks. It reduces the latency of processing tasks, improves response time, and enhances user experience, and it plays an important role in achieving load balancing, reducing device energy consumption, and improving resource utilization. The main contributions of this article include the following:

(1)

Building a complete mobile edge computing offload network architecture: Based on the traditional cloud edge collaborative network architecture, we integrate the edge collaborative mechanism, introduce the concept of service cache, reduce duplicate data transmission, reduce communication delay and network load, and achieve efficient data transmission and processing.

(2)

Optimizing system energy consumption and delay balance: This article models task offloading and resource allocation as Markov Decision Processes and uses deep reinforcement learning algorithms to solve these problems through near-end policy optimization, thereby achieving the optimal overall energy efficiency of the system.

2. Related Work

One of the main goals of MEC networks is to offload computing tasks to network edge servers for processing, reduce the computing load on terminal devices, and thus improve application performance and system efficiency. Edge computing devices have certain computing, storage, and communication capabilities, which can provide effective computing support for mobile devices. Therefore, task offloading has become a core component of MEC. Task offloading can alleviate the computing pressure on terminal devices, reduce network load, optimize bandwidth utilization, and provide users with low-latency computing services [9]. The goals of task offloading decisions usually have three aspects: reducing latency, reducing energy consumption, and weighing the pros and cons between latency and energy consumption [10].

In terms of reducing latency, Reference [11] solved the problem of the low processing capacity of edge servers by offloading some tasks of certain mobile terminal users to the central cloud for computing. This effectively reduced the processing delay of tasks, thus avoiding network congestion. Reference [12] attached great importance to the peak aggregation of different edge servers, extensively explored the trade-off between communication delay and computation delay in task offloading, and proposed a hierarchical task offloading and resource optimization mechanism based on tree hierarchy. Reasonably scheduling tasks based on the processing capacity of each server significantly improved network resource utilization. Reference [13] studied the task offloading scheme for power terminals with limited computer resources in the smart grid environment. This scheme used an enhanced adaptive genetic algorithm to solve the optimal task offloading strategy. Reference [14] proposed the concept of end-to-end collaboration based on cloud–edge collaboration. Although the experimental results showed that the total delay of task execution could be reduced, the computation of the task scheduling (CTS) algorithm in this article was relatively complex. Reference [15] used Markov decision theory to solve the problem of minimizing delay. Reference [16] established a WPMECN network model based on deep reinforcement learning, which has significant advantages in CPU processing latency.

In terms of reducing energy consumption, Reference [17] proposed a home energy management system based on artificial neural networks, which significantly reduces household electricity costs and carbon dioxide emissions by integrating solar photovoltaic and maximum power point tracking (MPPT) technologies. This research showed how to optimize household energy use through intelligent algorithms and provides a useful reference for energy management in mobile edge computing. In the MEC environment, similar intelligent energy management strategies can be used to optimize the energy consumption of edge devices. Reference [18] proposed a new adaptive particle swarm optimization algorithm, which has better energy-saving effects. Reference [19] focused on the issues of power control and computing resource allocation and proposed the E2PC (Energy-saving Power Control) algorithm. Although this algorithm has significant advantages in reducing energy consumption, it is difficult to apply in practice.

In terms of balancing delay and energy consumption, Reference [20] proposed a joint optimization strategy to achieve the dual objectives of energy consumption and delay optimization, which is of great significance for promoting energy efficiency improvement in MEC applications. Reference [21] proposed an optimization framework for computing offloading and resource allocation to improve network computing efficiency and save energy. Reference [22] investigated the offloading decision problem of MEC networks in multi-layer 6G network environments. The service needs of users, network status, and computing resources were fully considered to optimize network performance and user experience. Reference [23] provided a framework with an alternating-direction multiplier method to optimize the decision-making process of task offloading in a way that optimizes resource utilization and system performance. By synchronously offloading computing tasks at three levels, terminal devices, edge servers, and cloud servers, this framework minimizes the computational burden on the network, reduces network latency, and improves energy efficiency to the greatest extent possible.

3. Preliminary

3.1. Markov Decision Process Theory

The Markov Decision Process (MDP) is constructed based on Markov theory and is used for optimal decision-making in dynamic environments [24]. The core is Markovian, which means that the future state is only determined by the current state and has nothing to do with history. The MDP can use methods such as dynamic programming to find the optimal strategy, thereby maximizing the cumulative reward.

MDP is generally represented by a five-part tuple (S, A, P, R, $\gamma$):

S represents the state space; it represents the set of all possible states in the system.

A represents the action space; it defines the set of actions that can be executed in each state.

P represents the state transition function; it refers to the probability that the system will transition to the next state given a specific current state, d, and action.

R stands for the reward function; it is used to measure the immediate reward or punishment obtained after performing a specific action in a certain state.

$\gamma$ denotes the discount factor, which assesses the importance of future rewards to current decisions, and generally takes values within [0, 1].

In practical applications, the MDP is widely used in fields such as artificial intelligence decision-making, resource allocation, and path planning, providing strong modeling tools. Common solutions include value iteration, strategy iteration, and deep reinforcement learning (DRL). During the interaction with the environment, the system observes the state, selects actions, obtains rewards, and continuously optimizes its decision-making strategy.

The decision-making process of the MDP has discreteness. In each time step, the system will select an action based on the current state. The environment will transition from the current state to the next state based on the probability of state transition and then provide feedback rewards. This process will continue until it reaches the termination state or the task is completed.

3.2. Deep Reinforcement Learning Theory

Reinforcement learning is a machine learning method that is related to deep learning, as shown in Figure 1. It learns by interacting with the environment and does not require a large amount of training data [25]. The agent decides actions based on reward signals and continuously explores and learns to maximize accumulated rewards. Deep reinforcement learning (DRL) combines the techniques of deep learning and reinforcement learning [26], using deep neural networks to represent decision strategies and training them with reinforcement learning algorithms. DRL can handle high-dimensional inputs and outputs and adapt to complex environments. Common methods include the Deep Q-Network (DQN) [27], DDPG [28], and Proximal Policy Optimization (PPO) [29].

4. Model System

Edge computing can reduce transmission delay, enhance privacy security, and reduce cloud load by sinking computing resources. However, limited by hardware resources and power consumption, edge devices make it difficult to independently complete computationally intensive tasks, and a single task offloading strategy cannot fully optimize computing efficiency, energy consumption, and resource utilization. Therefore, how to allocate computing tasks reasonably between the cloud and the edge and achieve a balance between computing efficiency, energy consumption control, and latency optimization has become a research hotspot.

This article proposes a task offloading scheme that utilizes the Proximal Policy Optimization (PPO) algorithm. The basic framework of this method is shown in Figure 2. In the traditional cloud edge collaboration architecture, the collaboration mechanism between edges is introduced to improve the utilization of computing resources by optimizing task offloading paths and resource allocation. Simultaneously, service caching technology is utilized to effectively reduce data transmission latency and network load.

4.1. Model

As shown in Figure 2, this chapter examines an edge computing task unloading network composed of multiple edge computing nodes and mobile terminal devices within their coverage areas. The network is connected to a cloud computing center to explore efficient task unloading strategies in a mobile edge computing environment. The network architecture is primarily divided into three layers: the Central Cloud Layer (CCL), the Edge Device Layer (EDL), and the Mobile Terminal Layer (MTL). The MTL consists of a large number of mobile terminal devices, including smartphones, tablets, Internet of Things (IoT) devices, and sensors, denoted as $\mathcal{N}=\{1,2,\dots ,N\}$, where n represents the device number, with $1\le n\le N$. These terminal devices interact with users and are responsible for generating computing tasks. The EDL comprises multiple edge computing nodes, denoted as $\mathcal{M}=\{1,2,\dots ,M\}$, where m represents the edge server number, with $1\le m\le M$. Due to the dense deployment of edge devices and overlapping coverage areas, end users located in overlapping areas choose the edge devices with the best channel conditions for association.

4.2. Service Cache Model

We assume that the edge computing nodes in this chapter require the local caching of corresponding services to provide services to mobile terminal users. However, due to the needs for data security and privacy protection, load balancing constraints, and the limitations of storage space on edge computing nodes, a single edge computing node cannot store all types of service caches. $\mathcal{S}$ is a set of service caches, which includes S different types of service cache resources. The data size of each cache resource is represented by $D_s$, where $s=\{1,2,\dots ,S\}$. The deployment status of service caches on edge computing nodes can be represented by the variable $\mathbf{C}={\left\{C_s\right\}}_{s=1}^S$, where $C_s$ is a Boolean variable. When the s-th type of service cache is deployed on the edge computing node, $C_s=1$; otherwise, $C_s=0$. Tasks generated by mobile terminal devices depend on a specific type of service cache, and the edge computing node can only correctly execute the corresponding computing tasks if it has pre-deployed the service cache resources required by the tasks. If the associated edge computing node of the mobile terminal device lacks the service cache resources needed for task execution, the task needs to be migrated to other edge computing nodes that have the required cache resources to execute. If the edge computing nodes also lack the necessary resources, the service cache must be downloaded from the central cloud.

4.3. Task Communication Model

In this article, edge node devices use Time Division Multiple Access (TDMA) to communicate with mobile terminal users. Edge node devices allocate different time slots for each associated mobile terminal user to transmit data, ensuring that only one user communicates on a specific channel at a time. Therefore, data transmission between different users will not conflict, effectively avoiding interference between channels and improving the communication efficiency and stability of the system.

(1)

Communication Model between Edge Nodes and Mobile Terminals

According to Shannon’s theorem, in an interference-free wireless communication calculation model, assuming that the channel transmission gain is g, and assuming that mobile terminal users do not move within a single time slot, then g is a constant value. The transmission power is $P_n^{tran,d}$, the noise power spectral density is $N_0$, and the wireless channel bandwidth is $B_t^m$. From this, the uplink channel transmission rate $r_{nm}$ from the terminal device n to the edge node m can be obtained as follows:

$$r_{nm}=B_t^m\times {log}_2\left(1+{\displaystyle \frac{P_n^{tran,d}\times g}{N_0}}\right),\forall n\in \mathcal{N},m\in \mathcal{M},t\in T$$

(1)

The transmission delay $T_{n,m}^{tran,d}$ for the terminal device n to offload a task of the data size $D_n$ to the edge node m is

$$T_{n,m}^{tran,d}={\displaystyle \frac{D_n}{r_{nm}}}={\displaystyle \frac{D_n}{B_t^m\times {log}_2\left(1+\frac{P_n^{tran,d}\times g}{N_0}\right)}},\forall n\in \mathcal{N},m\in \mathcal{M},t\in T$$

(2)

Given that the scale of task results is much smaller than that of the input data, and the uplink transmission rate from end users to edge devices is significantly lower than the downlink transmission rate from edge devices to end users, in the communication model between end users and edge devices, only the uplink is focused on, while ignoring the delay overhead generated when edge devices transmit task results back to end users.

(2)

Communication Model between Edge Nodes

Communication between multiple edge nodes is similar to the communication process between edge nodes and mobile terminal users, and the transmission also requires the use of a channel resource. This is represented as $r_{m_1{m}_2}$:

$$r_{m_1{m}_2}=B_t^m\times {log}_2\left(1+{\displaystyle \frac{P_{m_1}^{tran,e}\times g}{N_0}}\right),m\in \mathcal{M},t\in T$$

(3)

When an edge node, $m_1$, lacks the service cache type required for task execution, and it is necessary to offload the task to another edge node, $m_2$, for execution, the migration delay between $m_1$ and $m_2$ can be expressed as

$$T_{m_1,m_2}^{tran,e}={\displaystyle \frac{D_n}{r_{m_1{m}_2}}},\forall n\in \mathcal{N},m\in \mathcal{M},t\in T$$

(4)

(3)

Communication Model between Edge Nodes and the Central Cloud

When both the associated edge node $m_1$ and the cooperative edge node $m_2$ do not contain the service cache type required for task execution, it is necessary to download the required service cache resources from the central cloud. The transmission delay for the edge node m to download service cache from the central cloud through a return path is

$$T_m^{tran,e}={\displaystyle \frac{D_s}{W_m}},\forall s\in S,m\in \mathcal{M}$$

(5)

where $W_m$ represents the bandwidth of the return path.

4.4. Energy Consumption Model

(1)

Task Upload Energy Consumption

The energy consumption of mobile terminal devices uploading tasks to their associated edge computing nodes for processing is expressed as

$$E_{n,m}^{tran,d}=P_n^{tran,d}{T}_{n,m}^{tran,d},\forall n\in \mathcal{N},m\in \mathcal{M}$$

(6)

where $P_n^{tran,d}$ is the transmission power of the terminal device n.

(2)

Task Computing Energy Consumption

The computing delay for mobile terminal devices uploading tasks to edge computing nodes is

$$T_{n,m}^{comp,e}={\displaystyle \frac{D_n\times C_m}{f_m}},\forall n\in \mathcal{N},m\in \mathcal{M}$$

(7)

where $f_m$ represents the CPU frequency of the edge node.

The computing energy consumption for mobile terminal devices uploading tasks to edge computing nodes is

$$E_{n,m}^{comp,e}=P_m^{comp,e}{T}_{n,m}^{comp,e},\forall n\in \mathcal{N},m\in \mathcal{M}$$

(8)

where $P_m^{comp,e}$ represents the computing power of edge nodes.

(3)

Task Migration Energy Consumption

The migration energy consumption for offloading tasks between the associated edge node $m_1$ and the cooperative edge node $m_2$ is

$$E_{m_1,m_2}^{tran,e}=P_m^{tran,e}{T}_{m_1,m_2}^{tran,e},\forall m\in \mathcal{M}$$

(9)

(4)

Service Cache Transmission Energy Consumption

The transmission energy consumption for the edge node m to download service cache from the central cloud is

$$E_c^{tran,e}=P_c^{tran,c}{T}_m^{tran,e},\forall m\in \mathcal{M}$$

(10)

4.5. Task Offloading Model

The time period T for the task offloading of terminal devices is divided into several small time segments, t, assuming that the user’s location does not change during each small time segment, t. In the time segment t, the service request of the terminal device n is

$$j_n\left(t\right)=(F_n\left(t\right),S_n\left(t\right),D_n\left(t\right)),t\in T$$

(11)

These, respectively, represent the number of CPU cycles required for user-requested tasks, the service cache type on which task execution depends, and the data volume of user tasks.

As shown in Figure 3, within each time slot, each computation task generated by the terminal user is offloaded to the edge node device with the best channel quality associated with it. If the edge device associated with the terminal user lacks the service cache required for task execution, then the edge device will further offload the computation task to other collaborative edge node devices or download the required service cache from the central cloud.

4.5.1. Associated Edge Nodes Perform Tasks

If the mobile terminal user offloads the task to its associated edge node and this edge node has service cache, then the execution delay consists of two parts: first, the transmission delay for the mobile terminal user to send the task to the edge node; second, the computation delay for the edge node to execute the task.

$$T_{ch1}=T_{n,m}^{tran,d}+T_{n,m}^{comp,e}$$

(12)

The energy consumption corresponding to the two operation delays is

$$E_{ch1}=E_{n,m}^{tran,d}+E_{n,m}^{comp,e}$$

(13)

If the mobile terminal user offloads the task to its associated edge node and this edge node does not have service cache, then the execution delay consists of three parts: (1) the transmission delay for the mobile terminal user to upload the task to the edge node; (2) the transmission delay for the associated edge node to download service cache from the central cloud; (3) the computation delay for the associated edge node to execute the task.

$$T_{ch3}=T_{n,m}^{tran,d}+T_{n,m}^{comp,e}+T_m^{tran,e}$$

(14)

The energy consumption corresponding to the three operation delays is

$$E_{ch3}=E_{n,m}^{tran,d}+E_{n,m}^{comp,e}+E_c^{tran,e}$$

(15)

4.5.2. Collaborative Edge Nodes Execute Tasks

If the mobile terminal user offloads the task to a cooperative edge node, then the execution delay includes three parts: (1) the transmission delay for the mobile terminal user to upload the task to the edge node; (2) the migration delay for transferring data between the edge nodes $m_1$ and $m_2$; (3) the computation delay for the cooperative edge node to execute the task.

$$T_{ch2}=T_{n,m}^{tran,d}+T_{m_1,m_2}^{tran,e}+T_{n,m}^{comp,e}$$

(16)

The energy consumption corresponding to the three operation delays is

$$E_{ch2}=E_{n,m}^{tran,d}+E_{m_1,m_2}^{tran,e}+E_{n,m}^{comp,e}$$

(17)

4.6. Problem Description

The objective of this chapter is to minimize the user’s delay under the consideration of the delay constraints of computing tasks, enhance the quality of service for users, and ensure the minimization of energy consumption in cloud computing. Based on ensuring service quality and constraint conditions, system energy consumption should be reduced and green energy-efficient computing should be achieved. The problem is formulated as follows:

$$P:\underset{\alpha ,\beta ,\mathcal{N}\in \mathcal{N}}{min}\sum _{n=1}^N\sum _{m=1}^M\sum _{t=1}^T\left(\alpha ·T_{n,m,t}+\beta ·E_{n,m,t}\right)$$

(18)

subject to:

$$\begin{array}{cc}\hfill \mathrm{C}1:& \alpha +\beta =1\hfill \\ \hfill \mathrm{C}2:& \alpha \in [0,1],\beta \in [0,1]\hfill \\ \hfill \mathrm{C}3:& T_n\le T_n^{max}\hfill \\ \hfill \mathrm{C}4:& \sum _{m=1}^M{p}_m=1\hfill \\ \hfill \mathrm{C}5:& \sum _{m=1}^M{q}_m=1\hfill \end{array}$$

4.7. Problem Solving

4.7.1. Markov Decision Process

This paper models the offloading process of edge computing tasks as a Markov Decision Process (MDP). The core of the MDP lies in the fact that current decisions are based solely on the current state, independent of previous states; future decisions only need to consider the changes in the environment after the current decision, without being affected by previous decisions. Each choice made by the mobile terminal device is based on the current environmental state, without relying on past decisions and states, to determine the action strategy. The MDP typically includes three parts—the state space, action space, and reward function—with the ultimate goal being to select the optimal action strategy to maximize cumulative rewards.

4.7.2. State Space

During the time period T for task offloading on edge devices, before a certain time segment, t, starts, the system’s state space can be represented as $S_t=[j_n\left(t\right),x_n\left(t\right),c_m\left(t\right),B_m\left(t\right)]$, where $j_n\left(t\right)$ indicates the current task generation and upload status of the terminal device, $x_n\left(t\right)$ represents the current user’s location information, $c_m\left(t\right)$ represents the service cache information in the edge computing device, and $B_m\left(t\right)$ represents the current bandwidth situation of the edge computing device.

4.7.3. Action Space

The actions taken by the intelligent agent for decision-making are represented as

$$AC=\{p,q\}$$

(19)

where p represents the user’s choice of an edge cloud server to upload and execute the computing task, and q represents the cooperative server selected during the task offloading process. The appropriate offloading strategy is chosen based on the service cache status in the servers to ultimately execute the task.

4.7.4. Reward Function

To reduce the system’s delay and energy consumption during operation, a reasonable offloading strategy must be formulated, which means that the design of the reward function must align with the system’s optimization goals. The purpose of reinforcement learning is to maximize the long-term cumulative reward. The reward function is the key to connecting the system’s optimization goals with the reward calculation, and its expression is as follows:

$$R_t=-\sum _{n=1}^N\sum _{m=1}^M(\alpha ·T_{n,m,t}+\beta ·E_{n,m,t})$$

(20)

4.7.5. Unloading Algorithm Based on Near-End Policy Optimization

PPO is an optimization algorithm suitable for problems involving discrete and continuous action spaces, used for synchronous strategy optimization. Its optimization objective involves importance sampling, but it mainly relies on the data from the previous policy, ${\theta }^{\prime }$. Adding KL divergence constraints in the objective function limits the difference between the behavior policy ${\theta }^{\prime }$ and the target policy $\theta$; thus, PPO is classified as a policy optimization algorithm. PPO evolved from the Trust Region Policy Optimization (TRPO) algorithm. The optimization process of TRPO relies on trust region methods, making it relatively complex. To address this issue, OpenAI proposed POD, which is actually a simplified version of TRPO. PPO uses a penalty term to constrain policy updates instead of directly using KL divergence constraints, adding it to the optimization objective for adjustment. Like TRPO, PPO only adjusts the coefficients of the penalty term to ensure that policy changes are within the constraint range, thus avoiding complex dual optimization. To reduce computational complexity, PPO uses a clipping mechanism to simplify the KL divergence constraints. Compared to traditional TRPO, PPO relies on clipping policy ratios, omitting trust region optimization, which also ensures that strategy updates are not too aggressive, enhancing training stability. The objectives and algorithm of PPO optimization are shown in Formula (21) and

Table 1. PPO algorithm steps.

Step	Description	Expression
Input	Collect $\{a_t,s_t,r_t\}$ tuples	-
Output	Update model	-
$i=1,2,\dots ,N_1$	Run policy $\theta$ for T times, collect $\{a_t,s_t,r_t\}$, estimate advantage ${\widehat{A}}_t={\sum }_{t^{\prime }>t}{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}-V_{\theta }\left(s_t\right)$	-
$j=1,2,\dots ,N_2$	Optimize policy	$J_{PPO}^{\theta }\left(\theta \right)=J^{\theta }\left(\theta \right)-\beta KL(\theta ,{\theta }^{\prime })$
$k=1,2,\dots ,N_3$	Update value function	$L_{\mathrm{CLIP}}\left(\varphi \right)=-{\sum }_{t=1}^T{\left({\sum }_{t^{\prime }>t}{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}-V_{\varphi }\left(s_t\right)\right)}^2$

:

$$L\left(\theta \right)\approx \mathbb{E}[min(r_t\left(\theta \right)A_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ)A_t)]$$

(21)

5. Experiment and Performance Verification

5.1. Experimental Environment and Parameter Settings

The simulation experimental platform consisted of a device with a frequency of 2.7 GHz, 16 GB of RAM, and an Intel(R)Core(TM) i5-11400H CPU. The device runs a 64-bit Windows 10 operating system, and the simulation software was Pycharm2024.2.1.

This experiment included one central cloud server, M edge servers, and N mobile terminal devices. Each terminal device had a computation task that needed to be executed. Due to the limited computing power of mobile edge devices, all computation tasks in this chapter could only be offloaded to edge servers or the central cloud for execution. The complete parameter settings in the experiment are shown in

Table 2. Simulation delay parameter descriptions and corresponding value settings.

Symbol	Definition	Value Range
$B_t^m$	Bandwidth	1–20 MHz
$P_n^{tran,d}$	Transmission power of terminal device n	0.1–2 W
$P_m^{tran,e}$	Transmission power of edge node m	1–5 W
$P_c^{tran,e}$	Transmission power of central cloud c	10–50 W
$P_m^{comp,e}$	Computing power of edge node m	10–100 W
$P_c^{comp,c}$	Computing power of central cloud	50–500 W
$N_0$	Noise power spectral density	${10}^{-9}$ W/Hz
g	Channel transmission gain	$2\times {10}^{-10}\sim 2\times {10}^{-9}$
$C_m$	Cycles per bit required by edge node m	500–2000 cycles/bit
$f_c$	CPU frequency of cloud server	2–3.5 GHz
$f_m$	CPU frequency of edge node	1–2.5 GHz
$D_n$	Data size of computation tasks	1 KB–1 MB
$W_m$	Return path bandwidth	100–1000 Mbps

.

5.2. Comparative Experimental Setup

To more fully verify the effectiveness of the proposed solution, the experiment compared the following four different schemes:

(1)

A task offloading strategy based on the edge computing optimization algorithm (EMC-PPO).

(2)

A task offloading strategy based on the edge computing optimization algorithm without edge collaboration (EMC-PPO-NO), which is also solved using the PPO algorithm. However, there is no collaboration between edge nodes, and when an edge node lacks the required service cache for task execution, it can only download the cache from the central cloud.

(3)

Complete offloading to the central cloud (CLOUD).

(4)

A random offloading strategy (Random).

5.3. Parameter Analysis

Figure 4 illustrates the convergence performance of the EMC-PPO method under different learning rate conditions. From Figure 4, it can be observed that when the learning rate was set to 0.001, 0.01, and 0.1, due to the learning rate being too high or too low, the adjustment steps of the model parameters were either too large or too small, making it impossible to find the optimal solution. On the other hand, when the learning rate was 0.001, the model achieved the best convergence effect and reward outcome. Therefore, 0.001 was determined to be the optimal learning rate hyperparameter.

Figure 5 illustrates the convergence performance of the EMC-PPO method under different discount rates. From Figure 5, it can be observed that when the discount rate was set to 0.93, the model could achieve convergence in less than 4000 iterations. However, under discount rates of 0.99 and 0.90, even after 6000 iterations, the model failed to converge. Additionally, when the discount rate was 0.96, the convergence time of the model was relatively late. This paper adopted a discount rate of 0.93 as the optimal discount rate.

5.4. Performance Comparison

In Figure 6, we compare the average total rewards of four methods: EMC-PPO, EMC-PPO-NO, CLOUD, and Random. The reward value represents the weighted sum of cloud computing energy consumption and processing delay for each event. A higher reward value implies lower energy consumption and processing delay. The EMC-PPO algorithm achieved the highest reward in the final phase and had the highest reward value in the convergence phase. Although the initial reward value of the random offloading scheme was higher than that of the EMC-PPO algorithm, it decreased as the number of training rounds increased. The final results of the EMC-PPO offloading scheme were superior. The possibility of this situation arose because the random strategy lacks the ability to adapt to changes in environmental dynamics and cannot be optimized based on environmental states and historical experience. Since the bandwidth in this paper was set to simulate real-world conditions, it fluctuated randomly. In the early training phase of the EMC-PPO algorithm, due to the lack of sufficient experience, its performance may not have been as good as the random strategy. However, it could continuously adjust the strategy based on the reward signal, gradually finding the optimal offloading strategy, thereby reducing the total energy consumption and processing delay and ultimately achieving the highest reward value. Offloading all to the central cloud, although significantly reducing processing delay and energy consumption, increased the cost of transmission delay, resulting in poor overall performance. Without collaboration between edge clouds, tasks may have needed to be offloaded multiple times to the central cloud for processing, rather than being completed directly between edge clouds, thus increasing communication overhead and energy consumption.

Figure 7 summarizes the reward situation under different numbers of edge computing nodes. As the number of edge devices increased, the speed at which the reward value improved accelerated. Although the rate of improvement varied, the final stable value tended to be consistent across all scenarios, stabilizing around −50. The fluctuation was relatively large in the initial stage, but as training progressed, the volatility gradually decreased and eventually stabilized. This indicates that within a certain range, increasing the number of edge nodes could enhance the system’s learning efficiency by providing more space for collaboration.

6. Conclusions

In this paper, the task offloading problem of mobile edge computing in MEC scenarios was studied in depth, and a task offloading scheme based on the near-end strategy optimization algorithm was proposed. This method improves task scheduling and resource allocation through collaborative computing between edge devices, thereby increasing the efficiency of computing resources. Moreover, utilizing service caching technology can avoid redundant data transmission, reduce communication latency, and network load. Next, we constructed the task offloading and resource allocation problem as a Markov Decision Process (MDP). The problem was solved using the near-end policy optimization (PPO) reinforcement learning algorithm, which can dynamically and adaptively adjust. In scenarios where the simulated network environment undergoes significant changes, the algorithm comprehensively optimizes latency, energy consumption, and computational efficiency. However, the current service caching strategy is based on static variables. In the future, we will consider incorporating cache update strategies and more intelligent algorithms to better adapt to dynamic network environments and user needs, thereby making task offloading more flexible and adaptable.