Task Partition-Based Computation Offloading and Content Caching for Cloud–Edge Cooperation Networks

Abstract

With the increasing complexity of applications, many delay-sensitive and compute-intensive services have posed significant challenges to mobile devices. Addressing how to efficiently allocate heterogeneous network resources to meet the computing and delay requirements of terminal services is a pressing issue. In this paper, a new cooperative twin delayed deep deterministic policy gradient and deep-Q network (TD3-DQN) algorithm is introduced to minimize system latency by optimizing computational offloading and caching placement asynchronously. Specifically, the task-partitioning technique divides computing tasks into multiple subtasks, reducing the response latency. A DQN intelligent algorithm is presented to optimize the offloading path to edge servers by perceiving network resource status. Furthermore, a TD3 approach is designed to optimize the cached content in the edge servers, ensuring dynamic popularity content requirements are met without excessive offload decisions. The simulation results demonstrate that the proposed model achieves lower latency and quicker convergence in asymmetrical cloud–edge collaborative networks compared to other benchmark algorithms.

1. Introduction

In recent years, the demand for mobile data traffic has been increasing as emerging Internet of Things (IoT) applications, such as smart cities, healthcare, and augmented/virtual reality (AR/VR), place stringent requirements on low latency and high reliability [1,2,3]. Traditional cloud computing encounters a series of difficulties due to the distance between peripheral devices and centralization data centers with computing and storage resources [4,5,6]. To solve this problem, mobile edge computation (MEC) is proposed [7] to reduce latency for mobile users [8,9]. MEC pushes computing and storage resources to the edge of the network such as base stations (BSs) and access points (APs) to enable computation-intensive and delay-sensitive applications on resource-limited mobile devices [10]. However, MEC ignores the huge computation resources of cloud servers [11]. Both the edge and the cloud can provide computing services for mobile devices to enhance their performance [12,13]. The edge can reduce the conveying delay by providing local computing services, while the cloud can support enormous computing requirements [14]. Their cooperation can improve the utilization of computing resources and ensure the QoS, thus being critical to the asymmetrical cloud–edge cooperation computing models [15,16]. Therefore, the asymmetrical cloud–edge cooperation network computation systems, making full use of powerful resources at both the cloud and edge, will be a promising solution [17].

With the widespread adoption of various mobile applications, large amounts of data are requested repeatedly, placing a significant burden on the mobile core network and the backhaul link. Edge caching, which allows data to be stored on the mobile edge, is proposed as an effective solution to alleviate this problem [18,19]. However, edge servers face limitations in storage capacity, making it challenging to store a diverse range of content to satisfy the needs of a large number of users [20]. To address this challenge, the technique of cooperative content caching is introduced [21]. Cooperation techniques among servers ensure that directly connected BSs no longer redundantly store cached content from neighboring BSs [22]. Consequently, this approach provides users with a diverse range of data services, while also reducing the redundancy of computational offloading and execution frequency.

However, for the content not cached by the edge server, mobile users must rely on task offloading techniques to offload the remaining tasks to other edge servers or cloud servers for completion [23]. Combining computation offloading and content caching [24] to leverage the strengths of both is a primary focus of current research. Nonetheless, a notable concern in existing studies is the imbalance in resource allocation between computation offloading and content caching strategies, leading to the overloading of some edge servers or underutilization of resources [25,26]. Therefore, to address this challenge, we propose a joint optimization of computation offloading and content caching strategies [27,28] to provide high-quality services to users.

Despite the fact that existing work can produce reliable offloading and caching decisions, they have been unable to realize fine-grained task partitioning and parallel processing when processing a complete task, thus hindering the realization of rational resource allocation. To address this problem, we introduce the task-partitioning technique [29,30]. Considering the attributes of tasks and the status of network nodes, complex tasks are divided into multiple subtasks. These subtasks are then allocated proportionally to edge nodes associated with the user for parallel execution [31], thereby improving overall computational efficiency and system performance. Although task-partitioning techniques can improve the efficient use of resources, in resource-constrained network scenarios, the task-partitioning technique still has shortcomings in handling computationally intensive tasks and cannot reduce the number of repeated task offloads and executions. Therefore, task partitioning is envisioned as an effective complement to task offloading [32].

Task-offloading technology based on task partitioning can not only allocate network resources in finer granularity to improve the system’s parallel processing capability, but also reduce server load pressure and improve data access speed while reducing network data transmission [33,34]. Although existing works can effectively solve the problems of computation-intensive task partitioning and computation offloading [35], few studies focus on the joint optimization of the task partitioning, computation offloading, and content caching strategies. Therefore, we propose an optimization model based on joint task partitioning, content caching, and computation offloading. The main contributions of this paper are as follows:

• We formulate a minimizing network delay model to describe the joint optimization problem for computation offloading and content caching in cloud–edge collaboration networks, where the task-partitioning technique is introduced to divide computing tasks into multiple subtasks, reducing the response latency.
• For the joint optimization problem of computation offloading and content caching, we propose a cooperative twin delayed deep deterministic policy gradient and deep-Q network (TD3-DQN) algorithm. The proposed algorithm intelligently replaces the in-network contents and optimizes the computation offloading path to edge servers by observing the network caching status and network resource environments, respectively.
• We evaluate the performance of the proposed solution under different environments. The simulation results show that the proposed model can achieve lower network latency and fast convergence compared to its existing benchmarks in cloud–edge cooperation networks.

The rest of this paper is organized as follows. The system model and problem are described in Section 2. We propose a DRL-based cloud–edge cooperative caching and intelligent routing solution in Section 3. Our simulation results and discussions are given in Section 4. Finally, the conclusions are given in Section 5.

2. Related Work

2.1. Intelligent Task Offloading

To meet the differentiated service requirements of these delay-sensitive and computing-intensive tasks, intelligent task offloading has been drawing great attention as a promising solution [2,3]. By deploying computation offloading and content caching functionalities at edge nodes, the tasks from mobile users can be processed at the edge of wireless networks rather than sent to the cloud to handle, improving quality of experience (QoE) and quality of service (QoS) [4,36,37].

The computational offloading model in MEC systems has been extensively studied. Leveraging the rapid response capability of edge servers, they provide users with more convenient computing services [4]. Zhu et al. [13] addressed the computational task offloading issue in drone-assisted scenarios using a multi-agent reinforcement learning algorithm, minimizing the average task response time for drones. Liu et al. [38] formulated the multi-vehicle computation offloading problem as a multi-user computation offloading game and introduced a distributed computation offloading algorithm to reduce the computational cost of vehicles. Guo et al. [39] designed a suboptimal algorithm to jointly optimize computation offloading decisions, spectrum, power, and computing resource allocation to minimize the energy consumption of all user equipment (UE). To overcome the limited computational resources of edge servers, the collaborative cloud–edge computation offloading technique is introduced, capitalizing on the powerful computing capabilities of the cloud center [16]. Chen et al. [40] proposed a dependency-aware offloading scheme, utilizing a greedy algorithm for edge–cloud collaboration under task dependency constraints, aiming to minimize the overall system latency. Zhao et al. [17] proposed a collaborative computational offloading and resource allocation optimization scheme for the cloud–MEC collaborative computation-offloading problem, enhancing system utilization and response rates. However, existing research has focused solely on the impact of computation-offloading strategies on transmission delay or energy consumption, neglecting the advantages of data caching strategies for computation offloading.

Edge caching, as a technology that stores content on edge servers close to users, has garnered widespread attention for its ability to reduce the cost of task offloading [36,41,42]. Wang et al. [18] proposed an edge caching scheme based on the concepts of content-centric networking or information-centric networking and verified its various advantages in leveraging cached content in 5G mobile networks. Zhang et al. [19] introduced a hierarchical mobile-aware edge caching scheme that employs game theory to achieve optimal collaborative caching and computing strategies, thereby minimizing content access latency and enhancing cache resource utilization. However, due to limited storage capacity, edge servers cannot cache all content, leading to the need to search for uncached request content in cloud data centers or other neighboring servers [22]. Xiao et al. [43] designed a collaborative cloud–edge–end transcoding online caching solution for IoT video services, aiming to minimize overall system latency. Kwak et al. [21] implemented a hybrid content caching design for the central cloud unit (CU) and base stations’ (BSs) control decisions using Lyapunov optimization methods, minimizing average end-to-end service latency. Su et al. [44] proposed a dynamic content caching scheme based on cross-entropy, leveraging collaborative content caching between vehicle requests and roadside units (RSUs) to minimize content transmission latency. However, these efforts often neglect the impact of computation offloading costs on the network, and the proposed caching strategies, in many cases, overlook the updating of cached content.

The combination of computation offloading and content caching [45,46] not only reduces the burden on local devices and enhances computational capabilities, but also improves data access speed and reduces data transmission costs [25,27]. Yang et al. [28] investigated a hybrid mobile cloud–edge computing system, employing a distributed algorithm based on the alternating direction method of multipliers (ADMM) to achieve near-optimal decisions for computation offloading and data caching. Zhou et al. [47] introduced an innovative computation-offloading and service-caching mechanism based on deep reinforcement learning (DRL), utilizing an asynchronous advantage actor–critic (A3C) algorithm aiming to minimize computational latency and overall system costs. Dong et al. [48] designed the hybrid cache algorithm and the enhanced offloading algorithm, providing optimal computation offloading and edge content-caching strategies, leading to a further reduction in the average response time for computation and content task requests. While existing work can generate reliable offloading and caching decisions [24], there is a lack of research for hybrid tasks’ offloading where computing tasks and content tasks are tightly coupled.

2.2. Task Partition

Although the existing work can produce reliable offloading and caching decisions, it is unable to realize finer-grained task partitioning and parallel processing when handling a complete task, thus introducing the task-partitioning technique [31]. Task partitioning involves breaking down a complete large-scale task into multiple subtasks and then distributing all these subtasks to different network servers based on their characteristics and server status [30]. Various studies have focused on different design objectives when addressing the task-partitioning problem [29,30,31,49]. Lan et al. [49] addressed the issue of partitioning and orchestrating computer vision applications on CPU and GPU heterogeneous edge computing platforms and introduced algorithms for minimum latency task scheduling and minimum cost task scheduling to minimize processing latency and overall system costs. Hu et al. [29] tackled the joint optimization problem of unmanned aerial vehicle (UAV) positioning, time slot allocation, and computation task partitioning by employing an enhanced Lagrangian active set method to reduce the system energy consumption for all users. Fang et al. [30] introduced an iterative algorithm based on bisection search (BSS) for resource allocation in scenarios with multiple users, which jointly optimized the task allocation ratio and offloaded power allocation, aiming to minimize the task completion time in multi-user scenarios. Feng et al. [31] investigated task partitioning and user association issues in MEC systems and proposed two solutions based on dual-decomposition and matching between UEs and edge nodes (ENs), resulting in a significant reduction in average delays for both independent and dependent subtasks. However, the challenge persists in determining how to finely segment tasks and allocate subtasks reasonably when dealing with a complex joint content and computing task.

2.3. Task-Partition-Based Intelligent Task Offloading

From the above research, it is evident that the task-partition-based intelligent task-offloading techniques not only enable fine-grained resource allocation, but also facilitate efficient task offloading while reducing data transmission pressure [33,50,51]. Currently, various approaches are being employed in the research domain to address the joint optimization of task partitioning and task offloading techniques [32,33,34,50]. Ku et al. [34] presented a heuristic algorithm based on dynamic programming that performs real-time task partitioning and offloading strategies, aiming to minimize end-to-end latency for vehicular applications while maximizing application-level performance. Gao et al. [33] proposed a hierarchical computation partitioning framework for DNNs, optimizing both task partitioning and offloading strategies to minimize computation latency, energy consumption, and server costs. Liu et al. [50] investigated a dynamic offloading framework in multi-user scenarios, aiming for joint optimization of task partitioning and the allocation of communication and computing resources to minimize task execution latency. Gao et al. [32] proposed a DNN task-partitioning and -offloading mechanism based on mixed-integer linear programming (MILP), aiming to optimize task partitioning and offloading strategies to achieve minimal processing latency and alleviate the computational burden on mobile devices. While existing works have effectively addressed the challenges of computation-intensive task partitioning and offloading, few studies focus on the joint optimization of task offloading, content caching, and task partition strategies.

3. System Model

In this section, we briefly introduce the network, communication, caching, and delay model and formulate a latency minimization model for the joint optimization problems of task offloading and resource allocation in task-partition-based asymmetrical cloud–edge cooperation networks.

3.1. Network Model

As illustrated in Figure 1, the asymmetrical cloud–edge cooperation networks are composed of mobile users (MUs), BSs, and a cloud, where all contents requested by MUs are available at the cloud. The network is a directed graph $\mathcal{G}=(\mathcal{N},\mathcal{L})$, where $\mathcal{N}$ and $\mathcal{L}$ represent the sets of network nodes and links, respectively. The set of MUs is denoted by ${\mathcal{N}}_M=\{1,2,...,j,...,N_M\}$, the set of BSs by ${\mathcal{N}}_B=\{1,2,...,i,...N_B\}$, the set of MUs accessed at BS i by ${\mathcal{M}}_i=\{1,2,...,M_i\}$, the set of adjacent BSs of the i-th BS and the cloud by ${\mathcal{B}}_i=\{1,2,...,B_i\}$, and the set of slots by $\mathcal{T}=\{1,2,...,T\}$. The network contains a set of content tasks denoted by $\mathcal{F}=\{1,2,...,F\}$, where F is the number of different network contents in the cloud–edge cooperation system. In the asymmetrical cloud–edge cooperation networks, the cloud caches all network contents, and BSs equipped with MEC servers have finite caching and computing capacities, while MUs do not have caching and computing capacities. Network contents with greater diversity will deteriorate the cache hit rate of the whole system, consuming more network delay. When the node receives the content task request k, the node queries whether the requested content k is stored in its cache, and if the content k is cached in the node, it is returned to the user along the reverse routing path. If the cache does not cache content k, the request k will be forwarded to its neighboring nodes for processing until the node satisfies the task request k.

Any task f in the network can be divided into multiple independent sub-tasks for distributed processing and transmission. The set of subtasks is denoted as ${\mathcal{F}}_f=\{f_1,f_2,...,f_k,...,f_{F_f}\}$, where $F_f$ is the number of different subtasks for task f, and the size of each subtask is different. Therefore, the task f is denoted by

$$\begin{array}{c}\hfill \sum _{f_k\in {\mathcal{F}}_f}{f}_k=f,\forall f\in \mathcal{F}.\end{array}$$

(1)

In cloud–edge cooperation systems, we assume that content popularity follows a Zipf distribution [52]. Without loss of generality, the number of different contents in the network is F, and all contents rank in descending order according to their popularity. Therefore, $R^f$ represents the total number of MUs’ requests for content f based on the $Zipf$ distribution, which can be written as

$$\begin{array}{c}\hfill R^f=\frac{f^{-\alpha }}{{\displaystyle \sum _{k=1}^F}{k}^{-\alpha }},\forall f\in \mathcal{F},\end{array}$$

(2)

where R is the total number of task requests in the network and $\alpha$ is the skewness factor of the Zipf law. A higher $\alpha$ value indicates that more popular content in the network is requested more often by MUs.

3.2. Communication Model

The communication model can be divided into two parts: downlink transmission and uplink transmission. To prevent interference between BSs and users, we assume that these transmissions occur on different frequency bands. For transmission between the BS and MUs under the same BS, orthogonal frequency-division multiple access (OFDMA) technology is used, so there is no interference between MUs connected to the same BS since the carrier frequencies are different. However, co-channel interference may occur between MUs connected to different BSs. The transmission of wireless links is assumed to be static in the t-th slot. In the downlink, we use $P_i^{max}$ to denote the maximum transmit power of the i-th BS. Here, the transmit power profile of the i-th BS in the t-th slot is denoted as $P_i\left(t\right)=\{P_{i1}\left(t\right),P_{i2}\left(t\right)...,P_{ij}\left(t\right),...P_{i{M}_i}\left(t\right)\}$, and it must satisfy the following constraints:

$$\sum _{j\in {\mathcal{M}}_u}{P}_{ij}\left(t\right)\le P_i^{max},\forall i\in {\mathcal{N}}_B,j\in {\mathcal{M}}_u,t\in \mathcal{T},$$

(3)

where $P_{ij}\left(t\right)$ is the transmit power of the transmit link between the i-th BS and the j-th MU connected to it in the t-th slot. The constraint indicates that the sum of the transmit power after the power profile is less than or equal to the maximum transmit power of the i-th BS in the t-th slot.

In the uplink, we use $P_{ji}\left(t\right)$ to denote the transmit power of the transmit link between the j-th MU under the i-th BS and the i-th BS in the t-th slot, and it must satisfy the following constraints:

$$P_{ji}\left(t\right)\le P_j^{max}\left(t\right),\forall i\in {\mathcal{N}}_B,j\in {\mathcal{M}}_u,t\in \mathcal{T},$$

(4)

where $P_j^{max}\left(t\right)$ is the maximum transmit power of the j-th MU under the BS i in the t-th slot. The constraint indicates that the transmit power of each MU should be less than or equal to its own maximum transmit power.

In the downlink, all signals received by the j-th MU in the t-th slot when the i-th BS transmits information to the j-th MU connected to it are as

$$\begin{array}{cc}\hfill y_{ij}^m\left(t\right)=& \sqrt{P_{ij}^m}\left(t\right)h_{ij}^m\left(t\right)z_{ij}^m\left(t\right)+\sum _{i^{\prime }\in {\mathcal{N}}_B\setminus \left\{i\right\}}\sqrt{P_{i^{\prime }}^m}\left(t\right)h_{i^{\prime }j}^m\left(t\right)z_{i^{\prime }j}^m\left(t\right)+n_{ij}^m,\hfill \end{array}$$

(5)

where $h_{ij}^m\left(t\right)$ is the channel gain between the i-th BS and the j-th MU connected to it on subcarrier m in the t-th slot. $z_{ij}^m\left(t\right)$ is the signal of transmission of the subcarrier m between the i-th BS and the j-th MU connected to it on subcarrier m in the t-th slot. $\sqrt{P_{ij}^m}\left(t\right)h_{ij}^m\left(t\right)z_{ij}^m\left(t\right)$ indicates that the j-th MU receives the useful signal transmitted by the local BS i. ${\displaystyle \sum _{i^{\prime }\in {\mathcal{N}}_B\setminus \left\{i\right\}}}\sqrt{P_{i^{\prime }}^m}\left(t\right)h_{i^{\prime }j}^m\left(t\right)z_{i^{\prime }j}^m\left(t\right)+n_{ij}^m$ indicates that the j-th MU receives interference signals transmitted by BSs other than local BS i. $n_{ij}^m$ indicates white noise.

Thus, the transmit rate of the downlink between the i-th BS and the j-th MU connected to it on subcarrier m in the t-th slot is as

$$\begin{array}{cc}\hfill & r_{ij}^m\left(t\right)=B_{ij}^m\left(t\right)lo{g}_2\left(1+SIN{R}_{ij}^m\left(t\right)\right);\hfill \end{array}$$

(6)

$$\begin{array}{c}\hfill SIN{R}_{ij}^m\left(t\right)=\frac{P_{ij}^m\left(t\right){|h_{ij}^m\left(t\right)|}^2}{{\sigma }^2+{\displaystyle \sum _{i^{\prime }\in {\mathcal{N}}_B\setminus \left\{i\right\}}}{P}_{i^{\prime }}^m\left(t\right){|h_{i^{\prime }j}^m\left(t\right)|}^2},\end{array}$$

(7)

where $B_{ij}^m\left(t\right)$ is the link bandwidth of the downlink between the i-th BS and the j-th MU connected to it on subcarrier m in the t-th slot. ${\sigma }^2$ is the power of white noise. $SIN{R}_{ij}^m\left(t\right)$ is the signal-to-interference-plus-noise ratio (SINR) of the downlink between the i-th BS and the j-th MU connected to it on subcarrier m in the t-th slot.

In the uplink, all signals received by the i-th BS when the j-th MU under the i-th BS transmits information to i-th BS are as

$$\begin{array}{cc}\hfill y_{ji}^m\left(t\right)=& \sqrt{P_j^m}\left(t\right)h_{ji}^m\left(t\right)z_{ji}^m\left(t\right)+\sum _{i^{\prime }\in {\mathcal{N}}_B\setminus \left\{i\right\}}\sum _{j^{\prime }\in {\mathcal{M}}_{i^{\prime }}}\sqrt{P_{j^{\prime }}^m}\left(t\right)h_{j^{\prime }i}^m\left(t\right)z_{j^{\prime }i}^m\left(t\right)+n_{ji}^m,\hfill \end{array}$$

(8)

where $\sqrt{P_j^m}\left(t\right)h_{ji}^m\left(t\right)z_{ji}^m\left(t\right)$ indicates that the i-th BS receives the useful signal transmitted by the j-th MU under the i-th BS. $n_{ji}^m+{\displaystyle \sum _{i^{\prime }\in {\mathcal{N}}_B\setminus \left\{i\right\}}}{\displaystyle \sum _{j^{\prime }\in {\mathcal{M}}_{i^{\prime }}}}\sqrt{P_{j^{\prime }}^m}\left(t\right)h_{j^{\prime }i}^m\left(t\right)z_{j^{\prime }i}^m\left(t\right)$ indicates that the i-th BS receives the interference signals transmitted by the $j^{\prime }$-th under the BSs other than the local BS i. $n_{ji}$ indicates white noise.

Thus, the transmit rate of the uplink between the j-th MU under i-th BS and the i-th BS on subcarrier m in the t-th slot is as

$$\begin{array}{cc}\hfill & r_{ji}^m\left(t\right)=B_{ji}^m\left(t\right)lo{g}_2\left(1+SIN{R}_{ji}^m\left(t\right)\right);\hfill \end{array}$$

(9)

$$\begin{array}{c}\hfill SIN{R}_{ji}^m\left(t\right)=\frac{P_j^m\left(t\right){|h_{ji}^m\left(t\right)|}^2}{{\sigma }^2+{\displaystyle \sum _{i^{\prime }\in {\mathcal{N}}_B\setminus \left\{i\right\}}}{\displaystyle \sum _{j^{\prime }\in {\mathcal{M}}_{i^{\prime }}}}{P}_{j^{\prime }}^m\left(t\right){|h_{j^{\prime }i}^m\left(t\right)|}^2},\end{array}$$

(10)

where $B_{ji}^m\left(t\right)$ is the wireless bandwidth of the uplink between the j-th MU and the i-th BS connected to it on subcarrier m in slot t. $P_j^m\left(t\right)$ is the transmitting power of the j-th MU on subcarrier m in slot t; $h_{ji}^m\left(t\right)$ is the channel gain of the uplink between the j-th MU and the i-th BS on the subcarrier m in slot t.

In the wired communication model, we directly define the maximum link capacity of the wired link, and when the wired link transmits a content file, the link capacity is updated in real time, that is the current link capacity minus the content file size. The rate at which the content file size of the wired link transmits is the current remaining link capacity of the link, which is denoted by $r_{i{i}^{\prime }}$. $r_{i{i}^{\prime }}$ indicates the transmission rate from the BS i to its adjacent processing node $i^{\prime }$$\left(\forall i^{\prime }\in {\mathcal{B}}_i\right)$.

The routing process that MU j under BS i sends a task request f to BS i and obtains the corresponding content is as follows. Firstly, MU j sends all subtask requests ${\mathcal{F}}_f$ to BS i. Then, the local BS i and its adjacent processing node $i^{\prime }$$\left(\forall i^{\prime }\in {\mathcal{B}}_i\right)$ collaborate to process all subtask requests from MU j based on their cached content and computing resources. Finally, the corresponding content obtained by each processing node after processing its respective subtask request $f_k$ is transmitted back to the MU j. Note that the local BS i and the adjacent processing node $i^{\prime }$ process their respective subtask $f_k$ requests at the same time. Here, we use ${\mathcal{F}}_i^f$ to denote the subtask request set of task f actually processed at BS i. It must satisfy the following constraints:

$$\begin{array}{c}\hfill {\mathcal{F}}_f={\mathcal{F}}_i^f\cup \sum _{i^{\prime }\in {\mathcal{B}}_i}{\mathcal{F}}_{i^{\prime }}^f,\forall f\in \mathcal{F},i\in {\mathcal{N}}_B,i^{\prime }\in {\mathcal{B}}_i,\end{array}$$

(11)

where ${\displaystyle \sum _{i^{\prime }\in {\mathcal{B}}_i}}{\mathcal{F}}_{i^{\prime }}^f$ indicates that the total set of subtask requests of task f is actually processed by adjacent node $i^{\prime }$. The constraint indicates that the subtask request set of task f processed by local BS i and its adjacent processing node $i^{\prime }$ must form a complete task request f.

3.3. Caching Model

We assume that, for content tasks, the cloud has unlimited cache capacities to store all contents. On the other hand, BSs have limited cache capacity and can only cache a part of the contents. Additionally, the same content cannot be cached between connected BSs. $S^{f_k}$ is the size of the content block about subtask request $f_k$. The cache capacity of BS i is denoted by $O_i$. Let $X_i^{f_k}\left(t\right)$ be a Boolean variable indicating whether node i caches content block $f_k$ in slot t. $X_i^{f_k}\left(t\right)=1$ if node i caches content block $f_k$ in slot t, and $X_i^{f_k}\left(t\right)=0$, otherwise.

Thus, the size of all contents cached in BS i in the t-th slot must be less than or equal to the cache capacity of BS i, which can be written as

$$\begin{array}{cc}\hfill & {\displaystyle \sum _{f\in \mathcal{F}}}{\displaystyle \sum _{f_k\in {\mathcal{F}}_f}}{X}_i^{f_k}\left(t\right)S^{f_k}\le O_i,\forall i\in {\mathcal{N}}_B,t\in \mathcal{T},\hfill \end{array}$$

(12)

where $S^{f_k}$ represents the size of subtask $f_k$.

The same content cannot be cached between connected BSs in the t-th slot, which can be written as

$$\begin{array}{cc}\hfill & \sum _{i^{\prime }\in {\mathcal{B}}_i\setminus \left\{c\right\}}{X}_{i^{\prime }}^{f_k}\left(t\right)+X_i^{f_k}\left(t\right)\le 1,\forall f_k\in {\mathcal{F}}_f,i\in {\mathcal{N}}_B,t\in \mathcal{T}.\hfill \end{array}$$

(13)

Assume that the number of different contents provided by the BS i is k, and all contents rank in descending order according to content popularity. The higher ranking content will be requested more often.

3.4. Delay Model

In our system, the delay generated by the system is mainly divided into the sojourn delay generated by the processing node and the transmission delay generated by the link.

3.4.1. Sojourn Delay

Considering that the heterogeneous, dynamic, and limited network resources make it difficult to meet the differentiated service needs of mass access MUs, sojourn latency refers to the delay between the content request entering and leaving the node, and the delay in obtaining the content from $BSs$ or the cloud depends mainly on the number of CPU cycles required for the subtask request. Node i handles the sojourn delay of the subtask request $f_k$, expressed as $T_i^{p,f_k}\left(t\right)$, which can be expressed as

$$T_i^{p,f_k}\left(t\right)=\frac{C_i^{f_k}\left(t\right)}{V_i},$$

(14)

where $C_i^{f_k}\left(t\right)$ is the amount of CPU cycles consumed by node i to process subtask request $f_k$ in slot t, and $V_i$ is the maximal computing capacity of node i.

3.4.2. Transmission Delay

We assume that there exist computation and content transmission tasks in the network. The transmission delay is divided into an uplink transmission delay and a downlink transmission delay. For the computation task, the data related to the computing process from MU j are sent to BS i or the cloud node for processing, and the corresponding result is returned to MU j. The computing task occupies more uplink bandwidth. For the content transmission task, the requested content is sent back to MU j from the BS i or the cloud node, and the content transmission task consumes more downlink bandwidth.

The round-trip transmission delay between BS i and MU j to process subtask $f_k$ on subcarrier m in slot t, denoted by $T_{ij}^{tr,m,f_k}\left(t\right)$, can be expressed as

$$T_{ij}^{tr,m,f_k}\left(t\right)=\frac{D^{f_k}}{r_{ji}^m\left(t\right)}+\frac{S^{f_k}}{r_{ij}^m\left(t\right)},$$

(15)

where $D^{f_k}$ indicates the computed data size of the subtask $f_k$ request. $\frac{D^{f_k}}{r_{ji}^m\left(t\right)}$ indicates that the uplink transmission delay of MU j transmitting subtask $f_k$ requests to BS i in the t-th slot. $S^{f_k}$ indicates the content data size of the subtask $f_k$ request. $\frac{S^{f_k}}{r_{ij}^m\left(t\right)}$ indicates the downlink transmission delay of BS i transmitting the content file corresponding to subtask $f_k$ to MU j in the t-th slot.

Similarly, the round-trip transmission delay between BS i and its adjacent cloud–edge node c to process subtask $f_k$ in slot t, denoted by $T_{ic}^{tr,f_k}\left(t\right)$, can be expressed as

$$T_{ic}^{tr,f_k}\left(t\right)=\frac{D^{f_k}}{r_{ic}\left(t\right)}+\frac{S^{f_k}}{r_{ci}\left(t\right)},$$

(16)

where $\frac{D^{f_k}}{r_{ic}\left(t\right)}$ indicates that the uplink transmission delay of BS i transmitting subtask $f_k$ requests to its adjacent cloud–edge node c in the t-th slot. $\frac{S^{f_k}}{r_{ci}\left(t\right)}$ indicates the downlink transmission delay of cloud–edge node c transmitting the content file corresponding to subtask $f_k$ to BS i in the t-th slot.

Based on the subtask offloading process mentioned above, the total delay caused by task f from MU j at BS i, denoted by $T_{ij}^f\left(t\right)$, can be written as

$$\begin{array}{cc}\hfill & T_{ij}^f\left(t\right)=T_{ij}^{tr,m,f}+max\left\{{\displaystyle \sum _{f_k\in {\mathcal{F}}_i^f}}{T}_i^{p,f_k}{X}_i^{f_k}{Y}_i^{f_k},{\displaystyle \underset{i^{\prime }\in {\mathcal{B}}_i}{max}}\left[{\displaystyle \sum _{f_k\in {\mathcal{F}}_{i^{\prime }}^f}}\left(T_{i{i}^{\prime }}^{tr,f_k}+T_{i^{\prime }}^{p,f_k}\right)X_{i^{\prime }}^{f_k}{Y}_{i^{\prime }}^{f_k}\right]\right\},\hfill \end{array}$$

(17)

where $\left(T_{i{i}^{\prime }}^{tr,f_k}+T_{i^{\prime }}^{p,f_k}\right)$ indicates the delay caused by the BS i sending the subtask $f_k$ request to the connected processing node $i^{\prime }$ and obtaining the corresponding content from $i^{\prime }$. This delay includes the processing delay of processing node $i^{\prime }$ and the round-trip transmission delay generated. According to the routing model, we know that the local BS i and the connected processing node $i^{\prime }$ process their respective subtask $f_k$ requests simultaneously. Thus, the delay determining this part of the process is the maximum delay generated between BS i and the connected processing node $i^{\prime }$. $Y_i^{f_k}\left(t\right)$ is a Boolean variable indicating whether a subtask is locally processed. If node i locally handles subtask $f_k$ in slot t, $Y_i^{f_k}\left(t\right)=1$, and $Y_i^{f_k}\left(t\right)=0$, otherwise.

3.5. Problem Formulation

In order to improve resource utilization and content distribution, we present the optimal offloading problem based on task division as the minimum delay model in the cache-assisted cloud–edge cooperation system. The model designs the cloud–edge cooperation offloading scheme and realizes the integrated allocation of 3C resources and the joint optimization of cache and routing in the network. In the system, the total delay is minimized by jointly optimizing offload decisions and resource allocation, which can be formulated as

$$\begin{array}{}& \hfill min& {\displaystyle \sum _{t\in \mathcal{T}}}{\displaystyle \sum _{i\in {\mathcal{N}}_B}}{\displaystyle \sum _{j\in {\mathcal{M}}_u}}{\displaystyle \sum _{f\in \mathcal{F}}}{T}_{ij}^f\left(t\right)\mathrm{(18a)}& \hfill \mathrm{s}.\mathrm{t}.& P_{ji}\left(t\right)\le P_j^{max}\left(t\right),\forall i,\forall j,\forall t,\mathrm{(18b)}& & {\displaystyle \sum _{j\in {\mathcal{M}}_u}}{P}_{ij}\left(t\right)\le P_i^{max}\left(t\right),\forall i,\forall t,\mathrm{(18c)}& & {\displaystyle \sum _{f_k\in {\mathcal{F}}_f}}{f}_k=f,\forall f,\forall k,\mathrm{(18d)}& & {\displaystyle \sum _{f\in \mathcal{F}}}{\displaystyle \sum _{f_k\in {\mathcal{F}}_f}}{X}_i^{f_k}\left(t\right)S^{f_k}\le O_i,\forall i,\forall t,\mathrm{(18e)}& & {\displaystyle \sum _{i^{\prime }\in {\mathcal{B}}_i\setminus \left\{c\right\}}}{X}_{i^{\prime }}^{f_k}\left(t\right)+X_i^{f_k}\left(t\right)\le 1,\forall f_k,\forall i\in {\mathcal{B}}_i\setminus \{c,i^{\prime }\},\forall t,\mathrm{(18f)}& & {\displaystyle \sum _{f\in \mathcal{F}}}{\displaystyle \sum _{f_k\in {\mathcal{F}}_f}}{Y}_i^{f_k}\left(t\right)C_i^{f_k}\le C_i,\forall i,\forall t,\mathrm{(18g)}& & {\displaystyle \sum _{i^{\prime }\in {\mathcal{B}}_i}}{Y}_{i^{\prime }}^{f_k}\left(t\right)+Y_i^{f_k}\left(t\right)\le 1,\forall f_k,\forall i\in {\mathcal{B}}_i\setminus \left\{i^{\prime }\right\},\forall t,\mathrm{(18h)}& & {\mathcal{F}}_f={\mathcal{F}}_i^f\cup {\displaystyle \sum _{i^{\prime }\in {\mathcal{B}}_i}}{\mathcal{F}}_{i^{\prime }}^f,\forall f,\forall i\in {\mathcal{B}}_i\setminus \left\{i^{\prime }\right\},\mathrm{(18i)}& & f_{l_{ij}}\le B_{l_{ij}},\forall i,\forall j,\mathrm{(18j)}& & X_i^{f_k}\left(t\right),Y_i^{f_k}\left(t\right)\in \{0,1\},\forall f_k,\forall i,\forall t.\end{array}$$

In the above constraint, (18a) indicates that each user’s transmit power should be less than or equal to its own maximum transmit power, and the sum of the transmit power after the power profile is less than or equal to the maximum transmit power of the i-th BS in the t-th slot. (18b) means that all subtasks of task f must be combined to form a complete task f, and all subtasks are independent of each other. (18c) indicates that the size of all content cached in BS i must be less than or equal to the cache capacity of BS i. (18d) means that the same content cannot be cached between adjacent BSs, that is adjacent BSs should be cooperatively cached. (18e) indicates that the total computing power required by node i to process subtasks should be less than or equal to the computing power of processing node i. (18f) indicates that the same subtask request cannot be processed repeatedly between processing nodes. (18g) indicates that the subtask request set of task f processed by the local BS i and the subtask request set of task f processed by its neighbor processing node $i^{\prime }$ must form a complete task f request. (18h) means that the total transmit traffic $f_{l_{ij}}$ generated on any link (wired or wireless) should not exceed its corresponding link bandwidth $B_{l_{ij}}$. (18i) means that all Boolean variables must be 0 or 1.

4. Task Offloading and Resource Allocation via Deep Reinforcement Learning

In order to minimize the delay in (18), the proposed joint optimization problem of task offloading and resource allocation is a sequential decision-making process that belongs to the Markov decision process. The DRL algorithm can be used for joint optimization by learning historical knowledge and establishing mappings between network states and actions through neural networks and reinforcement learning strategies, which can be defined by a tuple $\left\{𝒮,\mathcal{A},\mathcal{P}\left(s_{t+1}|s_t,a_t\right),\mathcal{R}\left(s_t,a_t\right)\right\}$. $𝒮$ is the set of states to represent current cloud–edge cooperation environments. $\mathcal{A}$ is the set of all possible actions of the Markov decision process (MDP). $\mathcal{P}\left(s_{t+1}|s_t,a_t\right)$ is the probability of transforming from state $s_t$ to state $s_{t+1}$ after performing action $a_t$ at time t. $\mathcal{R}\left(s_t,a_t\right)$ represents the reward when action $a_t$ is performed in state $s_t$. Therefore, cloud–edge collaboration systems must collect much information about the network topology, user requests, node cache states, and network resources to process massive content requests. In the decision-making process of offloading, the algorithm based on DQN observes the network environment, captures the state information (network resources, user requests, etc.) in step t, outputs the optimal offloading decision, and serves the request routing in the network. Moreover, TD3 is employed to optimize caching replacement based on its property of being good at optimizing continuous variables in caching decision action.

4.1. Model Design of DQN-Enabled Task Offloading

The optimization problem of this model is an MDP, which can be defined by a tuple $\left\{𝒮,\mathcal{A},\mathcal{P}\left(s_{t+1}|s_t,a_t\right),\mathcal{R}\left(s_t,a_t\right)\right\}$. Here, $𝒮$ is the set of states describing the current network environment, including the node status, link bandwidth size, transmit power size, and other network state information. $\mathcal{A}$ is the set of all possible actions in the MDP process, such as all possible node actions that can be selected in this network structure. $\mathcal{P}\left(s_{t+1}|s_t,a_t\right)$ represents the probability of transforming from state $s_t$ to state $s_{t+1}$ after performing action $a_t$. The state transition equation satisfies the characteristics of a Markov process. The state of $s_{t+1}$ only depends on $s_t$ and the action selected in that state, and has no relation to the previous state. $\mathcal{R}\left(s_t,a_t\right)$ represents the reward obtained when executing action $a_t$ under state $s_t$. In the MDP, the core goal is to find a way to select action $a_t$ to obtain the maximum reward value.

The MDP can be solved using various reinforcement learning algorithms. Among them, Q-learning is a reinforcement learning algorithm that optimizes rewards by dynamically acquiring environmental state information and storing action values [53]. Its advantage is that it does not require prior knowledge or models and can learn through interaction with the environment. Each update only needs to consider the current state and action, without considering the entire state transition process. However, in the complex network environment of the cloud–edge, the Q-table will store a large number of state and action values, resulting in a high-dimensional state space and resource waste [54]. In contrast, the DQN, as a branch of DRL, can solve these problems by using deep neural networks. Therefore, the DQN can better handle complex problems than traditional Q-learning algorithms and has better convergence and performance. In this section, we propose a new task-offloading strategy based on the DQN, which aims to reduce latency consumption through collaborative caching and offloading decisions.

Figure 2 illustrates the workflow of the task-offloading algorithm based on the DQN. As shown in the figure, the DQN consists of two neural networks: an evaluation network and a target network, both having the same neural network structure, but different parameters. The evaluation network is used to output the action value Q by inputting the state $s_t^f$ at time slot t. To avoid the algorithm from getting stuck in a local optimal value, we use an $\epsilon -greedy$ strategy to randomly select actions with probability $\epsilon \in \left(0,1\right)$ or random actions with probability $1-\epsilon$. In our DQN policy, we use backpropagation (BP) and gradient descent algorithms to update the parameters of the evaluation network, i.e., by randomly selecting a set of historical information from experience replay to calculate and minimize the loss function, thereby adjusting the relevant parameters. The loss function can be calculated as

$$\begin{array}{cc}\hfill & L{\left(w\right)}^f=E\left\{{\left[r_t^f+\gamma {\displaystyle \underset{a_t^f}{max}}{Q}^{\prime }\left(s_{t+1}^f,a_t^f;w^{-}\right)-Q\left(s_t^f,a_t^f;w\right)\right]}^2\right\},\hfill \end{array}$$

(19)

where $r_t^f$ indicates the actual reward obtained by taking action $a_t^f$, $\gamma$ is the discount rate, w is the weight of the evaluation network, and $w^{-}$ is the weight of the target network. $Q\left(s_t^f,a_t^f;w\right)$ is the predicted Q-value generated by the evaluation network. ${\displaystyle \underset{a_t^f}{max}}{Q}^{\prime }\left(s_{t+1}^f,a_t^f;w^{-}\right)$ indicates the maximum Q-value of the target network, and $r_t+\gamma {\displaystyle \underset{a_t^f}{max}}{Q}^{\prime }\left(s_{t+1}^f,a_t^f;w^{-}\right)$ is the actual Q-value.

Finally, to improve the stability and convergence of the system, the target network copies the network parameters from the evaluation network to update its own neural network in each specific training cycle.

4.2. Task-Partition-Based Intelligent Offloading Procedure

In the network environment, the network state generated by task request f at time slot t is defined as $s_t^f=\left\{n_t,C_{n_t},f,{\mathcal{A}}_{n_t},X_{n_t},f_{l_{n_t,j}},j\in {\mathcal{A}}_{n_t}\right\}$, where $n_t$ represents the current node that processes task request f; $C_{n_t}$ represents the type of current node, which can be an MU, BS, or the cloud; f represents the task f; ${\mathcal{A}}_{n_t}$ is the set of adjacent processing nodes of the current processing node $n_t$; $X_{n_t}=\left(X_{n_t}^1,X_{n_t}^2,...,X_{n_t}^F\right)$ represents the cache status of node $n_t$; $f_{l_{n_t,j}}$ represents the traffic of link $l_{n_t,j}$. At time slot t, the action $a_t^f$ of task request f is defined as $a_t^f=\left\{n_{t+1},n_{t+1}\in {\mathcal{A}}_{n_t}\right\}$, which indicates the behavior action of the current node choosing the next node.

To obtain the optimal offloading decision, we designed a reward function that combines environmental feedback signals with optimization objectives. During the routing process, when a request is satisfied, the system receives a reward signal based on the delay. If the request is satisfied at the network edge (local BS and adjacent BS), the system will receive more rewards to reduce the content transmission delay. If the request packet is lost during the routing process, the system cannot receive rewards. At time slot t, the reward value function obtained by task request f is

$$\begin{array}{c}\hfill r_t^f=\frac{\gamma }{T_t^f},\end{array}$$

(20)

where $\gamma$ represents the discount factor, which is used to adjust the proportional relationship between the reward value and the delay. $T_t^f$ represents the total delay that users need to consume during the period from sending task request f in time slot t to receiving the corresponding content of task request f.

In the proposed DQN algorithm, the core goal of the algorithm is to find the optimal decision-making strategy through training and learning of historical data in each time slot t, so as to maximize the sum of the expected reward values of all task requests sent by all mobile users. Because the reward value is inversely proportional to the delay, when the reward value is maximized, the delay is minimized accordingly. During the pathfinding process of DQN, if the current BS can meet the service requirements of the task request f, the system immediately terminates the pathfinding process of the current task request f and returns the corresponding content of the task request f to the corresponding user node. Conversely, if the current node cannot meet the service requirements requested by the task, the system will randomly select an action according to the $\epsilon -greedy$ policy, send the status information $s_t^f$ to the evaluation network, and obtain the next action $a_t^f$ from the evaluation network. After performing the action $a_t^f$, the system will report the corresponding reward $r_t^f$ and the next state $s_{t+1}^f$. At the same time, the system will also store this state information $\left(s_t^f,a_t^f,r_t^f,s_{t+1}^f\right)$ into the experience playback pool in order to obtain data from the experience playback pool for neural network training. This cycle repeats until the task request f is satisfied or the packet is lost. Here, DQN pathfinding is performed on a task, and then, DQN pathfinding is performed for the next task request. Algorithm 1 gives the task offloading training process based on the DQN, where $N_E$ represents the total number of time slots and $N_s$ represents the number of requests in each time slot.

Algorithm 1: DQN-based task offloading.

4.3. Model Design of TD3-Enabled Intelligent Caching

In the intelligent caching problem based on task partition, the optimal caching updating strategy is realized by using caching history information. The cache updating model is an MDP. The agent makes the action $a_t\in \mathcal{A}$ decision by sensing the current state $s_t\in 𝒮$ at time slot $t\in \mathcal{T}$, where $𝒮$ and $\mathcal{A}$ represent the state space and action space, respectively. After the agent has performed the action $a_t$, it is rewarded with $r_t$ based on immediate feedback, and the environment state is transformed to $s_{t+1}$ accordingly. The Markov transition tuple $\left(s_t,a_t,r_t,s_{t+1}\right)$ is recorded in the experience replay buffer for the agent’s training. Because the caching problem in heterogeneous scenarios is suitable for continuous action space solutions, while the Deep Q network (DQN), an important branch of the DRL algorithm, is suitable for caching decisions in discrete action space, the DQN is not considered a solution to the intelligent caching problem in heterogeneous scenarios. Therefore, considering another important branch of the DRL algorithm, TD3 is a reinforcement learning algorithm based on policy gradients that is suitable for a continuous action space in heterogeneous scenarios. Therefore, TD3 is considered as a solution to the intelligent caching problem. In order to obtain the optimal solution for the optimization problem, the state space $𝒮$, action space $\mathcal{A}$, and reward function $r_t$ in the proposed intelligent caching MDP model are designed as follows:

• State space: The state ${𝒮}_t$ in slot t includes network caching information ${\mathcal{C}}_t$, user request information ${\mathcal{F}}_t$, and network topology information ${\mathcal{G}}_t$. Thus, the state vector at slot t is expressed as ${𝒮}_t=\left\{{\mathcal{C}}_t,{\mathcal{G}}_t,{\mathcal{F}}_t\right\}$, and ${\mathcal{C}}_t$ is the content caching state of all BSs in slot t as ${\mathcal{C}}_t=\left\{{\mathcal{C}}_{i,t},\forall i\in {\mathcal{N}}_B\right\}$.
• Action space: In the caching decision process, the optimal caching decision action ${\mathcal{A}}_t$ in the resource-constrained cloud–edge collaborative network contains the caching contents of all nodes in slot t. ${\mathcal{A}}_t$ can be expressed as

  $$\begin{array}{cc}\hfill {\mathcal{A}}_t=& \left[{\mathcal{C}}_{1,t+1},{\mathcal{C}}_{2,t+1},...,{\mathcal{C}}_{i,t+1},...,{\mathcal{C}}_{N_B,t+1}\right],i\in {\mathcal{N}}_B,\hfill \end{array}$$

  (21)

  where ${\mathcal{C}}_{i,t+1}$ is the content caching state of BS i at in slot t + 1; the set of BSs is denoted by ${\mathcal{N}}_B=\{1,2,...,i,...N_B\}$.
• Reward: The design of the reward function directly affects the exploration of the caching updating problem (P1) and the convergence of the algorithm. These factors are considered in the design of the reward function in order to optimize the content cache based on meeting the user’s service requirements. Therefore, the reward function in slot t can be formulated as

  $$\begin{array}{c}\hfill r_t=\lambda \sum _{k=1}^{N_s}{R}_k+(1-\lambda )\sum _{i\in {\mathcal{N}}_B}{\mathcal{H}}_t^i,\end{array}$$

  (22)

  where $N_s$ is the maximum number of training steps; $\lambda$ is the weight parameter; ${\mathcal{H}}_t^i$ is the number of cache hits for BS i at time slot t; $R_k$ represents user satisfaction at step k.

Figure 3 illustrates the framework of the proposed TD3-based content caching model. As shown in Figure 3, there are six neural networks in TD3, including the actor network, the twin critic networks, the target actor network, and the target twin critic networks. In particular, to facilitate the training of the neural networks, the target network will have the same structure as its corresponding actor or critic networks to be used as a training label for the loss function.

The TD3 algorithm mainly relies on actor and critic networks to make action decisions and evaluations, respectively. Specifically, the actor network selects the action $a_t$ in a given state by fitting the action decision function $\pi \left(s_t\mid {\omega }_{\varphi }\right)$ for state $s_t$ in time slot t. On the other hand, the critic network evaluates the value of the action choice in a given state by fitting the state–action value function $Q\left(s_t,a_t\mid {\omega }^{\theta }\right)$. The parameters of the actor and critic networks are denoted by ${\omega }_{\varphi }$ and ${\omega }^{\theta }$, respectively.

In order to overcome the overestimation of the Q-value, TD3 adopts the double-evaluation network strategy to estimate the actual Q-value, which uses the smaller of the two Q-values to form the targets in the Bellman error loss functions, and $y_t$ is calculated by the following equation:

$$\begin{array}{cc}\hfill & y_t=r_t+\gamma min\left\{Q\left[{\mathit{s}}_{t+1},\pi \left({\mathit{s}}_t\mid {\mathit{\omega }}_{{\varphi }^{\prime }}+\epsilon \right)\mid {\mathit{\omega }}^{{\theta }_1^{\prime }}\right],Q\left[{\mathit{s}}_{t+1},\pi \left({\mathit{s}}_t\mid {\mathit{\omega }}_{{\varphi }^{\prime }}+\epsilon \right)\mid {\mathit{\omega }}^{{\theta }_2^{\prime }}\right]\right\}\hfill \end{array}$$

(23)

where ${\mathit{\omega }}_{{\varphi }^{\prime }}$ and ${\mathit{\omega }}^{{\theta }^{\prime }}$ represent the parameters of the target actor network and critic networks, and $\epsilon$∼$\mathcal{N}(0,\xi )$ represents Gaussian noise with scale $\xi \left(\sigma \right)$ related to the training episode $\sigma$. In order to enhance the exploration of the TD3 algorithm, a relatively large value of $\xi \left(\sigma \right)$ is set in the early stage of training, and as the algorithm iterates, $\xi \left(\sigma \right)$ gradually decreases to improve the exploitation of the algorithm.

The experience generated by the algorithm during different training steps is stored in the replay buffer to be sampled to calculate the TD-error, and then, the critic networks are updated. $y_t$ will be used as a tag to train the two critic networks by minimizing the loss function as follows:

$$\begin{array}{c}\hfill L\left({\mathit{\omega }}_{\varphi }\right)=\frac{1}{m}\sum _m{\left[y_t-Q\left({\mathbf{s}}_t,a_t\mid {\mathit{\omega }}^{\theta }\right)\right]}^2,\end{array}$$

(24)

where m is the batch size from the experience replay buffer. The target networks adopt a delay soft updating policy to iterate their parameters $\pi \left({\mathit{s}}_t\mid {\mathit{\omega }}_{{\varphi }^{\prime }}\right)$.

This policy refers to the fact that the TD3 algorithm updates the target actor–critic networks less frequently than the actor–critic networks to reduce the oscillation of the TD3, which can be expressed as

$$\begin{array}{cc}\hfill & {\mathit{\omega }}^{{\theta }^{\prime }}\leftarrow \tau {\mathit{\omega }}^{\theta }+(1-\tau ){\mathit{\omega }}^{{\theta }^{\prime }};\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill & {\mathit{\omega }}_{{\varphi }^{\prime }}\leftarrow \tau {\mathit{\omega }}_{\varphi }+(1-\tau ){\mathit{\omega }}_{{\varphi }^{\prime }},\hfill \end{array}$$

(26)

where $\tau$ is the update coefficient and $0<\tau \ll 1$.

Finally, the actor network is trained by the critic network gradient:

$$\begin{array}{cc}\hfill {\nabla }_{{\mathit{\omega }}_{\varphi }}J\left({\mathit{\omega }}_{\varphi }\right)=& \frac{1}{m}\sum _m{\nabla }_{a}Q\left({\mathbf{s}}_t,a_t\mid {\mathit{\omega }}^{{\theta }_1}\right){|}_{a_t=\pi \left({\mathbf{s}}_t\right)}{\nabla }_{{\mathit{\omega }}_{\varphi }}\pi \left({\mathbf{s}}_t\mid {\mathit{\omega }}_{\varphi }\right).\hfill \end{array}$$

(27)

4.4. Task-Partition-Based Intelligent Caching Procedure

In the proposed process of intelligent caching based on task partition, the aim is to use the caching history information to achieve optimal caching updating, maximize the sum of the expected reward, and achieve a low-latency content distribution. As shown in Algorithm 2, the workflow of caching updating optimization based on DRL is shown to achieve the optimal node caching updating. When all nodes receive task requests in time slot t, the caching updating decision $a_t$ is output to update the caching status of all nodes. After entering the time slot $t+1$, all nodes receive the task request again, and the system receives the reward $r_t$ and the next state $s_{t+1}$. At the same time, the transition tuple $\left(s_t,a_t,r_t,s_{t+1}\right)$ was stored in the experience replay buffer to train the six neural networks. The intelligent caching updating process terminates when the maximum number of training steps is reached.

Algorithm 2: TD3-based optimization for caching updating.

5. Simulation and Results

In this section, to evaluate the task-partition-based intelligent offloading scheme in heterogeneous cache-assisted cloud–edge cooperation environments, the simulation environment consisted of one cloud, 4 BSs, and 12 MUs. In the simulation, all solutions were evaluated from the perspectives of offloading modes, caching policies, and task partition to demonstrate the merits of our proposed model:

• Offloading policy: All schemes were evaluated in different offloading modes, e.g., DRL-based offloading and the popular open shortest path first (OSPF) routing in the cache-aided cloud–edge collaboration system.
• Caching policy: All schemes were evaluated in different caching policies, where two typical and efficient caching policies, denoted by “Pop” and “TD3”, were utilized in cache-enabled asymmetrical cloud–edge cooperation networks. In “Pop”, each BS cooperatively caches network contents according to the known content popularity at the beginning of the simulation and does not update its content caching state. In “TD3”, each BS adjusts its content caching state before entering the next slot according to the historical access frequencies of tasks sent by end-users.
• Task partition: All schemes were evaluated by dividing tasks or not to explore the effect of task partition on offloading and content caching.

Figure 4 shows the network latency for all solutions at different cache sizes for each BS. As shown in Figure 4, the performance of all solutions improves when the cache size increases. The reason for this is that, given a larger cache size, more popular content is stored on edge nodes, shortening the transmission path for user tasks and reducing network latency for all solutions. At the same time, the increase in the cache hit ratio reduces the number of lost packets and reduces the performance gap between tasks and task-offloading schemes. As the cache size increases and the cache hit ratio increases, the performance gap between DQN intelligent routes and OSPF routes becomes less obvious. However, considering that “Pop” can achieve the best content cache based on the known content popularity, the TD3 smart cache gradually learns according to the current network cache state and the return value of previous decisions. As the cache size increases, the network cache state dimension becomes larger, and the learning effect of TD3 agents deteriorates. The TD3 smart cache strategy becomes less and less accurate, so the performance gap between solutions with “Pop” and “TD3” widens with increasing cache sizes.

Figure 5 shows the network latency for all solutions as the amount of different network content varies. As shown in Figure 5, an increase in the diversity of network content means a decrease in the number of task requests for popular content, which reduces the cache hit rate of each BS, which in turn increases the number of packet drops and increases the network latency of all solutions. At the same time, the increase in the diversity of network content causes more content to go to the cloud node for uninstallation. As a result, the link transmission burden increases and the number of packets increases. The packet loss delay of tasks is much larger than that of subtasks, so the performance gap between tasks and subtask uninstallation schemes increases. At the same time, with the increase in the diversity of network content, users send more requests to obtain unwelcome content, and the cache hit rate of each BS will gradually decreases. The DQN intelligent offloading scheme was getting better and better than the OSPF scheme, so the performance gap between them became larger. In addition, as the amount of different network content grew, the performance gap between solutions with “Pop” and “TD3” increased because the TD3 smart cache gradually learned based on the current network cache state and the return value of previous decisions, and the accuracy of the content popularity decreased as content diversity increased. As the learning effect of TD3 agents deteriorated, the accuracy of the TD3 smart cache strategy became worse and worse, so the increasing content diversity had a greater impact on the cache hit ratio of “TD3” than that of “Pop”.

Figure 6 shows the network latency for all solutions as content popularity changes. As Figure 6 shows, the performance of all solutions increased as the popularity of content increased. The reason for this is that, as the popularity of content increased, users sent more task requests to obtain the popular content, thus increasing the cache hit rate in a given BS cache size, and therefore, the network latency for all solutions decreased. At the same time, the content popularity increased, the cache hit ratio improved, the number of packet losses reduced, the utilization of task segmentation network resources was not very obvious, and the performance gap between task and subtask offloading schemes narrowed.

At the same time, the increase in content popularity improved the cache hit rate of each BS, and the performance gap between the DQN intelligent uninstall scheme and the OSPF scheme became less and less obvious, so the gap became smaller. However, as the cache size grew, the performance gap between solutions with “Pop” and “TD3” first increased and then decreased. When the content popularity was small, the reason why the gap between “Pop” and “TD3” was small is that the content in the network was requested, and the TD3 smart cache gradually learned according to the current network cache state and the return value of previous decisions, so the TD3 smart learning effect became worse, and the accuracy of TD3 smart cache strategy became very poor. At this time, no matter what caching method the BS used, the performance was similar, and it would go to the cloud node to find the requested content. With the gradual increase in the popularity of content, the reason why the gap between “Pop” and “TD3” would become larger is that the number of requests for different content was not very big. However, because the Pop cache was the most popular content, the TD3 intelligent learning effect was not very good, and the accuracy of the TD3 intelligent caching strategy was not very high. Therefore, the performance gap between the TD3 smart cache and Pop slowly opened up. When the content’s popularity increased to a certain extent, the gap between “Pop” and “TD3” narrowed because users requested more popular content. The TD3 intelligent cache gradually learned according to the current network cache state and the returned the value of previous decisions. The TD3 intelligent learning effect was very good, and it more easily perceived the popular content. When the accuracy of the TD3 smart cache policy gradually improved, the performance gap between the TD3 smart cache policy and Pop gradually narrowed.

Figure 7 shows the network latency for all solutions as the number of packets sent per user changed. As shown in Figure 7, as the number of packets sent by each user increased, the total number of requests in the network increased, thereby increasing the network latency of all solutions. At the same time, as the number of packets sent by each user increased, the link transmission burden on the network increased, and the number of packets lost increased. After the task was divided, the utilization of network resources was higher than that of the undivided task, and the packet loss delay of the undivided task was much larger than that of the subtask. As a result, the performance gap between the task and the subtask uninstallation scheme increased. As the number of requests increased, the link transmission burden on the network increased. Compared with OSPF, the DQN can detect the network link status and perform intelligent pathfinding better. In addition, the DQN pathfinding had no extra delay for one request, while OSPF pathfinding generated an extra delay for each request. However, as the number of user requests increased, the training times of TD3 agents increased, and the accuracy of the TD3 smart cache policy gradually improved. As a result, the gap between the cache hit rate of the TD3 smart cache policy and the cache hit rate based on the popularity (Pop) cache policy became smaller.

Figure 8 shows the network latency of all solutions as the transmit power of each BS changed. As shown in Figure 8, with the increase in transmit power of each BS, the channel SINR of the downlink between the BS and its service users improved, the link transmission rate increased, and the packet loss rate and network delay of all solutions significantly reduced. At the same time, with the increase in transmit power of each BS, the resource utilization rate of subtask offloading was higher. With the improvement of the SINR of the downlink channel, the link transmission rate increased, so the performance gap between the task and the subtask offloading scheme increased. As the transmission power of each BS increased, the network delay gap between the DQN intelligent uninstall and OSPF solutions gradually narrowed due to the optimal offload decision in the resource-constrained cloud edge collaboration environments. However, as the transmit power of each BS increased, the increase in the transmit power of the BS had no effect on the channel SINR of the uplink between the BS and its access terminal, leaving the recorded historical access information of the BS almost unchanged, so the performance gap between the solutions with “Pop” and “TD3” barely widened.

Figure 9 shows the network delay of all solutions as the transmit power of each MC changed. As shown in Figure 9, with the increase in the transmit power of each MC, the channel SINR of the uplink between the user and its connected BS improved, the link transmission rate increased, and the packet loss rate and network delay of all solutions significantly reduced. At the same time, with the increase in transmit power of each MC, the resource utilization rate caused by subtask offloading was higher. With the improvement of the uplink channel SINR, the link transmission rate increased, so the performance gap between the task and the subtask offloading scheme increased. As the transmission power of each MC increased, due to the optimal offloading decision in the resource-constrained cloud edge collaboration environment, the DQN intelligent offloading and OSPF solutions could quickly send the requested content to the BS, thus reducing the packet loss rate and the additional delay caused by OSPF packet loss, so the network delay gap between them gradually narrowed. However, as the transmit power of each MC increased, it affected the channel SINR of the uplink between BS and its access terminal, indirectly affecting the transmission rate of the uplink, but it did not affect the user’s interest preference, so the performance gap between the solutions with “Pop” and “TD3” hardly widened.

Figure 10 shows the average weighted reward for all task requests per slot in a DQN agent with different learning rates. As shown in Figure 10, the “DQN” solution at different learning rates always converged quickly and performed best when the learning rate was 0.0003. This is because the large learning rate indicates that, when the current system DQN task offloading decision is made, the old Q-value will have a weak influence on the new Q-value, and the weight update was less affected by the old value and more dependent on the new value. Therefore, the learning rate was neither larger nor smaller; only the appropriate learning rate would make the solution perform best.

Figure 11 shows the average weighted reward of all task requests per time slot for different learning rates in the TD3 agent. As shown in Figure 11, the “TD3” smart cache solution at different learning rates always converged quickly and performed best when the learning rate was 0.0003. This is because a larger learning rate means that the step length of the weight update of the actor network is larger, which indicates that the actor network will adjust the strategy faster every time the weight is updated, making the actor network more sensitive and adjusting the actions taken in each state more quickly. This can lead to more rapid changes in caching policies and faster exploration of different caching policies. However, the excessive learning rate may also lead to the instability of the strategy, making it difficult for the “TD3” algorithm to converge. So, bigger is not better, and smaller is not better; only the right learning rate will make the caching solution perform best.

6. Conclusions

In this paper, a joint optimization of task offloading and content caching is proposed aimed at enhancing resource allocation efficiency in task-partition-based asymmetrical cloud–edge cooperation networks. First, we considered a joint optimization of task offloading and content caching in task-partition-based asymmetrical cloud–edge cooperation networks and formulated the problem as mixed-integer non-linear programming, which aims at minimizing the network delay of the system. Subsequently, utilizing network state information, we designed a new cooperative TD3-DQN algorithm for making joint optimization decisions of computation offloading and content caching. According to numerical results, the proposed TD3-DQN algorithm can significantly reduce the network latency of the system and can outperform some other benchmark algorithms under different scenarios.

In future work, we will explore the impact of caching different items (e.g., data, models, and content) on the computing tasks. Moreover, it is also worth investigating the use of multi-agent DRL by deploying multi-agent cooperative control networks at different nodes of the network. Finally, exploring the collaborative control of discrete and continuous variables in cloud edge networks by a single agent is also a future research direction.