D2D-Assisted Multi-User Cooperative Partial Offloading in MEC Based on Deep Reinforcement Learning

Abstract

Mobile edge computing (MEC) and device-to-device (D2D) communication can alleviate the resource constraints of mobile devices and reduce communication latency. In this paper, we construct a D2D-MEC framework and study the multi-user cooperative partial offloading and computing resource allocation. We maximize the number of devices under the maximum delay constraints of the application and the limited computing resources. In the considered system, each user can offload its tasks to an edge server and a nearby D2D device. We first formulate the optimization problem as an NP-hard problem and then decouple it into two subproblems. The convex optimization method is used to solve the first subproblem, and the second subproblem is defined as a Markov decision process (MDP). A deep reinforcement learning algorithm based on a deep Q network (DQN) is developed to maximize the amount of tasks that the system can compute. Extensive simulation results demonstrate the effectiveness and superiority of the proposed scheme.

1. Introduction

In recent years, with the development of wireless networks and the popularity of smart mobile devices, mobile applications such as augmented reality (AR), virtual reality (VR), and facial recognition payment have grown exponentially [1,2]. These applications tend to be computation intensive and require low latency, but the battery capacities, computation resources, and storage capacities of mobile user equipment (UE) are very limited. As a result, most emerging applications may not be suitable for local execution on mobile devices [3,4]. To address this daunting challenge, the functions of the central network are increasingly moving towards the edge of the network [4].

Mobile edge computing (MEC) is regarded as a promising paradigm. This technique moves service platforms with computing, storage, and communication capabilities to the edge node (base station (BS)) nearest to the mobile devices [5,6,7,8,9]. MEC allows resource-constrained mobile terminals to migrate part or all of the complex applications to the edge cloud, becoming a low-latency, low-energy, and efficient solution [10,11,12]. However, the heterogeneous characteristics of BSs and MEC servers and the limited resources of communication and edge computing bring challenges to computational offloading methods [13].

Computing offloading is one of the key issues in MEC [14,15]. Its main task is to plan the offloading scheme of computing tasks and the allocation scheme of computing resources to reduce delays, save energy consumption, and improve computing resource utilization. There are two main task offloading strategies. In full offloading, each task can be completely offloaded to the resource device or run completely locally [2,16,17]. In partial offloading, each task can be partially offloaded to the resource device and partially left local [18,19,20].

Although the above studies have demonstrated the advantages of MEC computation offloading for improving wireless network computing performance, the limited computing resources of BSs are not always sufficient to support all mobile devices within their coverage. To solve this problem, some work studies offload computation to the neighboring devices through device-to-device (D2D) communication links [21,22,23,24,25]. The authors of [21] believe that each mobile user has its corresponding mobile peer; thus, the computing task can be partially offloaded to the edge cloud or its mobile peer. Successive convex approximation (SCA) and geometric programming (GP) are used to solve the energy minimization problem. Reference [22] solves the overall energy consumption problem of the system under the three-layer network architecture integrating cloud computing, MEC, and D2D communication through game theory. Reference [23] regards one near computing device with sufficient resources near each mobile user as a distributed computing node (DCN), and the mobile user can connect to the associated DCN through D2D. The problem of complete offloading of computing tasks between local, edge cloud, and DCN is studied relying on game theory, thereby reducing the delay and energy consumption of the system.

Although the above-mentioned research results have laid a solid foundation for utilizing the computing resources of idle devices in the MEC, the utilization efficiency of resources on idle devices has yet to be improved. By assuming that each mobile user is associated with a distributed computing node (DCN), Ref. [23] does not consider the DCN selection. Ref. [20] uses idle devices for D2D-assisted transmission and optimizes transmission scheduling, which does not use idle device resources for computing assistance. Although [25,26,27] optimizes the pairing between TD and RD, they stipulate that each device with an idle resource can only offer a D2D offloading service to one other device at most, and limits the pairing method between devices to one to one. Ref. [28] utilizes the resources of idle devices for computing assistance through D2D communication, and assumes that each idle device can accept multiple computing tasks. However, complete offloading is considered in this paper. Additionally, Ref. [28] assumes that when multiple computation tasks are offloaded to the same idle device, the tasks will share the computation resource equally.

Most of the works on MEC computing offload are aimed at reducing latency, saving energy, reducing energy efficiency, etc., and many state-of-the-art works only consider integrating multiple indicators for optimization [27,29]. However, the computing capability of the system can also be determined by the number of devices the system can serve. Because reduction in latency is often accompanied by the consumption of computing resources in the system, the number of devices supported by the system can be maximized subject to the constraint of maximum delay. In this way, the utilization efficiency of computing resources can be further improved without affecting user experience. As a result, the proposed scheme can cope with the continuous increase in the number of devices in the 6G network. In addition, traditional methods, such as dynamic programming, game theory, and GP, are often used to reduce the delay and energy consumption in full offloading, since such problems can be expressed as mixed-integer programming (MIP).

Compared with traditional optimization methods, reinforcement learning (RL) can deal with more complex MEC problems [30]. Reference [16] uses Q-learning and a deep Q network (DQN) to solve the binary offloading problem between the local and edge clouds in dynamic systems, and addresses the task assignment problem in a dynamic vehicular fog computing (VFC) environment. It simultaneously considers task priority, vehicle service availability, and computational resource sharing incentives, as well as using a soft actor–critic algorithm to maximize the utility of offloaded tasks.

In this paper, we consider adaptive user association, partial offloading, and resource allocation among multiple user devices and idle devices in a small area covered by a single BS. Unlike traditional algorithms considering latency and energy consumption, we maximize the computing power of the entire system. An improved algorithm based on DQN is presented to solve the proposed problem. The main contributions of this paper are summarized as follows:

• We construct a D2D-MEC framework that combines D2D communications and MEC technology. The user equipment with limited computational capability can offload part of its computation-intensive tasks to the MEC server located in the BS and the idle equipment nearby, and the allocation of computing resources is the responsibility of the BS. In order to maximize the computing power of the whole system under the condition of limited computing resources, we propose an MEC framework including partial offloading, resource allocation, and user association under the maximum delay constraint of the application.
• We propose an optimization problem with constraints on both delay and computational resources, which is NP hard. By analyzing the internal structure of the optimization problem, it is decomposed into two sub-problems. We prove that the optimal solutions of the two sub-problems constitute the optimal solution of the original problem. The convex optimization is employed to solve the optimal solution of the first subproblem. The second subproblem is described as a Markov decision process (MDP) used to maximize the amount of tasks calculated by the system, in which offload decisions, resource allocation, and user association are determined simultaneously. A DQN-based modeless reinforcement learning algorithm is proposed to maximize the objective function.
• Extensive simulations demonstrate that the proposed algorithm outperforms traditional MEC schemes, Q-learning, DQN, and other conventional algorithms under different system parameters.

The rest of this paper is organized as follows. In Section 2, we present the system model, including a network model, a channel model, a computation model, and problem formulation. In Section 3, we decompose the original problem into two subproblems. In Section 4, we propose a reinforcement learning algorithm based on DQN to solve subproblem 2. In Section 5, we show the simulation results. Finally, we conclude this study in Section 6.

2. System Model

2.1. Network Model

As shown in Figure 1, the scenario we consider consists of a single BS equipped with an edge cloud server and mobile devices within the coverage area of the BS. A mobile device sends data to the BS through the cellular network and sends data to a nearby mobile device through D2D. Within the range of D2D communication, the mobile devices are divided into Task Devices (TDs), denoted as $\mathcal{U}=\left\{u_i\right|i=1,2,\dots ,\mathrm{U}\}$, and D2D Resource Devices (D2D RDs), represented by $k_i,i=1,2,\dots ,\mathrm{K}$. Set $\mathcal{K}=\left\{k_i\right|i=0,1,2,\dots ,\mathrm{K}\}$ represents all resource devices, where $k_0$ is the BS. RDs can provide computing resources for tasks on TDs. The same as [31,32,33], the applications considered in this study are all oriented to data partitioning. The computing task on TD can be arbitrarily divided into three parts, and the computation is performed in parallel on the local, edge cloud, and a D2D RD at the same time.

We divide the system time into several time slots. The system state is constant in time slots, but changes between slots. Each slot BS allocates computing resources. The computing task on $i\in \mathcal{U}$ is denoted as ${\varphi }_i=\{{\mathrm{Q}}_i,{\mathrm{C}}_i,{\tau }_i,f_i\}$, where ${\mathrm{Q}}_i$ is the size of the task data, ${\mathrm{C}}_i$ indicates the number of CPU cycles per bit of data, representing the computational complexity of the application, ${\tau }_i$ is the maximum delay, and $f_i$ is the local computing capacity. Computing resources of $k_i\in \mathcal{K}$ are denoted by ${\mathrm{F}}_i,i=0,1,2,\dots ,\mathrm{K}$.

2.2. Channel Model

We assume that the wireless channel state remains constant when each computation task is transmitted between the TD and RD. The transmission rate between the TD i and the BS is calculated by

$$R_{i{k}_0}=B_{i{k}_0}lo{g}_2(1+\frac{p_i^c{\left|h_{i{k}_0}\right|}^2}{N_0}),\forall i\in \mathcal{U}.$$

(1)

The transmission rate between TD i and D2D RD j is calculated by

$$R_{ij}=B_{ij}lo{g}_2(1+\frac{p_i^d{\left|h_{ij}\right|}^2}{N_0}),\forall i\in \mathcal{U},\forall j\in \mathcal{K}/k_0$$

(2)

where $h_{ij}$ is the channel power gain between TD i and RD j; $B_{ij}$ is the bandwidth allocated to the cellular channel or D2D channel between TD i and RD j; $p_i^c$ is the transmission power of cellular from TD i to BS; and $p_i^d$ is the transmission power of D2D from TD i to a D2D RD. Since the two powers are limited by the maximum uplink power $p_i^{max}$ of TD, they are subject to constraints:

$$\begin{array}{cc}\hfill & 0\le p_i^c\le p_i^{max},\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & p_i^d=p_i^{max}-p_i^c,\forall i\in \mathcal{U}.\hfill \end{array}$$

(4)

2.3. Computation Model

The computing task on TD is divided into three parts, which are computed on a local, edge cloud, and D2D RD, respectively. $x_{ij}\in \{0,1\},\forall i\in \mathcal{U},\forall j\in \mathcal{K}/k_0$ is the user association between TD i and RD j, $x_{ij}=1$ indicates that TD i offloads part of the computing task to D2D RD j, and otherwise, $x_{ij}=0$. Since a TD selects, at most, one D2D RD for computational offloading, there are constraints: ${\sum }_{j=k_1}^{k_{\mathrm{K}}}{x}_{ij}\le 1,\forall i\in \mathcal{U}$. Let ${\alpha }_i\in [0,1],{\beta }_i\in [0,1],i\in \mathcal{U}$ denote the proportion of a computing task on TD i that is offloaded to the edge cloud and D2D RD, respectively. Since the locally computed ratio should be non-negative, ${\alpha }_i$ and ${\beta }_i$ should satisfy the constraint: $0\le {\alpha }_i+{\beta }_i\le 1$. Let $f_{ij},\forall i\in \mathcal{U},\forall j\in \mathcal{K}$ denote the computational resource allocated by RD j to TD i. Since RDs have limited computing resources, there are constraints: ${\sum }_{i=u_1}^{u_{\mathrm{U}}}{x}_{ij}{f}_{ij}\le {\mathrm{F}}_j,\forall j\in \mathcal{K}$.

• Local Computing: The local computation delay of the task on TD i can be computed as

  $$D_i^{l,c}=\frac{(1-{\alpha }_i-{\beta }_i){\mathrm{Q}}_{\mathrm{i}}{\mathrm{C}}_{\mathrm{i}}}{f_i}.$$

  (5)
• Edge Computing: The total latency of edge computing on TD i consists of three parts: (1) time for uploading computing tasks $D_i^{e,t}$, (2) time for executing tasks on the MEC server $D_i^{e,c}$, and (3) time for downloading computing results. Similar to [31,34], this study ignores the delay of sending results back to TDs from MEC server. This is because the size of the results is usually much smaller than the size of the transmitting data. Therefore, according to Equation (1), the delay of TD i to complete edge cloud computing can be computed as

  $$D_i^e=D_i^{e,t}+D_i^{e,c}=\frac{{\alpha }_i{\mathrm{Q}}_i}{R_{i{k}_0}}+\frac{{\alpha }_i{\mathrm{Q}}_i{\mathrm{C}}_i}{f_{i{k}_0}}.$$

  (6)
• D2D RDs Computing: Similar to edge cloud computing, the delay of TD i to complete D2D RD computing can be obtained by (1) D2D transmission delay $D_i^{D,t}$ and (2) remote-execution delay $D_i^{D,c}$

  $$D_i^D=D_i^{D,t}+D_i^{D,c}=\sum _{j=k_1}^{k_{\mathrm{K}}}{x}_{ij}(\frac{{\beta }_i{\mathrm{Q}}_i}{R_{ij}}+\frac{{\beta }_i{\mathrm{Q}}_i{\mathrm{C}}_i}{f_{ij}}).$$

  (7)

Therefore, according to Equations (5)–(7), the total delay for completing the task ${\varphi }_i$ on TD i is:

$$D_i=max\{D_i^{l,c},D_i^e,D_i^D\}.$$

(8)

2.4. Problem Formulation

In this paper, we improve the computing capability of the whole system, and the computing capability of the system is reflected by the number of devices supported by the system. Under the constraints of computing resources and maximum delay, the number of devices served by the system indicates the computing capability of the system [25].

We regard the number of TDs that can complete the computing task as the optimization target, and $o_{u_i}=\left\{\begin{array}{cc}\hfill 1& \mathrm{if}{D}_i\le {\tau }_i,\hfill \\ \hfill 0& \mathrm{if}{D}_i>{\tau }_i,\hfill \end{array}\right.$ is the completion of the computing task on TD i. $o_{u_i}=1$ indicates that the task is completed within the maximum delay; otherwise, $o_{u_i}=0$. The list of notations is given in abbreviations part. The optimization problem of this study can be formulated as:

$$\begin{array}{}\mathrm{(9)}& \mathcal{P}1:& \underset{\{\mathit{x},\alpha ,\beta ,\mathit{f}\}}{max}\sum _{i=1}^{\mathrm{U}}{o}_{u_i},\mathrm{(10)}& \mathrm{s}.\mathrm{t}.& x_{ij}\in \{0,1\},\forall i\in \mathcal{U},\forall j\in \mathcal{K}/k_0,\mathrm{(11)}& & \sum _{j=k_1}^{k_{\mathrm{K}}}{x}_{ij}\le 1,\forall i\in \mathcal{U},\mathrm{(12)}& & 0\le {\alpha }_i\le 1,\forall i\in \mathcal{U},\mathrm{(13)}& & 0\le {\beta }_i\le 1,\forall i\in \mathcal{U},\mathrm{(14)}& & 0\le {\alpha }_i+{\beta }_i\le 1,i\in \mathcal{U},\mathrm{(15)}& & \sum _{i=u_1}^{u_{\mathrm{U}}}{x}_{ij}{f}_{ij}\le {\mathrm{F}}_j,\forall j\in \mathcal{K}/k_0\mathrm{(16)}& & \sum _{i=u_1}^{u_{\mathrm{U}}}{f}_{i{k}_0}\le {\mathrm{F}}_0,\end{array}$$

where $\mathit{x}=\{x_{u_1{k}_1},x_{u_1{k}_2},\dots ,x_{u_1{k}_{\mathrm{K}}},\dots ,x_{u_{\mathrm{U}}{k}_1},x_{u_{\mathrm{U}}{k}_2},\dots ,x_{u_{\mathrm{U}}{k}_{\mathrm{K}}}\}$ denotes the user association vector between the TD and the D2D RD; $\alpha =\{{\alpha }_{u_1},{\alpha }_{u_2},\dots ,{\alpha }_{u_{\mathrm{U}}}\}$ is the offloading decision vector of edge cloud computing; $\beta =\{{\beta }_{u_1},{\beta }_{u_2},\dots ,{\beta }_{u_{\mathrm{U}}}\}$ is the offloading decision vector of D2D RDs computing; and $\mathit{f}=\{f_{u_1{k}_0},f_{u_1{k}_1},\dots ,f_{u_1{k}_{\mathrm{K}}},\dots ,f_{u_{\mathrm{U}}{k}_0},f_{u_{\mathrm{U}}{k}_1},\dots ,f_{u_{\mathrm{U}}{k}_{\mathrm{K}}}\}$ is the allocation decision of computing resources on RDs. Additionally, (10) indicates that the user-associated variable is a binary variable, $x_{ij}=1$ indicates that TD i offloads the task part to D2D RD j, or $x_{ij}=1$. (11) ensures that a TD can only select one of multiple D2D RDs devices for task offloading. Meanwhile, (12)–(14) indicate that the cloud, D2D RD, and local data ratios are all positive and cannot exceed one. Additionally, (15) and (16) ensure that RD j cannot allocate more computing resources to all TDs than its maximum computing capability.

Theorem 1.

$\mathcal{P}1$ is an NP-hard problem.

Proof. 

See Appendix A.    □

Theorem 1 shows that $\mathcal{P}1$ is a NP-hard problem, and the object function of (9) is non-convex, which is a difficult problem to solve. In order to solve $\mathcal{P}1$, we first decompose and simplify the problem, and then solve it using a reinforcement learning method instead of a conventional optimization method.

3. Problem Decomposition

We maximize the number of TDs served by the system under the constraints of limited computing resources. The requirement for the completion of tasks on TDs is that the computing time is less than the maximum delay. According to $D_i^{e,c}$ in Equation (6), the computing resource $f_{i{k}_0}$ allocated by the edge server to TD i is proportional to the data size ${\alpha }_i$. Similarly, according to $D_i^{D,c}$ in Equation (7), the computational resource $f_{ij}$ allocated by D2D RD j to TD i is proportional to the data size ${\beta }_i$. Based on the above analysis, reducing the value of ${\alpha }_i+{\beta }_i$ can alleviate computing resource TD i, which is occupied for the entire system. As a result, the system can have more computing resources to provide services for other TDs.

Based on the above analysis, in order to solve $\mathcal{P}1$, we first determine the value of ${\alpha }_i+{\beta }_i,\forall i\in \mathcal{U}$ and the resource allocation scheme $\mathit{f}$, and then determine ${\alpha }_i,{\beta }_i$ and user association $\mathit{x}$. Finally, we prove that the optimal solution obtained in this way is the same as the optimal solution of $\mathcal{P}1$. Define variables ${\gamma }_i={\alpha }_i+{\beta }_i$, ${\alpha }_i^{{}^{\prime }}=\frac{{\alpha }_i}{{\alpha }_i+{\beta }_i}$, ${\beta }_i^{{}^{\prime }}=\frac{{\beta }_i}{{\alpha }_i+{\beta }_i},\forall i\in \mathcal{U}$. We have ${\alpha }_i={\gamma }_i{\alpha }_i^{{}^{\prime }}$, ${\beta }_i={\gamma }_i{\beta }_i^{{}^{\prime }}$, and ${\beta }_i^{{}^{\prime }}=1-{\alpha }_i^{{}^{\prime }}$. The variables in $\mathcal{P}1$ change to $\{\gamma ,{\alpha }^{{}^{\prime }},\mathit{x},\mathit{f}\}$.

In $\mathcal{P}2$, the values of $\{{\alpha }^{{}^{\prime }},\mathit{x}\}$ are fixed, and the optimal $\{\gamma ,\mathit{f}\}$ are calculated. The goal of optimization is to minimize the demand for computing resources of the system. The mathematical formulation of the problem is expressed as

$$\begin{array}{}\mathrm{(17)}& \mathcal{P}2:& \underset{\{\gamma ,\mathit{f}\}}{min}\sum _{j=k_0}^{k_{\mathrm{K}}}\sum _{i=u_1}^{u_{\mathrm{U}}}{x}_{ij}{f}_{ij},\mathrm{(18)}& \mathrm{s}.\mathrm{t}.& 0\le {\gamma }_i\le 1,\forall i\in \mathcal{U},\mathrm{(19)}& & \frac{(1-{\gamma }_i){\mathrm{C}}_i}{f_i}\le {\tau }_i,\forall i\in \mathcal{U},\mathrm{(20)}& & {\gamma }_i{\alpha }_i^{{}^{\prime }}{\mathrm{Q}}_i(\frac{1}{R_{i{k}_0}}+\frac{{\mathrm{C}}_i}{f_{i{k}_0}})\le {\tau }_i,\forall i\in \mathcal{U},\mathrm{(21)}& & {\gamma }_i(1-{\alpha }_i^{{}^{\prime }}){\mathrm{Q}}_i\sum _{j=k_1}^{k_{\mathrm{K}}}{x}_{ij}(\frac{1}{R_{ij}}+\frac{{\mathrm{C}}_i}{f_{ij}})\le {\tau }_i,\forall i\in \mathcal{U},\end{array}$$

where constraint (18) is set according to (14), and (19)–(21) represent constraints on local computing delay $D_i^{l,c}$, edge cloud computing delay $D_i^e$, and D2D computing delay $D_i^D$, respectively.

Theorem 2.

The optimal solution of $\mathcal{P}2$ is given by $\{{\gamma }^{*},{\mathit{f}}^{*}\}$:

$$\begin{array}{cc}\hfill & {\gamma }_i^{*}=1-\frac{{\tau }_i{f}_i}{{\mathrm{C}}_i},\forall i\in \mathcal{U},\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill & {f_{i{k}_0}}^{*}=\frac{{\mathrm{C}}_i{\alpha }_i^{{}^{\prime }}{\gamma }_i^{*}{\mathrm{Q}}_i{R}_{i{k}_0}}{{\tau }_i{R}_{i{k}_0}-{\alpha }_i^{{}^{\prime }}{\gamma }_i^{*}{\mathrm{Q}}_i},\forall i\in \mathcal{U},\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill & f_{ij}^{*}=\frac{{\mathrm{C}}_i{\gamma }_i^{*}{\mathrm{Q}}_i{R}_{ij}(1-{\alpha }_i^{{}^{\prime }})}{{\tau }_i{R}_{ij}-{\gamma }_i^{*}(1-{\alpha }_i^{{}^{\prime }}{\mathrm{Q}}_i)},\forall i\in \mathcal{U},\forall j\in \mathcal{K}.\hfill \end{array}$$

(24)

Proof. 

See Appendix B.    □

It can be observed that ${\gamma }_i^{*}$ is a constant independent of $\{\mathit{x},\alpha \}$, and $f_{ij}^{*}$ can be represented by variable ${\alpha }_i^{{}^{\prime }}$. Therefore, substituting the solution of $\mathcal{P}2$ into $\mathcal{P}1$ can obtain:

$$\begin{array}{}\mathrm{(25)}& \mathcal{P}3:& \underset{\{\mathit{x},{\alpha }^{{}^{\prime }}\}}{max}\sum _{i=1}^{\mathrm{U}}{o}_{u_1}^{{}^{\prime }},\hfill \mathrm{(26)}& \mathrm{s}.\mathrm{t}.& x_{ij}\in \{0,1\},\forall i\in \mathcal{U},j\in \mathcal{K}/k_0,\mathrm{(27)}& & \sum _{j=k_1}^{k_{\mathrm{K}}}{x}_{ij}\le 1,\forall i\in \mathcal{U},\mathrm{(28)}& & 0\le {\alpha }_i^{{}^{\prime }}\le 1,\forall i\in \mathcal{U},\mathrm{(29)}& & \sum _{i=u_1}^{u_{\mathrm{U}}}{x}_{ij}{f}_{ij}\le F_j,\forall j\in \mathcal{K},\end{array}$$

where $o_{u_i}^{{}^{\prime }}$ is obtained by

$$\begin{array}{cc}\hfill & o_{u_i}^{{}^{\prime }}=\left\{\begin{array}{cc}\hfill 1& \mathrm{if}max\{D_i^{e*},D_i^{D*}\}\le {\tau }_i,\hfill \\ \hfill 0& \mathrm{if}max\{D_i^{e*},D_i^{D*}\}>{\tau }_i,\hfill \end{array}\right.\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill & D_i^{l,c*}=\frac{(1-{\gamma }_i^{*}){\mathrm{C}}_i}{f_i}={\tau }_i\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill & D_i^{e*}={\gamma }_i^{*}{\alpha }_i^{{}^{\prime }}{\mathrm{Q}}_i(\frac{1}{R_{i{k}_0}}+\frac{{\mathrm{C}}_i}{{f_{i{k}_0}}^{*}}),\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill & D_i^{D*}={\gamma }_i^{*}(1-{\alpha }_i^{{}^{\prime }}){\mathrm{Q}}_i\sum _{j=k_1}^{k_{\mathrm{K}}}{x}_{ij}(\frac{1}{R_{ij}}+\frac{{\mathrm{C}}_i}{f_{ij}^{*}}).\hfill \end{array}$$

(33)

Theorem 3.

The optimal solutions $\{{\gamma }^{*},{\mathit{f}}^{*}\}$,$\{{\mathit{x}}^{*},{\alpha }^{{}^{\prime }*}\}$ obtained by $\mathcal{P}2$ and $\mathcal{P}3$ are the optimal solutions of $\mathcal{P}1$.

Proof. 

See Appendix C.    □

4. DQN-Based Computation Offloading

In order to solve problem $\mathcal{P}3$, we express it as a MDP, which can be solved using model-free reinforcement learning, and propose an improved reinforcement learning algorithm based on DQN. Compared with the conventional DQN, the improved reinforcement learning algorithm can enable the agent to learn a better solution and converge to solve the problems. In this paper, we improve the number of task users that the system can serve by optimizing the offloading strategy and the allocation of computing resources in the system. Obviously, the optimal solution is unknown, so the agent can easily fall into a sub-optimal solution after finding a feasible allocation scheme and obtaining a positive reward. Therefore, in order to make the agent constantly search for a better solution, we compare and update the recorded action trajectory during the learning process, and then make random selection with a certain probability under the condition that the agent learns a lot of the recorded action trajectory, so that the agent has the opportunity to learn a better solution. In addition, we set the reward value given by the environment when learning the recorded action trajectory. This is different from using the Q-network to decide the action, which is used for fast convergence.

We call the proposed algorithm DQN-PTR, which integrates the priority action trajectory replay method into DQN. In this section, we first define the three key elements of the MDP problem, i.e., the state space, action space, and reward function. Then, we explain the detailed implementation of our proposed algorithm for reinforcement learning.

4.1. Three Key Elements for MDP

• State Space:
  At each time slot, the agent observes and collects all device information within the range of the BS. At step t, the state of the system consists of two parts: ${\mathit{s}}_{\mathit{t}}=\{\mathit{F}\left(\mathit{t}\right),\varphi \left(\mathit{t}\right)\}$. Among them, $\mathit{F}=\{F_0\left(t\right),F_1\left(t\right),\dots ,F_{\mathrm{K}}\left(t\right)\}$ represents the computing resources of all resource devices in the system, and $\varphi \left(\mathit{t}\right)=\{{\varphi }_{u_1},{\varphi }_{u_2},\dots ,{\varphi }_{u_{\mathrm{U}}}\}$ where ${\varphi }_i\in \{0,1\},i\in \mathcal{U}$ represents the completion of the tasks on the task devices. We define ${\mathit{s}}_{\mathbf{0}}$ as the system state observed by the BS at the beginning of the time slot; that is, ${\mathit{s}}_{\mathbf{0}}=\{F_0,F_1,\dots ,F_{\mathrm{K}},0,0,\dots ,0\}$.
• Action Space:
  The action space consists of two parts: ${\mathit{A}}_{\mathit{t}}=\{\mathit{x},{\alpha }^{{}^{\prime }}\}$, where $\mathit{x}=\{{\mathit{x}}_{{\mathbf{u}}_{\mathbf{1}}},{\mathit{x}}_{{\mathbf{u}}_{\mathbf{2}}},\dots ,{\mathit{x}}_{{\mathbf{u}}_{\mathrm{U}}}\}$, ${\mathit{x}}_{\mathbf{i}}=\{x_{i{k}_1},x_{i{k}_2},\dots ,x_{i{k}_{\mathrm{K}}}\},i\in \mathcal{U}$ represents the offload association between the TDs and the D2D RDs. ${\alpha }^{{}^{\prime }}$ is the task offload ratio of the currently assigned TDs. According to constraint (26) and (27), it is stipulated that action ${\mathit{a}}_{\mathit{t}}$ at step t satisfies condition $\sum \mathit{x}\left(\mathit{t}\right)=1$.
• Reward Function:
  The objective function of $\mathcal{P}3$ is the sum of the TD devices that complete the calculation. Considering that the size of the computing tasks of each device is different, to ensure the fairness of the evaluation, the reward function is defined as the sum of the size of the computing tasks completed in the current time slot:

  $$R({\mathit{s}}_{\mathit{t}},{\mathit{a}}_{\mathit{t}})=\left\{\begin{array}{cc}\hfill \sum _{i=1}^{\mathrm{U}}{o}_{u_i}{\mathrm{Q}}_i,& \mathrm{if}\sum _{i=1}^{\mathrm{U}}{o}_{u_i}>0,\hfill \\ \hfill -1,& else,\hfill \end{array}\right.$$

  (34)

In the reinforcement learning process, in each episode, as the environment performs the t step action, the state and action space of the next step also change. Assume that the action selected by the agent at step t is ${\mathit{a}}_{\mathit{t}}=\{x_{ij}=1,{\alpha }^{{}^{\prime }}\left(t\right)\}$. ${\mathit{a}}_{\mathit{t}}$ is executed by the environment when ${\mathit{s}}_{\mathit{t}},{\mathit{a}}_{\mathit{t}}$ satisfy the condition, given by

$$\begin{array}{cc}\hfill & {\varphi }_i=0,\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill & {\mathrm{Q}}_i{\mathrm{C}}_i{\gamma }_i{\alpha }^{{}^{\prime }}\left(t\right)\le F_j\left(t\right),\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill & {\mathrm{Q}}_i{\mathrm{C}}_i{\gamma }_i(1-{\alpha }^{{}^{\prime }}\left(t\right))\le F_0\left(t\right).\hfill \end{array}$$

(37)

With the successful execution of ${\mathit{a}}_{\mathit{t}}$, the remaining resources of the RDs and the TDs that have not allocated computing resources are reduced, and are given by

$$\begin{array}{cc}\hfill & F_j(t+1)=F_j\left(t\right)-{\mathrm{Q}}_i{\mathrm{C}}_i{\gamma }_i{\alpha }^{{}^{\prime }}\left(t\right),\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill & F_0(t+1)=F_0\left(t\right)-{\mathrm{Q}}_i{\mathrm{C}}_i(1-{\gamma }_i{\alpha }^{{}^{\prime }}\left(t\right)),\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill & {\mathit{A}}_{\mathit{t}+\mathbf{1}}={\mathit{A}}_{\mathit{t}}/{\mathit{x}}_{\mathbf{i}}.\hfill \end{array}$$

(40)

If any of the conditions (35)–(37) are not met, ${\mathit{a}}_{\mathit{t}}$ is infeasible, i.e., ${\mathit{s}}_{\mathit{t}+\mathbf{1}}={\mathit{s}}_{\mathit{t}}$, ${\mathit{A}}_{\mathit{t}+\mathbf{1}}={\mathit{A}}_{\mathit{t}}$.

4.2. Algorithm Design Based on DQN

The structure of the DQN-PTR algorithm proposed in this paper is shown in Figure 2, which mainly includes four parts: the environment, the replay buffer, the networks, and the trajectory record. The environment performs actions, computes rewards, and gives transitions to the state and the action space. The replay buffer stores the task offloading experiences, which are used to train the Q-Network. The network part includes two networks, which are used to predict the Q value and target the Q value, respectively. Network parameters are updated according to the differences between the two Q values. The recorded action trajectory is updated at the end of each training episode. If the result of the current episode is better than the recorded result, the action trajectory of the current episode is used to replace the originally recorded action trajectory.

We propose an DQN-PTR-based task offloading algorithm in Algorithm 1, and explain the main steps in detail, as follows:

• Initialize the action-value function $Q(\mathit{s},\mathit{a})$ and the target action-value function $\widehat{Q}(\mathit{s},\mathit{a})$ with parameters $\theta$ and ${\theta }^{{}^{\prime }}$, respectively. Initialize experiment replay buffer D to an empty set of size N.
• Initialize $ϵ=0$, and specify that the growth rate of $ϵ$ is ${ϵ}_{increment}=0.0001$ and grows to ${ϵ}_{max}=0.9999$. This parameter determines the probability of random selection when the agent selects an action, and the probability of random selection decreases with the update of the network parameters.
• Initialize the optimal trajectory record $\mathit{O}=⌀$ and the maximum total return value $R=0$.
• In each episode, the agent in the BS collects the ${\mathit{s}}_{\mathbf{0}}$. Initialize $R^{{}^{\prime }}=0$ to calculate the total return value of this episode. $\mu \in [0,1]$ is used to determine whether all actions in this episode are determined by the Q-Network or the optimal trajectory.
• If the training episode is less than or equal to 100, $ϵ$ is used to decide whether the choice of action is randomly selected or selected according to the maximum Q value.
• When the training episode is greater than 100: $\mu >0.9$, the selection of actions in this episode is the same as that in point number five; when $\mu \le 0.9$, the actions are performed in accordance with the optimal trajectory $\mathit{O}$ in this episode. It should be noted that the reward settings in environment 1 and environment 2 are different. They are expressed as follows.

  $$\begin{array}{cc}\hfill Environment1:& R({\mathit{s}}_{\mathit{t}},{\mathit{a}}_{\mathit{t}})=\left\{\begin{array}{cc}\hfill \sum _{i=1}^{\mathrm{U}}{o}_{u_i}{\mathrm{Q}}_i,& \mathrm{if}\sum _{i=1}^{\mathrm{U}}{o}_{u_i}>0,\hfill \\ \hfill -1,& else,\hfill \end{array}\right.\hfill \\ \hfill Environment2:& R({\mathit{s}}_{\mathit{t}},{\mathit{a}}_{\mathit{t}})=\left\{\begin{array}{cc}\hfill \sum _{i=1}^{\mathrm{U}}{o}_{u_i}{\mathrm{Q}}_i,& \mathrm{if}\sum _{i=1}^{\mathrm{U}}{o}_{u_i}>0.\hfill \\ \hfill 0,& else.\hfill \end{array}\right.\hfill \end{array}$$
• When step t ends, store $({\mathit{s}}_{\mathit{t}},{\mathit{a}}_{\mathit{t}},r_t,{\mathit{s}}_{\mathit{t}+\mathbf{1}})$ in the experience replay buffer; update the total reward value of the current cycle $R^{{}^{\prime }}=R^{{}^{\prime }}+r_t$; retrieve multiple records from the experience replay buffer to update Q-Network; increase the value of $ϵ$.
• At the end of each episode, compare the values of R and $R^{{}^{\prime }}$, and compare the number of training steps in this period with the length of $\mathit{O}$ to determine whether to update the action trajectory record.

The advantages of the improved DQN-PTR includes:

• When adding a judgment item to the outer layer of the traditional DQN, the setting of the reward function becomes flexible.
• In addition to the traditional DQN replay memory D, a new cache space $\mathit{O}$ is added, which can be used to record the excellent action trajectory.
• Depending on the type of problem solved, the action space of the DQN-PTR can vary with the execution of each action.

Algorithm 1: DQN-PTR to solve $\mathcal{P}3$

01: Initialize the Q-Network Q with random weights $\theta$ 02: Initialize the Target Q-Network $\widehat{Q}$ with weights ${\theta }^{{}^{\prime }}=\theta$ 03: Initialize replay memory D to capacity N 04: Initialize $ϵ=0,{ϵ}_{increment}=0.0001,{ϵ}_{max}=0.9999$ 05: Initialize optimal trajectory $\mathit{O}$ to empty and maximum total return $R=0$ 06: For $episode=1,\mathcal{M}$ do 07:  Initialize sum reward $R^{{}^{\prime }}=0$ 08:  Initialize state $s_0$, $\mu =rand\in [0,1]$ 09:  For each step t do 10:   If $episode\le 100$ or $\mu >0.9$ then 11:    If $rand\in [0,1]\ge ϵ$ then 12:     Select a random action $a_t$ 13:    else 14:     Set $a_t=arg{max}_{a}Q(s_t,a;\theta )$ 15:    end if 16:    Execute action $a_t$, observe next state $s_{t+1}$ and reward $r_t$ according to environment 1 17:   else 18:    Set $a_t$ according $\mathit{O}\left[t\right]$ 19:    Execute action $a_t$, observe next state $s_{t+1}$ and reward $r_t$ according to environment 2 20:   end if 21:   $R^{{}^{\prime }}=R^{{}^{\prime }}+r_t$ 22:   Store transition $(s_t,a_t,r_t,s_{t+1})$ in D 23:   If episode terminates at step $t+1$ then 24:    If $R^{{}^{\prime }}>R$ then 25:     Replace the trajectory in $\mathit{O}$ with $\{a_1,\dots ,a_t\}$ 26:     $R=R^{{}^{\prime }}$ 27:    else if $R^{{}^{\prime }}=R$ and $t<len\left(\mathit{O}\right)$ then 28:     Replace the trajectory in $\mathit{O}$ with $\{a_1,\dots ,a_t\}$ 29:    end if 30:    break 31:   end if 32:   Sample random mini-batch of transitions $(s_i,a_j,r_j,s_{j+1})$ from D 33:   Set $y_j=\left\{\begin{array}{cc}& r_j,\mathrm{if}\mathrm{episode}\mathrm{terminates}\mathrm{at}\mathrm{step}j+i,\hfill \\ & r_j+\gamma {max}_{a^{{}^{\prime }}}\widehat{Q}(s_{j+1},a^{{}^{\prime }};{\theta }^{{}^{\prime }}),\mathrm{otherwise},\hfill \end{array}\right.$ 34:   Perform a gradient descent step on ${(y_j-Q(s_j,a_j;\theta ))}^2$ with respect to the network parameters $\theta$ 35:   If $ϵ<{ϵ}_{max}$ then 36:    $ϵ=ϵ+{ϵ}_{increment}$ 37:   end if 38:   Every C steps reset $\widehat{Q}=Q$ 39:  end for 40: end for

5. Analysis of Simulation Results

In this section, we evaluate the performance of the computational offloading scheme proposed in this paper and the DQN-PTR algorithm through computer simulations. We first present the simulation parameters of the system. Then, we discuss the experimental results.

5.1. Simulation Setup

In the simulation, we assume the following scenario. We consider partial offloading between multiple devices in a small area covered by a single base station.The edge server on the base station side is equipped with an reinforcement learning agent which can make decisions about the offloading scheme in this area. In addition, the computing resources in the system are limited, and computing tasks cannot be completed locally. According to references [7,25,26,29,34,35], we set the simulation parameters to match our research scenario. We consider that the transmission power, channel bandwidth, and background noise of each device are 2 W, 10 MHz, and −170 dBm. The computing capacity of TDs and RDs are 24 Mcycles/s and 35 Mcycles/s, respectively. The data size and maximum latency of each task are 2.15 Mbits and 1 s, respectively. The number of CPU cycles is set 20 Cycles/bit.

We compare the proposed algorithm with traditional MEC schemes, two benchmark algorithms, and reinforcement learning algorithms:

• Full Local: All TDs execute their tasks via local computing.
• Local-cloud: The computing tasks on TDs can be divided into two parts, which are computed on the local and edge cloud, respectively. In order to make full use of computing resources, all computing resources on TD i are allocated to task ${\varphi }_i$. If the local resources are insufficient, the computing resources will be supplemented by the edge cloud.
• RBA: The TD i randomly selects a D2D RD device for computing offload and utilizes all the computing resources of the D2D RD device. If the computing resources of the two places are insufficient, the edge cloud will supplement the computing resources.
• GBA: The TD i selects the D2D RD device with the largest remaining resources. Under the condition of making full use of local computing resources, the remaining computing tasks are evenly sent to the D2D RD device and edge cloud for computing.
• Q-learning: Q-learning is a basic reinforcement learning algorithm. Using Q-learning to solve $\mathcal{P}3$, in simulation, the state space, action space, and reward function of Q-learning are all the same as those in DQN-PTR.
• DQN: DQN is an improved reinforcement learning algorithm based on Q-learning. In the simulation, the state space, action space, and reward function of DQN are all the same as those in DQN-PTR.

5.2. Simulation Result

In Figure 3, we show the number of supported (or unexecuted) TDs versus the total number of TDs in the system. The computing capacity of the MEC server is 50 Mcycles/s, and the number of D2D RDs is $\frac{1}{2}$ of the number of TDs. Since the scenarios we study mainly concern computationally intensive tasks, none of the tasks are computed locally. Relying on the computing resources of the local and MEC server, when the number of TDs reaches five, the upper limit of the system computing capacity can be reached. When considering D2D RDs in the system, the total number of computing tasks that the system can complete naturally increases as the number of TDs increases. In Figure 3, the DQN-PTR method proposed in this paper can achieve the best results. The DQN algorithm works satisfactorily when the number of devices is relatively small, but the results become worse as the number increases. However, the gap between Q-learning and DQN-PTR is always large. This is because as the number of devices increases, the number of actions and states in reinforcement learning also increases, which results in worse training results under the same training episodes. Additionally, when the number of TDs is greater than 10, the results obtained by the RBA and GBA algorithms are both smaller than the algorithm proposed in this paper.

In Figure 4, we show the number of supported (or unexecuted) TDs as the computing resources of the MEC server increases. The number of TDs and D2D RDs in the system is 30 and 15, respectively. It can be observed from Figure 4 that if the DQN-PTR algorithm is used for computing offload planning, the MEC server only needs to provide 100 Mcycles/s of computing resources to compute all tasks in the system. In addition, if the computing resources of D2D RDs in the system are utilized, even the relatively poor allocation algorithm (DQN, RBA and GBA) can save about 300 Mcycles/s of cloud computing resources compared with traditional local-cloud offloading.

In Figure 5, we present the number of supported (or unexecuted) TDs versus the number of D2D RDs. The computing power of the MEC server is fixed at 50 Mcycles/s and the number of TDs is 20, so the total data volume of all tasks in the system is 43.00 Mbits. As can be seen from Figure 5, our proposed DQN-PTR algorithm requires the fewest D2D RD devices. DQN, GBA, and Q-learning algorithms require 15 D2D RD devices to complete all tasks, while RBA requires 20 D2D RD devices.

Figure 6 shows the learning curves of Q-learning, DQN, and DQN-PTR. The number of TDs and D2D RDs is 20 and 10, respectively. In order to ensure fairness, the three algorithms have the same parameters, i.e., the maximum number of steps allowed per episode is 50. The $\epsilon -greedy$ value increases from 0 to 0.95 with a growth rate of 0.0001. Learning rate $\gamma$, replay memory size, and mini-batch size are 0.0001, 10,000, and 200, respectively. Reference [36] for an analysis of our curves, we found that DQN-PTR performed better than DQN and Q-learning. Although Q-learning has the fastest learning speed, our proposed algorithm is more stable than DQN in terms of the fluctuation of the curve, and obtains the highest average return value.

6. Conclusions

This paper has proposed an integrated framework for multi-user partial offloading and resource allocation, combining MEC and D2D technologies. Under this framework, we can make the decision of computational offloading and the resource allocation of MEC and idle devices. We have also designed a convex optimization method to simplify the problem. Finally, we have derived a DQN-based reinforcement learning algorithm to solve the problem. Simulation results have shown that the proposed scheme has better performance than other benchmark schemes under different system parameters.