Edge Computing Offloading Method Based on Deep Reinforcement Learning for Gas Pipeline Leak Detection

Abstract

Traditional gas pipeline leak detection methods require task offload decisions in the cloud, which has low real time performance. The emergence of edge computing provides a solution by enabling offload decisions directly at the edge server, improving real-time performance; however, energy is the new bottleneck. Therefore, focusing on the gas transmission pipeline leakage detection scenario in real time, a novel detection algorithm that combines the benefits of both the heuristic algorithm and the advantage actor critic (AAC) algorithm is proposed in this paper. It aims at optimization with the goal of real-time guarantee of pipeline mapping analysis tasks and maximizing the survival time of portable gas leak detectors. Since the computing power of portable detection devices is limited, as they are powered by batteries, the main problem to be solved in this study is how to take into account the node energy overhead while guaranteeing the system performance requirements. By introducing the idea of edge computing and taking the mapping relationship between resource occupation and energy consumption as the starting point, the optimization model is established, with the goal to optimize the total system cost (TSC). This is composed of the node’s transmission energy consumption, local computing energy consumption, and residual electricity weight. In order to minimize TSC, the algorithm uses the AAC network to make task scheduling decisions and judge whether tasks need to be offloaded, and uses heuristic strategies and the Cauchy–Buniakowsky–Schwarz inequality to determine the allocation of communication resources. The experiments show that the proposed algorithm in this paper can meet the real-time requirements of the detector, and achieve lower energy consumption. The proposed algorithm saves approximately 56% of the system energy compared to the Deep Q Network (DQN) algorithm. Compared with the artificial gorilla troops Optimizer (GTO), the black widow optimization algorithm (BWOA), the exploration-enhanced grey wolf optimizer (EEGWO), the African vultures optimization algorithm (AVOA), and the driving training-based optimization (DTBO), it saves 21%, 38%, 30%, 31%, and 44% of energy consumption, respectively. Compared to the fully local computing and fully offloading algorithms, it saves 50% and 30%, respectively. Meanwhile, the task completion rate of this algorithm reaches 96.3%, which is the best real-time performance among these algorithms.

1. Introduction

With the advent of 5G, high-performance computing and other technologies in industry have developed in the direction of high real-time engagement and low energy consumption, and many delay-sensitive and computationally intensive applications and services have emerged. Although cloud computing can provide sufficient computing resources, a large amount of traffic generated in the process of task delivery to the cloud will likely lead to network congestion, unpredictably high delay, and massive transmission energy consumption, and a distributed computing method is needed to solve these problems. Edge computing makes this feasible. To move computing to the edge of the network solves the problem of high latency of cloud services and makes up for the lack of computing resources of end devices to a certain extent.

Although edge computing provides a feasible solution for such scenarios, it entails the problem of using limited resources to realize high real-time performance and low energy consumption. Much research has been performed in this field, with good results. The main concern is to balance low latency and low energy consumption, which can effectively solve the offloading problem when the attributes of the task set to be processed are known. However, such solutions have the common limitation of low robustness, which will lead to a chain reaction when unexpected tasks enter the system, sharply degrading system performance. This is more likely to occur when tasks arrive in real time. Sun et al. [1] proposed a task offloading algorithm based on a hierarchical heuristic strategy, aiming to minimize the task delay and energy consumption, but it assumes the task set to be scheduled is known, without taking into account sudden tasks. Similarly, Li et al. proposed a task offloading algorithm based on deep reinforcement learning, which is based on a known task set, to schedule tasks [2].

Taking the leak detection of a natural gas transmission pipeline as an example, once a leak occurs, there is a great danger. Detectors need to work in the leak area. The faster they locate the leak point, the less the security risk; hence this scenario demands high reliability in real time. Many portable gas leak detectors depend on the collection of infrared or other spectral images for image analysis [3]. Because the detector must constantly change its position during operation, it needs to feedback results immediately so that it will not miss the leak point. However, due to the size of the detector, its computing and battery capacity have certain limitations. It is difficult to complete some complex recognition tasks on time, which greatly affects detection efficiency and accuracy. Figure 1 shows the workflow of the solution. By introducing edge computing, complex image processing tasks generated by the detection equipment can be uploaded to the cloud and processed quickly, which can enable the accurate and quick location of the leak point.

This paper proposes a natural gas leak detection algorithm that combines edge computing task offloading with portable natural gas leak detection technology—a real-time multi-leak detection algorithm based on the improved advantage actor-critic (AAC) method—to improve the detection efficiency and endurance of instruments. We consider a three-tier edge computing architecture with cloud-side and end-to-end collaboration, where the portable gas leak detector is at the end of the system and has some computing power itself. To improve the efficiency and range of the detector, the image analysis task must be offloaded, so as to determine where to process the task and allocate resources to it. The current system state is determined, and this is input to the constructed AAC network to determine the processing position of tasks in the system. The results obtained from the network are optimized using the proposed heuristic algorithm. At the same time, the allocation strategy of communication resources is determined, and tasks are scheduled and executed according to the offloading results. By analyzing the problem, improving the detection efficiency and range of the existing detection instruments can address the objectives in the edge computing task offloading problem, that is, to improve the real-time performance and minimize the energy consumption of the edge computing system as much as possible. This paper makes the following contributions:

• The system real-time requirements and energy consumption limits are modeled in a unified manner. Tasks arrive and are unloaded in real time in order to be close to the real situation to the maximum extent, which enables the proposed algorithm in this study to achieve good practical relevance.
• A real-time multi-leak detection algorithm based on the improved AAC is proposed to solve the problem in that traditional reinforcement learning methods are difficult to converge and traditional heuristics cannot fully consider various influencing factors. The proposed algorithm in this paper allows the AAC algorithm to complement the traditional heuristics, and the AAC algorithm can fully take into account the impact of various environmental factors on the unloading results while also taking into account the long-term payoff of the system. However, the AAC algorithm is a reinforcement learning algorithm whose convergence effect is not stable enough, so this paper supplements the AAC algorithm with a heuristic algorithm to correct the obtained results in order to ensure that the proposed algorithm can at least achieve the performance of the heuristic algorithm. Moreover, through detailed mathematical analysis, the condition that the proposed heuristic algorithm obtains the minimum value of total system cost (TSC) is proved to hold.
• We compare the performance of the proposed algorithm with that of the deep Q network (DQN), the artificial gorilla troops optimizer (GTO), the black widow optimization algorithm (BWOA), the exploration-enhanced grey wolf optimizer (EEGWO), the African vultures optimization algorithm (AVOA), and the driving training-based optimization (DTBO), and two other baseline algorithms. Experiments show that the proposed algorithm reduces the energy consumption by 56% compared to DQN. Compared with GTO, BWOA, EEGWO, AVOA and DTBO algorithms, the energy consumption is reduced by 21%, 38%, 30%, 31% and 44%, respectively. The energy consumption is reduced by 50% compared to the fully local computing algorithm, and by 30% compared to the fully offloading algorithm. Meanwhile, the task completion rate of this algorithm reaches 96.3%, which is the best real-time performance among these algorithms. In addition, the proposed algorithm in this paper has a faster convergence speed than the DQN algorithm.

The remainder of this paper is organized as follows. Section 2 describes related work. Section 3 presents the proposed system model and describes the problem. Section 4 details the main steps of the proposed algorithm. Section 5 compares the performance of the proposed algorithm with baseline algorithms such as DQN and GTO through experiments. Section 6 concludes the paper.

2. Related Work

Many studies have been conducted on the task offloading problem of edge computing, which is NP-hard, and all solutions thus far have been approximate. However, different optimization techniques can be used such that the approximate solution converges to the optimal solution. These solutions start either with machine learning or traditional means such as greedy heuristics, integer optimization, branch delimitation, game theory, or convex optimization. The two most important factors in edge computing are latency and energy consumption.

2.1. Traditional Task Offloading Methods

Kan et al. proposed a heuristic algorithm for offloading tasks to MEC servers considering radio and computational resources with the goal of minimizing the average task latency, which was shown by experiments to achieve excellent results under different latency requirements [4]. Due to the relative lack of infrastructure, they introduced drones to assist in edge computing, and proposed a USS algorithm [5] that can satisfy the task processing latency constraint in the multiuser case. Wang, Shen, and Zhao introduced a dynamic penalty function in a study of edge computing in the smart grid domain, and proposed an improved algorithm for solving Lagrange multipliers [6], which overcomes the shortcomings of traditional grid systems that cannot provide deterministic services, and can effectively improve the overall system revenue and reduce the average delay of user tasks. Li et al. considered event-triggered decision systems, whose goal is to optimize the average system revenue to satisfy the average delay constraint for different priority services [7]. Ref. [8] presented designs of online computing task scheduling methods for multi-server edge computing scenarios [8]. Sun et al. [9] considered an ultra-dense network environment that supports edge computing. Constantly moving users dynamically generate computational tasks in the network, which need to be offloaded to the base station for computation. In order to minimize the average delay given a limited energy budget, users need to make mobility management decisions about base station association and switching based on their service requirements without knowing future information.

System energy consumption has long been a concern among edge computing researchers, as an important component of system cost, and especially in mobile edge computing, where energy consumption directly affects system endurance and reliability. Michael proposed a hybrid method based on particle swarm optimization and the gray wolf optimizer [10] to optimize the energy consumption of MEC task offloading. Ding and Zhang [11] proposed a game theory-based computational offloading strategy for massive IoT devices, which improves data transfer and reduces task energy consumption using the beneficial task offloading theory.

Delay and energy consumption factors are usually considered together, and are important factors affecting the user experience. Researchers can decide whether to optimize delay or energy consumption based on specific requirements. Some studies have considered the minimum energy consumption while satisfying the latency constraint using heuristic algorithms [12,13]. Others have proposed a more flexible optimization objective, synthesizing both into a cost objective, where the weights of delay and energy consumption in the cost formulation can be changed according to the case [14,15]. Ref. [16] considered two different cases of adjustable and non-adjustable CPU frequency of APs. A linear relaxation based approach and an exhaustive search based approach are proposed to obtain the offloading decision for these two cases, respectively. The method aims to minimize the total task ground execution delay and the energy consumption of the mobile device (MD) [16]. In order to trade-off the two metrics of energy consumption and computational latency, a Liapunov-based algorithm was proposed in Ref. [17] for computing task offloading decisions in mobile edge computing systems. The algorithm greatly reduces the energy consumption of the device while satisfying the latency constraint [17]. Ref. [18] investigated the computational offloading and scheduling problem, which seeks to minimize the cost per mobile device, where the cost is defined as a linear combination of task completion time and energy consumption. In addition, the literature considers inter-device communication and competition for computational resources. The problem is also defined formally using a game model, and a decentralized algorithm is designed to achieve a pure policy Nash equilibrium [18]. Tang et al. modeled the multi-user computational offloading problem in an uncertain wireless environment as a non-cooperative game based on PT, and then proposed a distributed computational offloading algorithm to obtain a Nash equilibrium, which minimizes the user overhead [19]. Yi et al. considered that tasks are randomly generated by mobile users and proposed a mechanism based on queuing model. This is used to maximize social welfare and achieve the equilibrium of the noncooperative game among mobile users [20].

Task offloading algorithms based on the above studies are based on ideal mathematical models, and cannot consider all the factors that affect the optimization objective, which limits their task offloading performance. To solve this problem, a new class of offloading methods has been proposed, using deep learning techniques, with good results.

2.2. Machine Learning Task Offloading Methods

To cope with the variability of edge computing application environments, Wang and Jia et al. proposed a meta-reinforcement learning-based approach to solve the computational offloading problem [21], which enables fast adaptation to dynamic scenarios without updating too many parameters. A joint task offloading and bandwidth allocation problem was considered for multiuser computational offloading, with the goal of minimizing the overall delay in completing user tasks, using a DQN approach to find the optimal solution [22].

Wang Jin et al. [23] found that studies using DRL for task offloading rarely focus on the dependencies between tasks, and proposed a DRL offloading method that can address dependent tasks. The general dependency of tasks was modeled as a directed acyclic graph (DAG), and an S2S neural network captured the features of the DAG and output the offloading strategy. The method can use delay, energy consumption, or tradeoffs of both as optimization objectives.

In Ref. [24], the authors are the first to attempt to consider end-device energy consumption in a deep learning-based modeling of MEC partial offloading schemes [24]. They propose a novel partial offloading scheme EEDOS based on a fine-grained partial offloading framework, in which the cost function comprehensively considers important parameters such as residual energy of end-devices and energy consumption of previous application components. Dai and Niu [25] used unmanned aerial vehicles (UAVs) to assist edge servers for task offloading, minimizing the energy consumption of all mobile end devices by jointly optimizing UAV trajectories, task association, and the resource allocation of computation and transmission. They reduced the problem complexity by decomposing the joint optimization problem into the subproblems of UAV trajectory planning, task association scheduling, and resource allocation of computation and transmission. A proposed hybrid heuristic and learning-based scheduling strategy (H2LS) algorithm incorporated long short-term memory neural networks, fuzzy c-means, deep deterministic policy gradients, and convex optimization techniques.

As with traditional optimization techniques, most of the research on the application of deep learning in offloading edge computing tasks focuses on the integrated consideration of delay and energy consumption. To focus on only one of these aspects can bring the results closer to the optimal solution, at the price of a narrow range of practical applications. Yang and Lee proposed a deep supervised learning-based dynamic computing task offloading approach (DSLO) for mobile edge computing networks [26], minimizing the delay and energy consumption by jointly optimizing the offloading decision and bandwidth allocation problem. Cao et al. proposed a multi-intelligent deep reinforcement learning (MADRL) scheme [27] to solve the multichannel access and task offloading problems in edge computing-enabled Industry 4.0, which allows edge devices to collaborate and significantly reduce computational latency and mobile device energy consumption relative to traditional methods. Huang et al. [28] considered a mobile edge computing system, in which each user has multiple tasks transferred to the edge server over a wireless network. They proposed a deep reinforcement learning based approach to solve the problem of joint task offloading and resource allocation. In Refs. [29,30], the authors proposed to use deep reinforcement learning methods to solve the task offloading problem in mobile edge computing, and made some progress, obtaining better latency and energy consumption than when using deep learning [29,30].

Although the above deep learning-based solutions have achieved good results, they have limitations if we only consider how to optimize the latency and energy consumption of task processing. In the problem addressed in this paper, each image analysis task is generated in real time, and the optimization goal of low latency can cause some tasks to have low processing latency, at the cost of some subsequent tasks that exceed their deadlines; hence, they cannot guarantee overall high real-time performance. We propose an AAC and heuristic policy-based task offloading algorithm that simultaneously considers overall task execution in real time and low energy consumption, and use it to optimize the performance of a portable gas leak detector. The algorithm reduces the energy consumption of the detector as much as possible by jointly optimizing the task offloading location and resource allocation problems while ensuring completion within the deadline.

3. System Model and Problem Description

3.1. System Model

The edge computing system (ECS) consists of a cluster of cloud servers, a wireless communication base station with small edge servers, and K portable gas leak detectors, $\gamma$ = {$U_1$, $U_2$, $U_3$, ……, $U_k$}; each detector $U_i$ can generate in time order a series of independent image recognition tasks, each task of all detectors is generated in real time and cannot be split, and the set of tasks can be denoted by ${\mathsf{Г}}_i$ = {$T_{i,1}$, $T_{i,2}$, $T_{i,3}$, ……$T_{i,N}$}; each task has six attributes, and any task $i$ can be denoted as $T_i$ = {$j$, $s_i$, $d_i$, $D_i$, $c{y}_i$, ${\omega }_i$}, where $j$ is the serial number of the detector, $s_i$ is the release time of task $i$ (in seconds), $d_i$ is the relative deadline of the task, $D_i$ is the size of the data carried by the task (in Mb), $c{y}_i$ is the CPU processing cycle required by the task, and ${\omega }_i$ is its priority. An example of the system model is shown in Figure 2. The cloud server has sufficient resources for the detectors, so there is no need to consider the waiting and preemption of tasks in the cloud, and only one task can be processed at a time on a detector. The task offloading algorithm is deployed on the edge server in the communication base station, and the information changes of each node are transmitted to the edge server in real time. In this model, the tasks to be offloaded are generated by the detector in real time, and each task is indivisible. Considering that the offloading decision requires knowledge of the global information of the system, while the system that transmits the main task parameters is not a complete model and the base station is very close to the detector and there is no conflict in the transmission process, the offloading algorithm generates comparable and almost negligible communication energy consumption and delay whether it is executed on the detector or on the base station equipped with the edge computing server [1]. The edge server has more arithmetic power and faster execution, so the communication base station is left in charge of the communication function and makes the offloading decision, based on which the detector offloads the computational task to the cloud or processes it locally. If offloaded to the cloud, the cloud server will return the results after processing, and the energy consumption of the detector during the offloading process includes that for transmission and local processing.

3.2. Description of problem

Since each task in the system can be chosen to be executed either locally or in the cloud, an offloading decision variable is introduced to indicate the execution location of a task,

$${\pi }_i^C=\{\begin{array}{c}0 Task executed locally,\\ 1 Task executed in the cloud\end{array}$$

(1)

The transmission power of all edge devices (detectors) is $P$. The data transmission rate assigned for any task $i$ is $r_i$, the average CPU frequency of the cloud server is $F^C$, and the CPU frequency of the edge device is $f_i^L$. Therefore, the time to locally execute task $T_i$ of edge device U is

$$t_i^L=\frac{c{y}_i}{f_i^L}$$

(2)

The local execution energy consumption of a task is

$$e_i^L=\mathsf{a}\ast {\left(f_i^L\right)}^2\ast c{y}_i$$

(3)

If a task is unloaded, its unloaded transfer time is

$$t_i^{LC}=\frac{D_i}{P}$$

(4)

The cloud processing time of a task is

$$t_i^C=\frac{c{y}_i}{F^c}$$

(5)

The offloading transmission energy consumption of a task is

$$e_i^T=\frac{P_i\ast D_i}{r_i}$$

(6)

where $\mathsf{a}$ is the chip-related energy consumption coefficient of edge device U [31].

The mathematical model described in this paper must optimize the objectives of the real-time system and the total energy consumption of edge devices for synergistic optimization while considering load balancing. To achieve the joint optimization of the above objectives, the model optimization objective is transformed to the total system cost TSC,

$$TSC=\underset{r_i,{\pi }_i^L}{\min }{{\displaystyle \sum }}_1^M\left(1-{\pi }_i^C\right)\ast \mathsf{a}\ast {\left(f_i^L\right)}^2\ast c{y}_i+{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast \frac{P_i\ast D_i}{r_i}$$

(7a)

$${\pi }_i^C=\left\{0,1\right\}$$

(7b)

$$0\le f_i^L\le F_i^L$$

(7c)

$$0\le E_i\le 1$$

(7d)

$$0\le {{\displaystyle \sum }}_{i=1}^M{r}_i\le R$$

(7e)

where $E_i$ is the remaining power percentage of edge device $i$, and $\overline{E}$ is the average power percentage of all devices that are idle and must perform offload tasks.

Equation (7a) is the weighted sum of the local execution energy consumption and offload transmission energy consumption for task $i$, and $\frac{\overline{E}}{E_i}$ is the distance between the remaining power of each device and the average power. A larger $\frac{\overline{E}}{E_i}$ indicates that the remaining power of the device is farther from the average power. To reduce the energy consumption of the device, it has the opportunity to share more communication resources (faster data transmission rate) when the system performs bandwidth resource allocation [1]

Constraints 7b, 7c, 7d, and 7e refer to the offloading decision variables, range of CPU frequency variation per device, range of power percentage variation per edge device, and range of data transfer rate variation per device, respectively.

The variables involved in the model are shown in

Table 1. Model variables.

Symbol	Definition
$s_i$	Task $i$ release time
$d_i$	Task $i$ relative deadline
$D_i$	Amount of data carried by task $i$
${\omega }_i$	Task $i$ priority
${\pi }_i^C$	Offload decision variables
$t_i^L$	Task $i$ local execution time
$t_i^{LC}$	Task $i$ offload transfer time
$t_i^C$	Task $i$ cloud execution time
$e_i^L$	Task $i$ local execution energy consumption
$e_i^T$	Task $i$ offload transfer energy consumption
$r_i$	Task $i$ assigned data transfer rate
$E_i$	Percentage of power remaining in detector
$\overline{E}$	Average power percentage of detectors idle and performing offload tasks
$TSC$	Total system cost

.

4. Task Offloading Algorithm

The proposed task offloading algorithm has two parts. The AAC algorithm gives the scheduling location of the task. The initial offloading decision is obtained by the heuristic algorithm, based on which the AAC network is updated. The heuristic algorithm can be solved quickly for the NP-hard problem, but the suboptimal solution found by this method has room for improvement. Reinforcement learning is used to optimize the obtained unloading strategy. The algorithms are described below.

4.1. Heuristic Algorithm

The heuristic algorithm considered in this paper takes Equation (7a) as the optimization objective. Since the optimization for the TSC is an NP-hard problem, the deep reinforcement learning algorithm is used to first determine whether the new arrival task is to be offloaded, and the Cauchy–Buniakowsky–Schwarz inequality is used to derive the transmission rate allocation, and thus the processing time for each task. If the processing time exceeds the task deadline, the processing position of the task is redetermined according to the priority of the task, and the transfer rate allocation is calculated. Iterations continue until an approximate optimal solution is found.

The Cauchy–Buniakowsky–Schwarz inequality is often applied to quickly solve n-dimensional inequalities [32] and its application to solve the system communication resource allocation can simplify computations and reduce the execution time of the offloading algorithm. When using this inequality, we must first ensure that the left-hand side of the inequality can be split into two non-negative expressions multiplied together.

Theorem 1.

If the inequality $R\ast {\displaystyle \sum }_1^M\frac{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}{r_i}\ge {\left({\displaystyle \sum }_i^M\sqrt{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}\right)}^2$satisfies both$R>0$and${\displaystyle \sum }_1^M\frac{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}{r_i}\ge 0$, the equality sign holds when and only when

$$r_i^{\ast }=\frac{R\ast \sqrt{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}}{{{\displaystyle \sum }}_{i=1}^M\sqrt{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}}$$

(8)

Additionally, when $r_i$=$r_i^{\ast }$, TSC obtains the minimum value.

Proof of Theorem 1.

It is known that $R$ is the total transmission rate of the system, which is always positive, and each term in ${\displaystyle \sum }_1^M\frac{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}{r_i}$ is greater than or equal to zero, which satisfies the condition of use of the Cauchy–Buniakowsky–Schwarz inequality. Then the following inequalities are solved by combining the constraints, and the specific solved process is stated in Equation (9) for the optimization objective expression (7a) and its constraint (7e):

$$R\ast {{\displaystyle \sum }}_1^M\frac{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}{r_i}\ge {{\displaystyle \sum }}_i^M{r}_i\ast {{\displaystyle \sum }}_1^M\frac{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}{r_i}\ge {\left({{\displaystyle \sum }}_i^M\sqrt{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}\right)}^2.$$

(9)

According to the Cauchy–Buniakowsky–Schwarz inequality, if there exists some $r_i$ not equal to 0, the equality sign holds when and only when there exists a real number $X$ such that for every $i=1,2,\mathrm{\dots },n$, there is $r_i\ast X+\frac{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}{r_i}=0$, i.e.,

$$r_i^{\ast }=\frac{R\ast \sqrt{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}}{{{\displaystyle \sum }}_{i=1}^M\sqrt{{\pi }_i^C\ast \frac{\overline{E}}{E_i}\ast P_i\ast D_i}}$$

(10)

and when $r_i$ = $r_i^{\ast }$, TSC obtains the minimum value, and the theorem is proved. □

In the task scheduling process of the real-time edge system in this paper, not only should we consider making the energy consumption of the edge devices as low as possible, but the tasks should meet the deadline requirements to the maximum extent to improve the real-time performance of the whole system. In traditional scheduling methods, often only the remaining execution time or deadline of a task is used to reflect that the urgency of task execution evaluation criteria is too singular. We propose a dynamic priority evaluation method that integrates the initial priority, remaining execution time, deadline, and idle time of a task. The dynamic task priority ${\Omega }_i$ is composed of the preemption cost ${\delta }_i$ of the task and the execution urgency ${\phi }_i$,

$${\Omega }_i={\delta }_i\ast {\phi }_i$$

(11)

where

$${\delta }_i=\frac{{\omega }_i}{t_i^{LC}+t_i^C}$$

(12)

Tasks have different levels of importance. Equation (12) integrates the initial priority of tasks and deadlines, which can ensure that important tasks can be completed on the basis of as many tasks as possible that are close to the deadline and can also be executed, which protects the tasks being executed to some extent. The task execution urgency is

$${\phi }_i=q^{\frac{t_i^{LC}+t_i^C}{d_i-t}}$$

(13)

where $t$ is the current moment and $q\in \left(1,\infty \right)$. The execution urgency of the task decreases as the task is executed, which in turn gives a somewhat greater chance of execution for newly arrived tasks.

4.2. Deep Reinforcement Learning Algorithms

To perform further optimization based on the task unloading decision obtained from the heuristic strategy, a deep reinforcement learning model, AAC, is considered to perform the unloading decision for the newly arrived task. The network structure of the model is shown in Figure 3.

From Figure 3 above, we can see that the AAC network is composed of two sub-networks, actor and critic, where the first two layers of the two sub-networks are shared, in order to reduce the complexity of the model and speed up the network convergence. Meanwhile, the hidden layers of both sub-networks consist of 256 × 128 neurons, which is the best combination chosen after several attempts in the experiments. Through keeping the other conditions of the experiment constant, only the number of neurons in the network was allowed to increase evenly between 64 × 64 and 256 × 256. We found that too few neurons make the training unstable and difficult to converge, while too many neurons lead to overfitting. Optimal model performance is only achieved when the number of neurons is varied to near 256 × 128.

The offloading decision is a prerequisite for resource allocation. We discuss the three elements of the reinforcement learning-based offloading decision method: environment, action, and reward.

• Environment state $S$

The superiority of the state will have a great impact on the final training effect of the model. In this model, the environment state includes the state of the task and the external environment. The model is updated only when a new task arrives, and these arrive in chronological order, so the task’s own state includes the properties of the new task, and the external environment state has the remaining power $E_i$ of each node in the system at this time, the average remaining power $\overline{E}$, the CPU speed $f_i^L$ of the node generating the task, the average CPU speed $F^C$ of the cloud, and the number of tasks to be transmitted in the system.

2

Action $a$

In the reinforcement learning model, the action is the decision made by the agent, and there are only two actions in this scheduling model: transmission and non-transmission.

3

Reward function

The output of the reinforcement learning model is the probability $p_{\theta }(a|S)$ of selecting different actions in a certain state. To measure the goodness of an action, the system cost TSC is used as the reward.

The AAC algorithm first defines an initial actor $\pi$ to interact with the environment, as shown in Figure 4. The collected information is used to train the critic network to estimate the value function $V$, which is the sum of the rewards received by the system after performing an action until the end of the interaction. The actor network is updated and iterated until both networks converge. The actor network parameters are updated as follows:

$$\widetilde{\nabla R_{\theta }}\approx \frac{1}{N}{{\displaystyle \sum }}_{n=1}^N{{\displaystyle \sum }}_{t=1}^{T_n}\left(LS{C}_t^n+V^{\pi }\left(S_{t+1}^n\right)-V^{\pi }\left(S_t^n\right)\right)\nabla log{p}_{\theta }(a_t^n|S_t^n)$$

(14)

$$\theta =\theta -\eta \ast \widetilde{\nabla R_{\theta }}$$

(15)

where $\widetilde{\nabla R_{\theta }}$ is the gradient of the mean of the reward sum of multiple trajectories, and $\theta$ is the parameter of the actor network. Since the optimization goal is to reduce energy consumption while satisfying the real-time performance of the task, which is the opposite of the goal of maximizing the reward of reinforcement learning, gradient descent is used to update the network.

4.3. Algorithm Process

We combine the heuristic algorithm and deep reinforcement learning algorithm, using the AAC network for offloading decisions, and a heuristic algorithm for resource allocation, as shown in Algorithm 1.

Algorithm 1: Offloading algorithm for edge computing tasks based on deep reinforcement learning.

Input: Number of training rounds $M$, set of tasks $G_{M\ast L}$ used for training, number of edge devices $N$, CPU frequency $f_i^L$, energy consumption factor $\alpha$, transmission power $P_i$, CPU frequency $F^C$ of cloud server, total data transmission rate $R$ available to system, learning rate of actor and critic networks

Output: Trained Advantage Actor-critic model, total energy consumption of system, and task completion rate

1. Initialize model and related parameters;

2. Let actor $\pi$ interact with environment

3. for i = 1 to M do

4.      Storing j-th task in i-th subtask set in training set in a list;

5.      Updating status of environment and storing it in the list;

6.    Input environmental state parameters to actor network, and record action $a$ selected by network with probability $p$;

7.     Determining value of offloading decision variable ${\pi }_i^C$ based on recorded actions;

8.      while flag = True 9.          Passing value of ${\pi }_i^C$ into Equation (7a) and using Equations (8) and (9) to derive allocation of data transfer rate $r$;

10.           Substitute $r$ back into Equation (7a) to find value of TSC at this point, and use Equations (4) and (5) to calculate expected transmission time $t_i^{LC}$ and execution time $t_i^C$ for task;

11.          if $d_i$ < $t_i^{LC}$+$t_i^C$

12.             Calculating priority of all tasks being transmitted and pending transmission and forcing selection of lowest priority task to be executed locally;

13.          end if

14.          if All tasks that are subject to offload meet deadlines

15.              flag = False

16.          end if

17.     Storing track data;

18.   end for

19.   Training critic network using stored trajectory data and storing output $V$ of critic network each time;

20.   Training actor network once more with data stored in steps 6, 18, and 20;

21. end for

22. return Trained Advantage Actor-critic model, total energy consumption of system, along with task completion rate;

The core part of Algorithm 1 uses a heuristic algorithm and the AAC network, which is a deep reinforcement learning network model. In the edge computing scenario considered in this paper, task offloading, and resource allocation is an NP-hard problem. At the same time, the uncertainty of task arrival poses a great challenge for task offloading. Facing this multi-objective optimization problem, traditional optimization techniques (e.g., linear programming) have difficulty in obtaining better results [33]. In addition, deep reinforcement learning has two advantages in facing the above problem: (1) compared with many one-time optimization methods, deep reinforcement learning can adjust the strategy with the change of environment; (2) its learning process does not need to know the relevant a priori knowledge about the law of network state change over time [34,35]. In fact, the heuristic algorithm is the basis on which the present model can operate efficiently, and the main purpose of the AAC is to further optimize the optimization results derived from the heuristic algorithm. The heuristic algorithm performs the optimization search by introducing the Cauchy–Buniakowsky–Schwarz inequality, which can reduce the number of iterations and greatly accelerate the solution efficiency by using the conclusion of Theorem 1.

The AAC model is improved from the actor-critic model. In the actor-critic model, both Q-network (Network to evaluate good and bad actions) and V-network (Network to evaluate good and bad status) need to be estimated, which is not only time-consuming, but also has greater uncertainty. In the AAC model, the expectation of the V-network is directly used to estimate the Q-network, that is, the critic network is allowed to learn the Advantage value directly instead of the Q value. In this way, the assessment of behavior is not only based on how good the behavior is, but also on how much the behavior can be improved. The benefit of the advantage function is that it reduces the variation in the values of the policy network, and stabilizes the model, giving the AAC model superior convergence.

5. Experimental Results and Analysis

Simulation experiments are used to demonstrate the performance of the proposed algorithm. All parameters are chosen according to real scenarios. As shown in

Table 2. Simulation parameters.

Parameter	Value
Number of items of testing equipment $K$	10
Local computing capability $f_i^L$	0.2 GCycles/s
Cloud server computing capability $F^C$	10 GCycles/s
Transmitted power $P_i$	(0.1,0.2) w
Total system transmission rate $R$	800 Mb/s
Computation cycles required per task $c{y}_i$	(0.01,0.3) GCycles

, the number of portable detection devices is set to 10, their computational power is 0.2 GCycles/s, and that of the cloud is 10 GCycles/s. The transmission power (w) of the portable devices is a random number in (0.1,0.2), with a total system transmission rate of 800 Mb/s. The amount of data for each task is (10,40) Mb, and the required computation period is (0.01,0.3) GCycles. The arrival time of the task conforms to a uniform distribution [36].

To demonstrate the performance of the improved AAC-based multi-leakage real-time detection algorithm, the algorithm and the DQN algorithm are trained simultaneously in the same environment. The proposed algorithm is also compared with two benchmark algorithms, that is, task fully local computation and full offloading. Meanwhile, in order to better represent the performance of the proposed algorithm in this paper, we also compare it with a series of excellent heuristics, such as GTO [37], BWOA [38], EEGWO [39], AVOA [40] and DTBO [41].

The variations in the total cost TSC per iteration based on the improved AAC multi-leakage real-time detection algorithm and the DQN algorithm in this experimental setting are shown in Figure 5. From Figure 5, it can be seen that the proposed algorithm in this paper has nearly stabilized and the model reached convergence at 50 rounds of training, while the DQN algorithm only shows a significant trough when the training reaches 700 rounds. Although both use a 256 × 128 network structure, the AAC algorithm allows for more stable training and faster convergence due to the presence of the critic network. The figure also shows that the total cost per round of the proposed algorithm is lower than that of the DQN algorithm, so it is better in terms of overall performance. In Figure 6, the vertical coordinate indicates the total system energy consumption. The energy consumption variation curve of the improved AAC-based multi-leakage real-time detection algorithm is approximately 56% lower than that of the DQN algorithm after convergence. The AAC algorithm is improved on the basis of the DQN algorithm, which overcomes the problem of unstable training of the DQN algorithm. Moreover, the AAC algorithm in this paper is not used alone, it works as a further enhancement after the heuristic algorithm gets the suboptimal solution of the model. Therefore, this algorithm can obtain a large improvement relative to the DQN algorithm.

Figure 7 and Figure 8 compare the line graphs of the energy consumption of the improved AAC-based multi-leakage real-time detection algorithm to the fully local computation algorithm and the fully offloading algorithm. Since these two algorithms are not machine learning algorithms, there is no training process, so it is not necessary to compare the algorithm convergence here. They show that the system using the improved AAC-based multi-leakage real-time detection algorithm has better total energy consumption than the two baseline algorithms, thus saving approximately 50% of the energy consumption compared to the fully local calculation, and saving approximately 30% of the energy consumption compared to the fully unloaded algorithm. To make the experiments more realistic, the test tasks have different amounts of data and complexity; thus, having them all executed locally or in the cloud would result in higher energy consumption due to the underutilization of system resources. At the same time, if we combine Figure 5, Figure 6 and Figure 7 together for comparison, we can see that the system energy consumption of the DQN algorithm is around 9, which would be slightly higher than the 6.5 energy consumption for local computation and 5.3 energy consumption for full offloading. This is due to the fact that in the scenario considered in this paper, the task to be offloaded is so random that the performance of the DQN algorithm is no longer sufficient for this scenario, and incorrect predictions can waste a lot of energy.

Figure 9 shows the comparison of the total system energy consumption between the proposed algorithm and some current excellent heuristics. With a simple calculation, we can conclude that the proposed algorithm saves 21%, 38%, 30%, 31% and 44% of energy consumption compared to the GTO, BWOA, EEGWO, AVOA and DTBO algorithms, respectively. If combined with Figure 6, Figure 7 and Figure 8, it can be seen that all these heuristics used for comparison in the experiments perform well. Nevertheless, the proposed algorithm outperforms them. Thus, we can say with more certainty that due to the addition of deep reinforcement learning, the performance of the traditional heuristic algorithm can be brought to a higher level.

In this experiment we also obtained another metric to evaluate the performance of the algorithms, namely the task completion rate. Based on the output of the code, we can obtain the task completion rate of 96.4% for the proposed algorithm and 93.2% for the DQN algorithm; the corresponding values were 92.8%, 90.3%, 89.4%, 94.3% and 91.1% for the GTO, BWOA, EEGWO, AVOA and DTBO algorithms, respectively. The task completion rates for the fully local computing and fully offloading algorithms were 86.7% and 93.4%, respectively. According to the experimental results, the use of the proposed algorithm in this paper allows the highest execution success rate of the tasks, indicating that this algorithm has the best real-time performance and can ensure that as many tasks as possible are completed before the deadline. The task completion rate of the fully local computing algorithm is the lowest, which is mainly due to the high complexity of the task and the limited computing power of the nodes.

From the experiments designed in this paper, we can know that this algorithm design idea is reasonable and effective. It is based on the principle of using heuristic algorithm for initial optimization at first, and then further optimization using deep reinforcement learning. It can bring about more efficient task offloading for edge computing, which not only ensures the real-time performance of the algorithm, but also further reduces the system energy consumption compared to the current better optimization-seeking algorithms such as GTO.

6. Conclusions and Future Work

We studied an edge computing task offloading and resource allocation problem in a natural gas pipeline leak detection scenario, with the optimization goal of minimizing energy consumption while ensuring high real-time performance of the system. Due to the unpredictability of computational tasks, deep reinforcement learning was used to solve this problem. Using the AAC algorithm framework, the final offloading strategy was obtained by fully considering minimizing the overall system cost and continuously optimizing the task offloading strategy, followed by optimizing the allocation of communication resources through a heuristic algorithm based on the Cauchy–Buniakowsky–Schwarz inequality. Simulation results show that this algorithm has a faster convergence speed compared to the DQN algorithm, while the energy consumption is reduced by 56%. Although heuristics such as GTO, BWOA, EEGWO, AVOA and DTBO have better performance than the DQN algorithm, the proposed algorithm still saves 21%, 38%, 30%, 31% and 44% of energy consumption compared to them, respectively. The energy consumption is reduced by 50% compared to the fully local computation, and by 30% compared to the fully offloaded algorithm. This algorithm also has the highest task completion rate with the highest real-time performance. Furthermore, this paper proves a sufficient condition for the heuristic algorithm to achieve a suboptimal solution using the Cauchy–Buniakowsky–Schwarz inequality. From the performance of the DQN algorithm in the experiments, due to the strong real-time nature of the scenario in this paper and the strong uncertainty of the system environment, the model convergence speed of the reinforcement learning algorithm alone is slow, and at the same time, incorrect offloading predictions also tend to lead to higher energy consumption. Finally, the proposed algorithm in this paper is also not optimal for certain application scenarios. This algorithm uses a complex deep reinforcement learning model in order to meet the performance requirements of task arrival scenarios in real time. In contrast, for deterministic scenarios where the set of tasks to be offloaded is known and no prediction of future tasks is required, simpler heuristics, such as linear programming algorithms, etc. can achieve the same or even better performance, and the latter is clearly the better choice.

In this paper, the communication environment of the system is simplified while modeling, and the interference factor of the channel is not considered. The allocation of network resources in the edge computing system is also idealized and will be studied in detail in the next work in conjunction with SDN technology. In future work, we will also further consider the mutual cooperation among edge nodes, in order to maximize the utilization of system idle resources and further reduce the system’s energy consumption. In order to further improve this model, we will also allocate computation and storage resources in edge and cloud servers in a more granular way.