HCOME: Research on Hybrid Computation Offloading Strategy for MEC Based on DDPG

Round 1

Reviewer 1 Report

The paper presents an interesting approach that is very relevant today in enabling IoT devices into our everyday lives. Generally, the article is well-written and the methodology is scientifically sound. Nevertheless, a couple of issues need to be resolved in order for the paper to be accepted:

- be careful with abbreviations - upon the first usage please provide the full meaning (e.g. "UD" in the first chapter)

- Figures 3 - 6: Please provide the units of the values shown in the figures (e.g. Delay - sec., msec, ... or processing power, energy consumption, ...)

- The final section 6 is very poor - please summarize your conclusions and draw them from your results. provide at least a short overview of open things/future work

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Based on a careful analysis, I can formulate the following remarks:

1) The aim of this article, based on the authors’ scrupulous investigations, is develop and validate by several experimental investigations a Hybrid Computation Offloading for Mobile Edge Computing (MEC) based on deep deterministic policy gradient framework (HCOME).

2) The topic represents in my opinion a relevant approach of the proposed theme in the field, based on meticulous theoretical and investigations, correlated with experimental results.

3)  In comparison with other published material, the authors' contribution adds to the subject area a new approach/methodology, with several significant contributions. First, one has to underline the fact that most of the latest techniques focus only on latency or energy consumption and ignore the computational and network load balancing of the whole system. In addition, even though a few papers consider load balancing, most of they can only perform binary offloading and not dynamic offloading at arbitrary ratios, which severely bounds the offloading decision.

In order to solve this issue, the authors propose a Hybrid Computation Offloading for Mobile Edge Computing (MEC) based on deep deterministic policy gradient framework (HCOME).

Their framework can perform hybrid computation offloading, allocate computation resources and network resources of MEC system in any ratio, and quickly adapt to dynamic network environment; the total delay and energy consumption of MEC system is taken as the optimization target, and the stability of the system is improved at the same time.

The experimental results show that the proposed scheme can effectively reduce the delay and energy consumption, as well as balance the network load and computational load, and has obvious advantages compared with other comparison algorithms.

Consequently, their proposal is better than the other reported ones from the literature.

4) In these modern times, with the advent of Internet of Things (IoT), the demand for central cloud real-time data storage, access, and processing has grown exponentially, and the huge amount of data generated by IoT devices will increase in the near future.

Consequently, forwarding such huge amount of data to the cloud infrastructure will lead to latency issues and may cause network bottlenecks in the future. In this sense, the edge computing, an emerging network paradigm has emerged as a promising solution to support mobile access in multiple networks for IoT. Most existing studies on task offloading rarely consider the load balancing of computational resources of edge servers. In the same time, many researchers have combined Software Defined Network (SDN) with MEC in issues such as resource allocation, minimizing latency, edge caching, and energy consumption. For example, a research team designed an intent-based network control framework for data propagation in the vehicular edge-computing ecosystem in order to meet the stringent latency requirements for applications in the underlying network infrastructure. Several deep reinforcement learning-based methods have been proposed; however, these methods are weak in adapting to new environments because they are inefficient in sampling and require comprehensive retraining to learn update strategies for new environments.

In order to overcome this weakness, one other research team proposed a task offloading method based on meta-reinforcement learning, which can quickly adapt to new environments with a small number of gradient updates and samples. This new offloading method can reduce the latency by 25% while being able to adapt quickly to new environments. Supplementary, there are some other limitations of MEC. In this sense, one can mention the high cost of infrastructure deployment and maintenance, and the severe pressure on MEC servers from the complex and variable edge-computing environment. Consequently, it becomes a huge challenge to rationally allocate computational and network resources to meet the needs of mobile devices under the variable MEC conditions. Unfortunately, only few works reflect this issue.

The authors of this contribution proposed to design a hybrid computational offloading scheme for MEC based on Deep Deterministic Policy Gradient (DDPG) that can adaptively allocate computational and network resources to achieve overall latency minimization, i.e., effectively reduce the overall computational overhead while ensuring full utilization of network and computational resources in the system.

The developed scheme they validated by several scrupulous experimental investigations.  These numerous experimental results, presented in detail by the authors, show that the algorithm outperforms the comparison algorithm and can reduce the system latency by 20%, while the network load and computational load are also more stable.

 5) In my opinion, the presented conclusions are suitable related to their research results and prove that they reached the proposed goal.

6) The references in my opinion are very appropriate and their number underlines the scrupulosity of the authors.

7) In this paper, the graphical illustration is well conceived and realized and consequently they contribute to a better understanding of the performed theoretical investigations as well as to underlining the experimental validation of the proposed methodology.

I encourage publishing in a new contribution their further results.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

This paper propose DDPG based partial task offloading strategy. The paper needs to clarify the novelty by discussing and compare this paper and the existing deep reinforcement learning (DRL) based solutions that also consider partial task offloading, e.g. Ale et al 2021, Truong et al 2021, and Deng et al 2021. Especially, this paper and the paper from Deng et al 2021 look very similar.

In addition, the paper needs to justify the rationale of using reinforcement learning to solve the problem. The paper assume that the “tasks can be offloaded to the MEC server for computation in arbitrary proportions”. This assumption is not realistic. In addition, with this assumption, it seems that the optimisation problem is no longer a mixed-integer programming problem. It can probably solved directly without using DRL.

The paper claims that the solution can “balance the network load and computational load”, however, the problem formulation does not show this consideration. It seem the optimisation objective only focues on minimize the overall delays from all the users.

The evaluation section only considered a small number of user devices. What is the expected number in the normal situation? Will the device dynamics, randomly join or leave the network, affect the algorithm performance?

 

Ale, L., King, S. A., Zhang, N., & Sattar, A. R. (2021, July). Deep Reinforcement Learning Aided Task Partitioning and Computation Offloading in Mobile Edge Computing. In 2021 IEEE/CIC International Conference on Communications in China (ICCC) (pp. 340-345). IEEE.

 

Truong, T. P., Nguyen, T. V., Noh, W., & Cho, S. (2021). Partial computation offloading in NOMA-assisted mobile-edge computing systems using deep reinforcement learning. IEEE Internet of Things Journal, 8(17), 13196-13208.

 

Deng, X., Yin, J., Guan, P., Xiong, N. N., Zhang, L., & Mumtaz, S. (2021). Intelligent delay-aware partial computing task offloading for multi-user industrial internet of things through edge computing. IEEE Internet of Things Journal.

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

The title as well as the introduction raised expectations about the manuscript and research done. The topic you are addressing would be a relevant addition to existing literature. Thank you for this valuable contribution. I will structure my feedback in (a) general remarks (these comments cover feedback applicable in the entire manuscript), and (b) specific remarks (feedback on sentence and/or word level). The specific remarks can include a quote from your original manuscript to refer to a specific section. Some of the specific remarks will refer to specific lines (e.g., L 15-16).

 

General remarks

This is a well written and easy to read paper where authors introduce a hybrid computation offloading framework based on Deep Deterministic Policy Gradient (DDPG) for IoT devices to outperform the existing algorithms. I do acknowledge the potential of the manuscript and the topic. The Introduction and methodology in this paper are presented clearly. The research problem is well stated in the Introduction and the methodology is well explained in the System Model and Problem Formulation section. The paper is well supported by mathematical equations and algorithm flow.

However, in abstract, the authors should highlight the research novelty and elaborate more to improve the problem statement to clarify the factors causing the instability in the mobile edge computing environment. Same for the conclusion, the authors need to improve this section by summarizing the results achieved in the Experimental Analysis section.

The overall manuscript still requires revisions to correct grammar mistakes, typo mistakes, make sure to state the full name before the acronym for example DRL. Define your acronyms before using them.

 

Specific remarks

L74-77 I suggest deleting this sentence as it is too early to reflect on the research results in the introduction.

L 147 “t” capital letter

L 137 Full name required for the acronym “DRL”, brief definition is expected

Eq.1 Confirm in the text C is the number of CPU cycles needed to complete task Si

Eq.4 How was this equation was driven. If it is based on a well know equation, then a citation for that resource is required

Eq.18 How was this equation was driven. Justify how it’s possible to add two parameters (energy and time) in one value? They are different

L347-361 It is better to include all the simulation settings in a table

L392-393 Figure 4 is showing the HCOME algorithm performing the same as the Q-Learning algorithm. Why didn’t the hybrid approach help in this aspect?

Comments for author File: Comments.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

The revised paper does not address the major concerns from this reviewer. Actually, this reviewer thinks the design is flawed. The reason is given below:

(1) The paper uses the same \omega_i to define the offloaded data size and the correspondent function (e.q. 1 & 3). This is normally not true, e.g. an the data size of an infinite loop is fixed.

(2) In the problem formation, the objective function (e.q. 13) only contains delay and energy consumption, however, the reward function contains parameters for network and computation load balance. If so, the problem formation section and the reinforcement learning section (the section to solve the problem) have no relations. Why is the problem formation section is required? 

(3) The balancing related parameters are not defined (b_net & b_cp). The previous section only defined var(l_net) & var(l_cp). Furthermore, what are the constraints of the load balancing parameters for the network and computation? In addition, this load balancing strategy is highly theoretical. Suppose a user device does not have task to offload. Will this situation limit other devices to offload if the algorithm is trying to “balance” the load? The test results should be given to show the performance for this perspective. 

(4) The paper claims the formed problem (e.q. 13) is a mixed integer programming problem. The reviewer highly suspect this conclusion. In this equation, only the objective function contains non-linear functions (i.e. max). However, max can be easily converted to a linear programming problem.

(5) This paper claims that it is different from a few previous papers. It should provide evaluation results to prove that the paper outperforms other papers.

An interesting point of the paper is that it considered both the energy consumption of the user devices and the edge server. It is interesting to see how the energy are consumed in each device/server.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 3

Reviewer 3 Report

The paper has been significantly improved. However, some additional modifications are required to clarify the contribution:

 

(1) In relation to the claim of the NP-Hard problem (Line 269). The reviewer still thinks that the problem is not an NP-hard problem after making \omega_i continuous. It essentially simplifies an NP-hard problem to a linear programming problem.

 

(2) The evaluation section added ref [16] into certain results. As the major difference between this paper and ref [16] is in considering energy consumption and load balancing, the correspondent evaluation results should include ref [16].

 

(3) The evaluation setup should be clarified further, e.g. more details on the task generation (is the task generated immediately after completing the current task or after waiting for a while?), what is the experiment duration (i.e. how many tasks have been executed by each UA during the experiment)?

Author Response

Please see the attachment.

Author Response File: Author Response.pdf