Virtual Machine Migration Strategy Based on Multi-Agent Deep Reinforcement Learning

Round 1

Reviewer 1 Report

This paper proposed a virtual machine migration strategy based on multi-agent deep reinforcement learning to ensure real-time performance. Overall, the article presents a good idea that only needs local information without any communication between servers. However, there are some weaknesses that the authors should particularly pay attention to and handle:

 

1. It would be better if the authors compare their work with other related work. For example, what difference between this work and the work done in these articles:
   • “Security-Aware Data Offloading and Resource Allocation For MEC Systems: A Deep Reinforcement Learning”.
   • “Joint computation offloading and task caching for multi-user and multi-task MEC systems: reinforcement learning-based algorithms”.
   • “Enhanced resource allocation in mobile edge computing using reinforcement learning based MOACO algorithm for IIOT”.
2. The following papers were done and related to Migration and offloading strategies so, the authors can add them as reference:
   • MCACC: New approach for augmenting the computing capabilities of mobile devices with Cloud Computing.
   • Enhanced resource allocation in mobile edge computing using reinforcement learning based MOACO algorithm for IIOT
   • An Enhanced Version of the MCACC to Augment the Computing Capabilities of Mobile Devices Using Cloud Computing.
3. The abbreviation must be written beside the words when it appears for the first time. So, the authors need to check all of them. Example: MEC was mentioned without writing the meaning the first time. Also, “Multi-Agent Deep Reinforcement Learning, MADRL) was written twice.
4. You have to check the references carefully because there was an error in the references on page 2.

Author Response

Please see the attachment.

Author Response File: Author Response.doc

Reviewer 2 Report

The paper is well organized with proper structure and length. The bibliography is sufficient and well given.

Specifically, the technical terms are explained in detail and the topic of the paper is clear and understandable.

Mathematical models are well written and appropriate refereed.

The presented methodology and the results are clearly communicated, with the necessary background for the readers included in the paper.

The review of the state-of-the-art is sufficient. It includes lots of references to other relevant studies that have been previously proposed for the discovery of relations.

The novel contribution of the paper is highlighted, as well.

The conclusion section includes a discussion about the results obtained by this work and the previous works on the analysis of the same or similar data.

Author Response

Thank you for giving us the opportunity to revise our article entitled “Virtual Machine Migration Strategy based on Multi-Agent Deep Reinforcement Learning” (applsci-1317631). We would like to give our thanks for the positive comments. And we are very grateful for the constructive suggestions from you. We have modified some syntax errors and descriptions to enhance the quality of the paper.

Reviewer 3 Report

• What type of virtual machine migration mechanism is targeted in the optimization? For this concern, the authors need to discuss clearly on the VM migration mechanisms. It could be a totally different story if different VM migration strategies are adopted. Also, how exactly the algorithm is adopted to perform the VM migration of data/files? Please check the stage of VM migration and discuss that. For this issue, I would recommend discussing based on a related reference DOI: 10.1016/j.future.2015.08.017
• Is there any reason to divide time into equal pieces of 1 second as mentioned in line 260?
• How are the tasks and servers modeled and simulated in the environment? The authors need to give clear information and development of this.
• In Figure 3, there is an abnormal behavior of the graph representing DIS_AC when the number iterations are close to 750 or 500, Is there any reason for these abnormal reward values? Please give a clear discussion.
• I don’t think it is Behrman equation as mentioned in 215. Please check the name and the format of the equation carefully.
• Figure 2 should be bigger and sharper
• Text error in line 251.

Author Response

Please see the attachment.

Author Response File: Author Response.doc

Round 2

Reviewer 1 Report

The authors improved their manuscript significantly. They were able to respond to my previous question satisfactorily. So, I will accept this paper for publication. But, you have to check the references carefully because there was an error related to references on page 3.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

• Authors have improved some of the aspects mentioned by the reviewers' team, however, have not succeeded in addressing my major concerns. In this context, I am missing the added value of the paper, to be published as a journal article.
• The authors did not give clear discussions and assumptions on the VM migration strategies as discussed in DOI:10.1016/ j.future.2015.08.017. Therefore, it is hard to convince the effectiveness of the proposed MADRL algorithms for VM migration without concerning its strategies. I have a major concern on this issue in this study. It seems to be a VM allocation/distribution using RL rather than VM migration. The authors need to clarify which VM migration strategy has been assumed to be adopted.
• From line 305-313: in the algorithm DIS_AC, why the agents do not communicate with each other to improve their learning capabilities? What if the authors consider attention in the proposed algorithms?

 

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 3

Reviewer 3 Report

The authors provided more explicit and comprehensive discussions on the concerns raised in previous round, which distinguished the differences and highlighted the contributions of this study in comparison with previous works. 

The VM migration is a sophisticated process, but of paramount importance in modern computing paradigms such as cloud/fog/edge computing. As discussed by the authors, it encounters with many issues such as uncertain failures. 

The discussion in the current submitted manuscript has clarified the VM migration strategy and process to be adopted in the deployment of the proposed MADRL algorithm. It also clearly presents the limitation of the proposed algorithm due to not taking into account the attention behaviours. 

However, I believe this study has opened a broad avenue in the adoption of AI models for optimal VM migration strategies. I think the future works of this study could consider uncertain failures, attention mechanisms and different VM migration strategies, etc. 

Under the above observation, I believe this work shows a significant contribution in the state-of-the-art studies on VM migration mechanisms. 

Therefore, I would like to recommend an acceptance for this study in the current form of the manuscript for publication in the targeted journal.