Tugboat Scheduling Method Based on the NRPER-DDPG Algorithm: An Integrated DDPG Algorithm with Prioritized Experience Replay and Noise Reduction

Round 1

Reviewer 1 Report

Dear Authors,

The manuscript entitled “Tugboat Scheduling Method Based on the NRPER-DDPG Algorithm: An Integrated DDPG Algorithm with Prioritized Replay and Noise Reduction” presents an interesting tugboat scheduling problem with the addition of reinforcement learning.

However, to raise the value of the article, please pay attention to these issues and make every effort to improve and supplement article.

1. Not all references are cited in the main text of the manuscript. For example, reference [1],[2] and [3] are not cited in the main text. Please, cite all references in the main text from the reference list or remove all items from the reference list that have not been cited in the text.
2. As I understood, there is limited research on the application of reinforcement learning in tugboat scheduling. But, is there a research on the application of reinforcement learning in scheduling problems? If there is, please add a new discussion section where you compare the findings of your study with the existing scientific literature in terms of increased and decreased results values percentages.

Kind regards

Author Response

Dear Reviewer,

I hope this letter finds you well. Firstly, I would like to express my deepest gratitude for your invaluable feedback on our manuscript. Your professional guidance has been instrumental in helping us refine our work.

In response to your concerns regarding literature citations, we have conducted a thorough review and made the necessary additions. All relevant citations are now fully incorporated into the text, and to facilitate your review, we have specifically highlighted them in red font.

Furthermore, addressing your suggestion on the literature review section, we have added a dedicated paragraph summarizing the application of reinforcement learning in port scheduling research. This comprehensive overview provides robust theoretical support and demonstrates the feasibility of applying reinforcement learning to tugboat scheduling in port operations.

Once again, I extend my heartfelt thanks for your thoughtful guidance and support. We sincerely hope that these revisions have enhanced the quality of our manuscript, aligning it more closely with academic standards and requirements. We look forward to your further review and any additional valuable feedback you may have.

Yours sincerely.

Reviewer 2 Report

This manuscript attempts to develop a solution to the tugboat scheduling problem using a deep reinforcement learning approach. It presents a fuzzy model that considers the minimization of the fuel and time costs of the process.  The proposed modified deep reinforcement learning approach is compared to two traditional reinforcement learning approaches. The results show its superiority, which is obviously expected.

Unfortunately, this manuscript neglects a large body of literature in scheduling theory and algorithms can be used for solving the stated problem. The considered problem is traditionally known in the scheduling literature as the parallel machine scheduling problem for which very efficient metaheuristic approaches exist in the literature. This manuscript neglects this body of literature and tries to enforce the usage of questionable techniques that are based on machine learning.

Neglecting previous contributions in the scheduling literature limits its contribution, and without having a solid computational comparison with the state-of-the-art scheduling approaches makes its results questionable.

For the above reasons, this manuscript is not recommended for publication.

The manuscript contains some typos, and using abbreviations without providing full description is not appropriate.

Author Response

Dear Reviewer,

I would like to express my deepest gratitude for your invaluable feedback on our research. Your comments have provided crucial guidance for us to refine our work. Thanks to your help, we have identified the shortcomings in our original study and made corresponding supplements and modifications.

Firstly, we fully agree with your observation regarding the lack of comparison with the most advanced scheduling algorithms in our research. To address this issue, we have specifically added the latest research findings on the application of reinforcement learning in the field of port scheduling to the literature review section, aiming to demonstrate the rationality and forward-looking nature of our research. Furthermore, we have ensured that all references are properly cited in the text.

Secondly, during the revision period, we have rebuilt the examples and conducted more rigorous comparative experiments. Following your suggestion, we have compared the performance of our method with the NSGA-II tugboat scheduling method mentioned in the latest literature, as well as the most widely used mathematical solver, CPLEX. The results of these comparative experiments prove the feasibility and effectiveness of our proposed method. Additionally, in the final example verification section of the article, we have also conducted comparative experiments with the NSGA-II and CPLEX mathematical solvers to demonstrate the practical value of our research.

Moreover, we have paid special attention to your comments on the use of English. To this end, we have added necessary descriptions after English abbreviations and corrected typos in the text.

The modifications mentioned above have been highlighted in red and annotated for your reference.

Once again, I sincerely thank you for your valuable feedback. Your careful guidance has greatly contributed to the advancement of our research. We genuinely look forward to your further guidance to further enhance the quality of our paper.

I wish you academic progress and success in your work.

Yours sincerely.

Reviewer 3 Report

- The introduction appears to be too brief to fully capture the complexity and significance of the topic. Consider expanding it to provide a more comprehensive overview of the issues at hand.

- Omit the section on "Thesis Outline" as it may not be necessary and could potentially detract from the flow of the introduction.

- Clarify the purpose of equations 1 to 12 and ensure they are presented more conventionally. Additionally, each equation should be accompanied by a clear definition of its parameters to aid understanding.

- The section on "Problem Description" requires revision to enhance clarity and coherence. Ensure that the information presented is structured logically and effectively communicates the problem statement to avoid confusion for the authors.

- Consider whether Tables 3 and 5 are essential to the presentation of your work. If not, they could be omitted to streamline the manuscript.

- Remove the unnecessary entry in Table 1, Line 652, to avoid any confusion or redundancy.

- Review and potentially revise Figures 9 and 10 to ensure they are clear and effectively convey the intended information to the readers.

- Expand the conclusion section to provide a more comprehensive summary of the findings and their implications. A more detailed conclusion will provide a stronger sense of closure to the manuscript.

Author Response

Dear Reviewer,

First and foremost, I want to express my deepest gratitude for your invaluable feedback. Your comments have not only highlighted the shortcomings in our research but also provided us with clear directions for improvement.

As per your request, we have removed the "Thesis Outline" section and rephrased the problem description to make it more concise and precise. Additionally, we have carefully examined the equations in our mathematical model and rectified any errors.

In terms of the literature review, we have specifically included recent studies on the application of reinforcement learning in port scheduling. These references not only enrich our research context but also provide solid support for our methodology, further justifying the rationality of our study.

Regarding your suggestions on our charts and tables, we have made the necessary modifications and annotated them in the article for your reference.

Moreover, to enhance our research and expand the conclusion section, making our findings more convincing, we have constructed examples of different scales and conducted fresh experiments during the revision period. We have also compared our method with the NSGA-II approach mentioned in the latest research literature and the widely used mathematical solver, CPLEX. The results of these comparative experiments further validate the reasonableness and effectiveness of our study.

Once again, I sincerely thank you for your precious feedback. This review process has been immensely beneficial for me, and your thoughtful guidance has greatly contributed to advancing my research work. I genuinely look forward to receiving further advice from you to further enhance the quality of our paper.

Yours sincerely.

Round 2

Reviewer 2 Report

The consideration of CPLEX solver and the NSGA-II metaheuristic in the computational comparisons represent a sufficient answer to my previous concern. However, the learning time needed by the proposed reinforcement learning approach is neglected in the comparison, even though it represents one major drawback of the proposed approach.

Another important concern that arises from the conducted comparisons with CPLEX and NSGA-II is that the comparisons must not be based on the absolute best values obtained for the two objectives (the fuel cost and the time cost). For such a multi-objective optimization problem, measures such as gravitational distance or hypervolme should be used to compare the non-dominated solutions obtained by all three methods.

Author Response

We sincerely appreciate your valuable feedback on our manuscript. Based on your comments, we have deeply recognized that using absolute value comparison for multi-objective analysis in our first revision experiment was indeed inappropriate. We have made improvements to your feedback, and the improved parts have been annotated and highlighted in red font in the article, and for a more detailed explanation, please refer to the attached documents.

1. Regarding multi-objective comparison: In our study, the fuel cost and time cost of tugboats are integrated in the reward function of reinforcement learning through weighted nonlinear summation. Therefore, we cannot directly apply metrics such as gravitational distance or hypervolume to comprehensively compare all non-dominated solutions. In our first revision, we chose to compare our results with an optimal solution obtained through Critic weighting from the NSGA-II algorithm. This was an oversight on our part, as we did not clearly elaborate on this point in the first revision. After carefully analyzing your review comments, we agree that such a comparison method is inappropriate.

To clarify this point for you, we have provided a detailed explanation in the first section of the attached "Comparative experiment explanation" document. Additionally, in our previous work, we have reproduced and compared the solutions of algorithms such as PSO, GA, ALNS, and NSGA-II for tug scheduling problems. We have included the comparison results of these heuristic algorithms in the second section of the attachment for your reference.

2. Regarding algorithm solution time: After thorough analysis and consideration, we have decided to modify the comparative experiment section to compare our algorithm with the single-objective ALNS algorithm. In the revised experimental section, we have adopted the commonly used standard deviation and solution time comparison for single-objective solutions. We compared the standard deviation and solution time for finding the optimal solution after ten iterations of both algorithms, thereby validating the effectiveness of our proposed algorithm in terms of both solution quality and speed.

Thank you again for your careful guidance and valuable suggestions regarding our research. We look forward to your further feedback and are willing to make additional modifications and improvements as needed.

Author Response File: Author Response.pdf