Risk-Aware Travel Path Planning Algorithm Based on Reinforcement Learning during COVID-19
Round 1
Reviewer 1 Report
What is the research motivation of this paper? Is this an academic research or an applied research?
The serial number at the end of Section 2.4 is wrong. You need to check the correctness of the citation in the paper in more detail.
How is the risk area in the map modeled by SUMO algorithm? Please specify in Section II.
The reinforcement learning methods compared in this paper are very old, which can not effectively reflect the performance of the methods proposed by the author. We need to compare the new reinforcement learning methods in recent years. In addition, the paper lacks necessary references to the compared methods.
The four methods in Figure 16 all adopt almost the same route. Why does the average total path length of the RRL-APF algorithm decrease by 7.51% and the average distance of the risk area increase by 10.32%?
More explanation and verification are needed for the formula in this paper. More detailed settings are required for parameters.
Add ablation experiments to prove the effectiveness of the artificial potential field method.
The authors should consider and think about it in this work ending these and it is interesting i.e. a mask attention interaction and scale enhancement network for sar ship instance segmentation, htc+ for sar ship instance segmentation, a polarization fusion network with geometric feature embedding for sar ship classification, balance learning for ship detection from synthetic aperture radar remote sensing imagery, high-speed ship detection in sar images based on a grid convolutional neural network, and depthwise separable convolution neural network for high-speed sar ship detection.
The author needs to improve English and the format of this article.
Author Response
Dear Reviewer 1,
Thank you very much for your review and comments on our manuscript. Our responses and revisions are detailed below. An revised manuscript using the “Track Changes” function has also been submitted.
Point 1: What is the research motivation of this paper? Is this an academic research or an applied research?
Response 1: The motivation of this research is to serve people's daily travel in the context of the Covid-19 and provide people a path planning scheme that considers both epidemic risk and travel distance. It is an applied research.
Point 2: The serial number at the end of Section 2.4 is wrong. You need to check the correctness of the citation in the paper in more detail.
Response 2: Thank you for your careful check, and the serial number at the end of Section 2.4 had been changed for Section 3.4. We have checked the correctness of the citation in the paper again, and we feel sorry for our carelessness.
Point 3: How is the risk area in the map modeled by SUMO algorithm? Please specify in Section II.
Response 3: When we use the SUMO simulator to build the road network model, we use the OSM open source map to define the location of each intersection in the road network and the connection mode between intersections. The location and coordinates of the risk area are recorded in the real epidemic data. We mapped the location of the actual risk area to the SUMO simulator in the same proportion as the map, thus defining the risk area. It has been explained in the Section 3.1 and the second paragraph of Section 4 of the article.
Point 4: The reinforcement learning methods compared in this paper are very old, which can not effectively reflect the performance of the methods proposed by the author. We need to compare the new reinforcement learning methods in recent years. In addition, the paper lacks necessary references to the compared methods.
Response 4: In the article, the Q learning algorithm and Sarsa algorithm we compared are really old, but they can reflect the great advantages of our algorithm (our algorithm has increased the initial reward value by about 1167%). At the same time, comparing the two algorithms can also show that Sarsa algorithm is better than Q learning algorithm in this scenario. In addition, the RLAPF algorithm we compared is relatively new, which was proposed by Xu Ke in 2020 (Research and implementation of intelligent travel route planning algorithm based on reinforcement learning [D]. Xidian University, 2020.). The comparison results in the article also reflect our advantages (the convergence speed of the algorithm is accelerated by 45% on average; the path length is reduced by 7.5% on average; the average distance from the risk of COVID-19 is increased by about 10%). In addition, we have indicated the reference of the algorithm in the Section 4 of the article.
Point 5: The four methods in Figure 16 all adopt almost the same route. Why does the average total path length of the RRL-APF algorithm decrease by 7.51% and the average distance of the risk area increase by 10.32%?
Response 5: The total path length of the algorithm in this paper is reduced by 7.51%, and the average distance from the risk area is increased by 10.32%, which is the analysis of the results in Scenario 1 and showed in Figure 15. As for the four methods in Figure 16 (Now it has become Figure 17), the optimal path results of them are exactly the same. We also added a detailed explanation and analysis in lines 111-222 of the article.
Point 6: More explanation and verification are needed for the formula in this paper. More detailed settings are required for parameters.
Response 6: More explanation and verification about the formula were added in the next paragraph of corresponding formula. In addition, the meaning of parameters has also been further explained, and we have also added Table 9 to illustrate the parameter settings of the four algorithms during the experiment. The headers of Table 3 and Table 8 have also been modified.
Point 7: Add ablation experiments to prove the effectiveness of the artificial potential field method.
Response 7: Ablation experiments has been added in the paper and relevant results have been analyzed. We have proved the effectiveness of the artificial potential field method and the restricted search mechanism. It was showed in lines 531-548 in Section 4.
Point 8: The authors should consider and think about it in this work ending these and it is interesting i.e. a mask attention interaction and scale enhancement network for sar ship instance segmentation, htc+ for sar ship instance segmentation, a polarization fusion network with geometric feature embedding for sar ship classification, balance learning for ship detection from synthetic aperture radar remote sensing imagery, high-speed ship detection in sar images based on a grid convolutional neural network, and depthwise separable convolution neural network for high-speed sar ship detection.
Response 8: We have considered and thought about the articles you mentioned above, and some algorithms and network models (such as balance learning, PFGFE-Net, G-CNN) for ship detection are indeed effective and interesting. These will have great application value in maritime disaster rescue and emergency military strategy formulation. We also have learned from these articles. For example, we can use relevant target detection algorithms to solve the problem of identifying the tracks of people infected with COVID-19; It can also be well applied to automatic driving target detection tasks, providing support for vehicle path decision-making. Therefore, we also put forward the prospect in the last paragraph of the article's conclusion, “In addition, it is also very important to identify the tracks of people infected with COVID-19. Some algorithms and network models (such as Balance Learning, PFGFE-Net, G-CNN) proposed by some scholars[30-34] for ship detection have also inspired us greatly. In the future, we can use relevant target detection methods to identify and track the movements of COVID-19 infected people, determine the location of COVID-19 risk related areas in the shortest possible time, and minimize the risk of COVID-19 transmission.”
Point 9: The author needs to improve English and the format of this article.
Response 9: The manuscript has been thoroughly revised and polished by native speakers, and the format of the article have been improved.
Thanks again for your valuable comments. We look forward to hearing from you soon, and hope that the correction will meet with approval.
Author Response File: 
Author Response.docx
Reviewer 2 Report
Research on risk-averse path planning algorithm during COVID-19 based on Reinforcement Learning
The authors are proposing a risk-averse travel path planning algorithm based on reinforcement learning to avoid epidemic risk in urban traffic scenarios by minimizing travel risk and travel distance concomitantly. I believe the research has useful applications in the transportation industry for safe travel path planning and pandemic spread control. However, the authors should make the following corrections to improve the quality of the manuscript.
11. It is insignificant to include ”research on” in the title of the manuscript. I suggest the following title that aligns more concretely with the content of the manuscript. “Risk-aware travel path planning algorithm based on reinforcement learning during Covid 19”.
22. There are many typos in the manuscript such as a decimal between two words that should be separated by spacing as in “COVID-19.Relevant”, “areas).Therefore”, “modelled.Since…”, “actions.This”.
33. The statement “If the health treasure It turns into….” is unclear.
44. How is n defined in Figure 2?
55. Correct the statement “… field unit vector, Its direction”, the letter I in Its should be small.
66. Define the norm symbol in equation (9).
77. There are many too-long sentences in the text running from 4 to 6 lines of texts. They should be reworked to improve the understandability of the sentences.
88. Figure 5 is not referenced in the text.
99. Replace “Field(RRL-APF)” with “Field (RRL-APF)”, “It indicate…” with “It indicates…” in all sentences where they appear, “Figure 11-13 shows” with “Figures 11-13 show”, “China implements…" with “China implemented…".
110. Define the dimension of the reward matrix and that of the Q table.
111. The reference list should be consistently formatted. Some names of authors are written in capital letters while others are small.
Author Response
Dear Reviewer 2,
Thank you very much for your review and comments on our manuscript. Our responses and revisions are detailed below. An revised manuscript using the “Track Changes” function has also been submitted.
Point 1: It is insignificant to include ”research on” in the title of the manuscript. I suggest the following title that aligns more concretely with the content of the manuscript. “Risk-aware travel path planning algorithm based on reinforcement learning during Covid 19”.
Response 1: Thanks for your valuable suggestion. The title has been changed for " Risk-aware travel path planning algorithm based on reinforcement learning during Covid 19"
Point 2: There are many typos in the manuscript such as a decimal between two words that should be separated by spacing as in “COVID-19.Relevant”, “areas).Therefore”, “modelled.Since…”, “actions.This”.
Response 2: The manuscript has been thoroughly revised and polished by native speakers，and all typos in the article have been corrected.
Point 3: The statement “If the health treasure It turns into….” is unclear.
Response 3: The sentence has been changed for " If the health kit turns into a yellow "home observation" status, and it indicates that you have been checked by the community and have visited Medium and high risk areas and other related places. "
Point 4: How is n defined in Figure 2?
Response 4: The “n” in the figure represents the number of intersections in the road network. It has been explained in the legend of Figure 2.
Point 5: Correct the statement “… field unit vector, Its direction”, the letter I in Its should be small.
Response 5: The sentence had been changed for “ is gravitational field unit vector, its direction is that the current state of the agent points to the state of the target node.”.
Point 6: Define the norm symbol in equation (9).
Response 6: We use the 2-norm of to express the value of the gravitational potential field of the agent at point o. The norm symbol was defined in the interpretation of equation 9.
Point 7: There are many too-long sentences in the text running from 4 to 6 lines of texts. They should be reworked to improve the understandability of the sentences.
Response 7: The manuscript has been thoroughly revised and polished by native speakers, and too long sentences in the article have been rewritten.
Point 8: Figure 5 is not referenced in the text.
Response 8: We have referenced Figure 5 in the last paragraph of Section 2.4.
Q9: Replace “Field(RRL-APF)” with “Field (RRL-APF)”, “It indicate…” with “It indicates…” in all sentences where they appear, “Figure 11-13 shows” with “Figures 11-13 show”, “China implements…" with “China implemented…".
Point 9: Thank you for your careful check. We had made the corresponding corrections according to your suggestions.
Point 10: Define the dimension of the reward matrix and that of the Q table.
Response 10: We have defined the dimension of the reward matrix and that of the Q table in the article. You can find it in line 408 and line 417 in the article.
Point 11: The reference list should be consistently formatted. Some names of authors are written in capital letters while others are small.
Response 11: Thank you for your careful check. All names of authors were changed to write in lowercase letters.
Thanks again for your valuable comments. We look forward to hearing from you soon, and hope that the correction will meet with approval.
Author Response File: Author Response.docx
Reviewer 3 Report
1. It helps to appreciate the paper by having a related work section. The authors should consider more recent research done in the field of their study (especially in the years 2020 and 2021 onwards). The reader may want to see how this work differs from other previous works.
2. The authors should clearly describe related work in more detail, contrasting the limitations of the related works. Moreover, the reviewer recommend to ease the overview related works by using overview tables.
3. The authors need to interpret the meanings of the variables. Some parameters and their values are unknown. It would be better to show all these parameters and explain the reason for those numbers in the table.
4. The proposed processes should be revised in a more formal pseudocode template. Moreover, the authors should include more technical details and explanations.
5. The experiment results show the performance with high accuracy, please show the parameter settings of each approach using a table.
6. More experiments and some comparisons with other up-to-date methods should be addressed or added to back your claims to expand your experiments and analysis of results further.
7. How about the computation complexity of the proposed method compared with related work? The comparison to other improved schemes (more current literature in the area) is required.
8. The conclusion and future work part can be extended to have a better understanding of the approach and issues related to that which can be taken into consideration for future work.
9. The paper is not novel in the sense that many path planning algorithms are already available with good performance. The contribution of the paper against the existing works is not well addressed.
Author Response
Dear Reviewer 3,
Thank you very much for your review and comments on our manuscript. Our responses and revisions are detailed below. An revised manuscript using the “Track Changes” function has also been submitted.
Point 1: It helps to appreciate the paper by having a related work section. The authors should consider more recent research done in the field of their study (especially in the years 2020 and 2021 onwards). The reader may want to see how this work differs from other previous works.
Response 1: Thanks for your valuable suggestion. We have added the relevant work of recent research and made some comparisons in the Introduction of the paper. Now, in the relevant research work, the vast majority are the research results of the past three years.
Point 2: The authors should clearly describe related work in more detail, contrasting the limitations of the related works. Moreover, the reviewer recommend to ease the overview related works by using overview tables.
Response 2: We have described related work in more detail. We have added a detailed description of the relevant work and contrasting the limitations of the related works in an overview table (Table 1).
Point 3: The authors need to interpret the meanings of the variables. Some parameters and their values are unknown. It would be better to show all these parameters and explain the reason for those numbers in the table.
Response 3: The variables and the reasons for the numbers in the relevant tables have been further explained in detail. In addition, we also explained the meaning of each number in some paragraphs near the relevant tables and formulas. However, we could not to show all these parameters and those numbers in the table, because it will take up too much space.
Point 4: The proposed processes should be revised in a more formal pseudocode template. Moreover, the authors should include more technical details and explanations.
Response 4: A pseudocode of the algorithm process and more technical details and explanations have been added in the Section 3.5 of the article.
Point 5: The experiment results show the performance with high accuracy, please show the parameter settings of each approach using a table.
Response 5: The parameter settings of each algorithm have been shown in Table 7 and explained in the article.
Point 6: More experiments and some comparisons with other up-to-date methods should be addressed or added to back your claims to expand your experiments and analysis of results further.
Response 6: Thanks for your valuable suggestion. We have added some ablation experiments and parameter comparison experiments in the paper, and we have proved the effectiveness of our proposed method. We also explored the influence of different probabilities of using the artificial potential field method on the convergence results of the algorithm. The process of experiment and conclusion analysis have been elaborated in the article.
Point 7: How about the computation complexity of the proposed method compared with related work? The comparison to other improved schemes (more current literature in the area) is required.
Response 7: The time complexity of this algorithm and related algorithms is O(n2). However, compared with the comparison algorithm, especially the Artificial potential field reinforcement learning algorithm proposed by some scholars in recent two years (Wang Keyin, et al. Path Planning for Mobile Robot Using Improved Reinforcement Learning Algorithm,2021; Xu Ke. Research and implementation of intelligent travel route planning algorithm based on reinforcement learning, 2020.), the algorithm in this paper takes less time when other parameters are the same.
Point 8: The conclusion and future work part can be extended to have a better understanding of the approach and issues related to that which can be taken into consideration for future work.
Response 8: The conclusion and future work in the article have been extended, and the methods and related problems that can be considered in the future work have also been described in the last paragraph in the article, “In real life, people may travel through multiple modes of transportation. Therefore, in the next stage, it is necessary to consider combining multimodal transport and dynamic road network information. This may involve the connection between different modes of transportation. We need to further study the coordination between public transport such as buses and subways with fixed time and on-line car hailing and taxis with variable time. At the same time, we should also consider the traffic capacity of the road network, the traffic efficiency of road intersections, and the travel habits and preferences of residents. In the near future, developing a travel route planning application with intuitive user interface is the re-search content of the next stage. In addition, it is also very important to identify the tracks of people infected with COVID-19. Some algorithms and network models (such as Balance Learning, PFGFE-Net, G-CNN) proposed by some scholars [30-34] for ship detection have also inspired us greatly. In the future, we can use relevant target detection methods to identify and track the movements of COVID-19 infected people, and determine the location of COVID-19 risk related areas in the shortest possible time, and minimize the risk of COVID-19 transmission”.
Point 9: The paper is not novel in the sense that many path planning algorithms are already available with good performance. The contribution of the paper against the existing works is not well addressed.
Response 9: Thanks for your valuable suggestion. Our article is really not particularly novel in terms of algorithms, but we have also made some innovations. The innovation of this paper was that we propose a method to extract road network model and impedance matrix based on SUMO simulator, which solves the complex problem of manually constructing road network model and greatly improves the efficiency of road network modeling. We designed the state and action space of agents in a "point-to-point" way, which is quite different from the fixed action space designed by most scholars. This modeling method can be well used in urban traffic path planning. Our restrictive search mechanism can effectively improve the traditional operation of initializing Q tables. Although some scholars have proposed the artificial potential field method before, we have made corresponding improvements to it, and dynamically adjust the greedy coefficient through the improved greedy strategy. This makes our algorithm more efficient than those proposed by relevant scholars in recent two years. On the other hand, although many path planning algorithms have good performance, there are few methods for residents' travel path planning during the new epidemic. In addition, this article is an applied research. We hope that the models and algorithms proposed by us can provide people with a travel plan that considers both epidemic risk and path length, so as to reduce people's travel risk and improve travel efficiency. The algorithm and model proposed in this paper can better achieve this goal. In addition, the article discusses that compared with the algorithm proposed by some scholars in recent two years (Wang et al. Mobile robot path planning using improved reinforcement learning algorithm, 2021; Xu. Research and implementation of intelligent travel route planning algorithm based on reinforcement learning, 2020), our algorithm will greatly reduce the convergence time. The results obtained in the article also show that our algorithm can ensure that the average path length is minimized while staying away from the risk area of COVID-19.
Thanks again for your valuable comments. We look forward to hearing from you soon, and hope that the correction will meet with approval.
Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
Accept. No more comments.
Reviewer 3 Report
This paper has edited and revised according to the reviewer's suggestions.
