Sustainable Smart Cities through Multi-Agent Reinforcement Learning-Based Cooperative Autonomous Vehicles

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

First, I would like to thank the authors for submitting their paper. The research on autonomous vehicles is growing exponentially as their level of usage is rapidly increasing. However, the specific topic is well researched and examined, which is not reflected in the references used and presented in the manuscript.

The authors often use the term sustainable urban mobility (SUM) to emphasize the importance of their research in relation to sustainable urban mobility principles. However, these principles are not mentioned once. The authors mention on page 2: "Central to the vision of sustainable cities, AVs navigate the intricate and dynamic traffic environments." I would like to mention some of SUM's basic principles: a) planning compact and accessible cities; b) developing transit-oriented cities; c) getting the infrastructure right; d) encouraging walking and cycling; e) advancing smart mobility management; f) enhancing public transport and shared mobility; g) parking: managing first, not supple; h) electrifying all vehicles; i) winning the support of stakeholders; j) empowering cities to avoid, shift, and improve. Electric AVs meet only one of these principles; therefore, such a statement should be avoided. But this is just an example of the authors' approach. Many researchers (Richter et al., 2022, Smart cities, urban mobility, and autonomous vehicles: How different cities need different sustainable investment strategies) highlight through their research the fact that AVs can be useful and aligned with SUM only for megacities (New York, Shanghai) and not all cities. Can AVs assist so that a city can become "smarter"? The answer is yes, but the proper question is in what ways AVs can promote SUM. To answer this question, we need to wait a few years for the infrastructure to become smart and the interaction between it and AVs to be well established and assisted by AI.

The paper examines multi-AV lane-changing scenarios within mixed traffic, claiming that the author's approach is better. However, authors do not refer to the extensive research implemented so far (Atagoziyev et al., 2016, Tang et al., 2023, Monteiro & Ioannou, 2023, Caldiran et al., 2021, Xing & Jakiela, 2018, Yuan et al., 2023, etc.) not only to strengthen their claim but also to enrich their literature review, which is very poor.

Concerning a rather technical issue, Sections 3 and 4 are developed using bullets. My suggestion to the authors is to avoid such an approach, especially for entire sections. I would also like to point out that the authors do not have to develop an algorithm for their proposed methodology. Also, authors should avoid cutting text by presenting figures (Figures 2, 3, and 4, Table 2), thus reducing the readability of the manuscript.

There are a few things that I cannot understand:

1) Why do the authors refer to maximum deceleration as eco-deceleration?

2) Have the authors taken into consideration what the traffic code and regulations impose as priorities and safe conditions for a vehicle to change lanes (especially in high-density urban environments)? It seems to the authors that politeness is crucial but not relevant to driving conditions, especially lane changing, as these issues are regulated by the Traffic Code.

3) In Section 5.2, the authors mention that "the study simulated high-density vehicle traffic in a modified highway-env simulator, incorporating sustainability metrics such as emission levels and energy efficiency." However, I could not find any figures, tables, or other comments on any findings regarding this.

Overall, I am not convinced of the novelty of this methodology. 

Author Response

We thank the reviewer for his pertinent comments. In the attached document, we provided answers to the reviewer's concerns.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The study presents the Multi-Agent Actor-Critic (MA2C) algorithm as a new method to improve urban mobility in smart cities. Its effectiveness in managing traffic flow, optimizing passenger comfort and promoting collaboration between agents is assessed. The results show that the algorithm is better than existing models. It has the potential to make interactions between autonomous vehicles and human-driven vehicles. The study highlights the role of the algorithm in sustainable urban transport, recognizes limitations and suggests directions for future research.

Key findings:

- The MA2C algorithm is adaptable to dynamic traffic patterns in high-density urban environments, supporting sustainability and efficient traffic management in smart cities. Comparative analysis with established MARL methodologies highlights the robust performance of MA2C, particularly in less congested urban scenarios. The algorithm is effective due to its multi-objective reward function and parameter sharing.

- An illustration of a lane change scenario involving autonomous vehicles shows how the algorithm can manage interactions and optimize passenger comfort during complex maneuvers.

- The text ends with a discussion and implications. The results of the study contribute significantly to the concept of sustainability in smart cities. They highlight the importance of localized reward mechanisms and collaborative learning in urban traffic management. The study recognizes limitations related to passenger comfort and assumptions of universal behavior. Emphasizes the need for more empirical studies.

Although the research presents promising results and contributions, it is important to recognize potential weaknesses and limitations.

One aspect to consider is the assumption and generalization made in the study, which assumes universal behavior for Human Driven Vehicles (HDVs). This may not accurately represent the diverse driving patterns in real-world smart city scenarios. To ensure that your algorithm can be applied to various traffic conditions and driver behaviors, it is crucial to address this limitation.

Also, the study highlights the importance of passenger comfort metrics. To reinforce the discussion, it is worth considering the possibility of providing a more detailed analysis of the limitations and challenges involved in quantifying and incorporating passenger comfort metrics.

Furthermore, the applicability and feasibility of the study could be improved by addressing challenges related to real-world implementation, such as regulatory considerations, infrastructure requirements, and potential social concerns. The performance of the algorithm is evaluated in a simulated environment.

Finally, when comparing MA2C with other MARL methodologies, it would be beneficial to provide a detailed analysis of the strengths and weaknesses of each approach, particularly in the context of specific smart city challenges. This could enrich the discussion.

Addressing these potential weaknesses will not only increase the credibility of your research, but will also guide future work towards refining and extending the proposed algorithm.

Author Response

We thank the reviewer for his pertinent comments. In the attached document, we provided answers to the reviewer's concerns.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

Coments EN:

The article is interesting, but there is a lack of a more detailed description of the concept in the introduction. It is necessary to supplement information about the type of transportation the study is intended for (public transport or personal). Is there already a smart city where the algorithm could be used? What are the limitations of the system regarding human driven vehicles (HDV)?

It is important to check the reference; in the article, I found a reference to source 38, while there are only 17 sources. Among the sources, there are quite a few from the author Louati, who is the author of the article.

In line 42: 'innovative MARL technique' - specify what makes it innovative.

Table 1 - the text in the table is very small.

In line 83 - at the beginning, explain what you are trying to improve and why.

In line 92 - the first use of the abbreviation 'MOBIL.' You should explain it. The explanation is provided in the second half of the article."

In lines 94 to 95 - explain how and based on what you will incorporate these parameters?

In line 108 - 'receives a state from the urban environment' - In what form is it expressed?

In line 123 - 'a POMDP framework' - What is it? The abbreviation POMDP is not explained.

In lines 132 to 133 - 'each AV (agent) communicating with neighboring vehicles to ensure smooth traffic flow' - Are only autonomous vehicles (AV) considered? In which city do you envision the majority of vehicles on the highway being AVs? Is this already applicable? How does the system communicate in the case of a human-driven Vehicle (HDV)? Can the system obtain any useful data from HDVs? How does communication work in general? What data is exchanged?

In line 135 - 'AV to react based on local observations' - Are there any limitations on the number of human-driven vehicle HDVs in the vicinity of an autonomous vehicle (AV)?

In line 143 - after the expression 'reward function,' add the variable designation from equation 1.

Equation 1 - the explanation of the calculation of individual rewards is provided, but it does not explain how the weights (omegas) are calculated. How did you define them?

In lines 156 to 157 - should the term 'eco-friendly' be replaced with 'safety'? If I drive without a collision, isn't that 'safety driving'?

In line 163 - 'Distance to the preceding vehicle' - How do you obtain it? Using lidar or camera?

In line 165 - 'eco-factor' - How was it determined?

Equation 4 - It would be helpful to clarify which variables are directly measured and which are empirically determined, or based on what they are determined.

In line 171 - 'Lane change comfort factor' - please explain what is meant by this.

In line 173 - 'fuel consumption, emissions' - also include their designations in Equation 6.

In lines 186-187 - additional information is needed on the process of this machine learning and the input data used.

 

Algorithm 1 - shouldn't the pseudocode have its label similar to a figure?

Explain the variables in bullet point 1.

Explain how the state of AV is defined in bullet point 6.

What actions can occur in bullet point 7?

 

In line 200 - 'by integrating environmental data into state' - What kind of environmental data? Doesn't this affect the speed of the system's decision-making and safety?

In line 202 - 'its longer-term impact on urban sustainability' - How? Does it refer to fuel consumption?

In line 205 - 'sustainable smart cities' - Is there currently a city where your model could realistically be implemented? Considering laws, readiness, the number of AVs in traffic, and so on.

In Equation 7 - 'bsafe' - If it is intended as a single variable, it would be better to designate 'safe' as a subscript.

In line 220 - 'bsafe' - this is a single variable, how do I obtain it? How do I apply Equation 8, where it is used?

Equation 8 - where are the descriptions of the variables from this equation?

Table 2 - Did you perform processing in the mentioned simulator? The reference to source [38] is invalid. Your sources end with the number 17.

In line 232 - Could you describe this in more detail? Is it a publicly accessible simulator? What exactly did you simulate, and how did the further processing take place, or what was the output?

-          What does it mean that the simulator was modified?

-          Provide a valid reference to this simulator.

 

In lines 241-243 - Isn't safety more important in autonomous lane change than the environment? Because safety can be critical in terms of system application. I would consider the environment as a valuable addition. What is the assumption about the practical use of the study? Are you aware of any opportunities to realistically test your solution in a specific location? Have you conducted such a survey?

In line 244 - could you please specify to what extent you focus on safety in autonomous lane change, given that sustainability and the environment are frequently mentioned.

 

For Figure 1 - the title of the figure is sufficient if it is provided in the figure caption.

-          The legend overlaps with the graph (consider placing the legend outside the graph).

-          The overlap between 'local reward design' (solid black curve) and 'separate actor-critic network' (dashed red curve) is poorly readable; try changing the color or adding some detail to distinguish the curves.

 

In lines 268-269 - 'Here’s how you might rephrase the section:' - Is there missing text after this sentence?

In line 274 - 'The politeness coefficient (p)' - How is it determined? It is already used in Equation 8.

 

In line 284 - 'for real-world application in intelligent transportation systems' - Can you comment on whether there is a country in the world with such an intelligent city system where this could be applied? The development of autonomous vehicles is advancing, but we still do not have fully autonomous vehicles in traffic. What if the ratio between autonomous and human-driven vehicles HDVs is unfavourable to autonomous vehicles? In such a case, is your system still applicable? Is it safe given the unpredictability of human drivers?

 

In subchapter 6.3, it is necessary to explain how the traffic density levels 1 to 3 are defined. It would also be appropriate to add a brief definition of the differences between the compared models, especially in what distinguishes them.

 

For Figure 2:

There is no textual commentary on Figure 2.

The legend overlaps with the graph line. Since all figures are on one side, you can use a single legend or move them appropriately within the graph area.

What do the titles density 1 to 3 represent in the graph? Can you move this description to the main figure caption?

 

For Figure 3:

Only include the graph titles in the figure caption.

The graph for density 2 is outside the range of the graph; adjust the vertical axis range.

Add a subtle grid for the Y-axis in the graphs.

Can you explain why your approach achieves worse results for density 3? Is it referring to traffic density because it is not explicitly mentioned anywhere?

 

For Figure 4:

Include only the graph titles in the figure caption.

Add axis titles.

The text in the graphs is too small.

Why do the columns in the graphs not have the same color as in the previous figures (blue, green, yellow, red)?

Please check the graph all the way to the right for the last column 'MAACKTR.' According to the previous figure, even the MAACKTR approach (green line - average around 50) should have a similar average reward as your MA2C.*

 

In line 296 - Your approach has high variance at traffic density 3. Can you comment on it, as it seems to react to traffic density exactly the opposite of the mentioned MADQN approach?

 

In line 304 - 'other benchmarks in terms of evaluation rewards' - please explain what is meant by benchmark? Initially, I considered it an experiment where based on one dataset of inputs, you present results in graphs 2 to 4. Were there multiple datasets? This should be clarified at the beginning of the experiment and, more importantly, characterize the individual densities 1 to 3.

 

For Figure 5:

Why is vehicle 5 not mentioned in the figure caption?

 

In line 313 - 'involving three autonomous vehicles' - There are four autonomous (red) vehicles labeled 5-8 in the image.

In line 317 - 'Ahead in the left lane is Autonomous Vehicle 7 (AV 7)' - Are there system limitations? For example, would the system allow a lane change even if vehicle 7 was a human-driven vehicle HDV?

 

In line 344 - 'in fluctuating urban densities' - please explain what is meant by this.

 

In line 350 - 'can encourage the use of public transportation' - Does your study have applications in public transportation, or is its application possible in regular personal transportation?

 

In line 371 - 'Furthermore, its parameter-sharing structure fosters collaborative behaviors among agents,' - Is there a condition on the number of AVs in relation to HDVs? Is there a limit on the number of HDVs?

 

In line 376 - 'interact seamlessly with human-driven vehicles in dense traffic' - How does the system communicate with HDVs? Can it receive data and possibly send commands to HDVs?

 

 

Author Response

We thank the reviewer for his 60 pertinent comments. In the attached document, we provided answers to the reviewer's concerns.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Firstly, I want to thank the authors for their reply and secondly for taking into cosnideration my comments.  

Line 248: Please remove you comment "To address the reviewer's concern..."

Comments on the Quality of English Language

Please check the manuscript once again for grammar and syntax errors as well as for required or double spacing.

Please check Figure 5, as car number 5 in Lane Changed is out of boundaries. 

Author Response

Firstly, I want to thank the authors for their reply and secondly for taking into consideration my comments.

--> We thank the reviewer for acknowledging our efforts.  

Reviewer concerns: Line 248: Please remove your comment "To address the reviewer's concern..."

Response: In the revised manuscript, we have removed the mentioned phrase.

Reviewer concerns: Please check the manuscript once again for grammar and syntax errors as well as for required or double spacing.

Response: We double-checked the manuscript to fix grammatical issues or typos.

Reviewer concerns: Please check Figure 5, as car number 5 in Lane Changed is out of boundaries. 

Response: In the revised manuscript, we updated Figure 5 to satisfy the reviewer's concern regarding vehicle 5. Please check the updated Figure 5 in the revised manuscript.

Reviewer 2 Report

Comments and Suggestions for Authors

Ok.

Author Response

We thank the reviewer for his efforts in evaluating our manuscript after revision.