Research on Behavioral Decision at an Unsignalized Roundabout for Automatic Driving Based on Proximal Policy Optimization Algorithm

Abstract

Unsignalized roundabouts have a significant impact on traffic flow and vehicle safety. To address the challenge of autonomous vehicles passing through roundabouts with low penetration, improve their efficiency, and ensure safety and stability, we propose the proximal policy optimization (PPO) algorithm to enhance decision-making behavior. We develop an optimization-based behavioral choice model for autonomous vehicles that incorporates gap acceptance theory and deep reinforcement learning using the PPO algorithm. Additionally, we employ the CoordConv network to establish an aerial view for spatial perception information gathering. Furthermore, a dynamic multi-objective reward mechanism is introduced to maximize the PPO algorithm’s reward pool function while quantifying environmental factors. Through simulation experiments, we demonstrate that our optimized PPO algorithm significantly improves training efficiency by enhancing the reward value function by 2.85%, 7.17%, and 19.58% in scenarios with 20, 100, and 200 social vehicles, respectively, compared to the PPO+CCMR algorithm. The effectiveness of simulation training also increases by 11.1%, 13.8%, and 7.4%. Moreover, there is a reduction in crossing time by 2.37%, 2.62%, and 13.96%. Our optimized PPO algorithm enhances path selection during autonomous vehicle simulation training as they tend to drive in the inner ring over time; however, the influence of social vehicles on path selection diminishes as their quantity increases. The safety of autonomous vehicles remains largely unaffected by our optimized PPO algorithm.

1. Introduction

With the advances in information technology, networking, digitalization, and artificial intelligence (AI), research on autonomous vehicles has gained significant attention in the public domain. It is anticipated that autonomous vehicles will gradually replace conventional vehicles in the future. In addition to performing traditional vehicle functions, such as acceleration, deceleration, braking, and motion control, autonomous vehicles also exhibit AI-driven behaviors including environment perception, task planning, path planning, vehicle control mechanisms, and intelligent obstacle avoidance [1,2,3,4]. The exploration and implementation of autonomous vehicles have the potential to alleviate road traffic congestion issues while reducing fuel consumption and carbon dioxide emissions, and thereby enhances the overall road efficiency. Although there are numerous advantages associated with autonomous vehicles’ application across various scenarios involving people or objects encountered during travel journeys, roundabouts still pose a substantial challenge [5].

The decision-making strategy of the target vehicle in a social vehicle environment is investigated as a scenario [6,7]. Shadi Samizadeh et al. focused on lane waiting time to address the issue of optimal lane selection and studied the method for selecting the best lane in a three-lane roundabout without signaling [8]. Rasool Mohebifard et al. examined the impact of different penetration rates for autonomous vehicles on roundabout operation and improved traffic operations under low saturation and half-saturation flow [9]. Mehdi Naderi et al. proposed a comprehensive general method for controlling the laneless path of autonomous vehicles to tackle large non-lane roundabouts [10]. Zhang introduced a layered control scheme based on an intelligent network environment to enhance traffic efficiency at circular crossings without traffic lights [11].

In summary, the previous research on the roundabout scenario has primarily focused on assessing its overall efficiency and simulating autonomous vehicles in an ideal state, assuming a perfect intelligent system network [12]. However, there has been limited investigation into the decision-making problems faced by individual autonomous vehicle entities at a roundabout. This study addresses this gap by considering separate autonomous vehicles operating at a penetration rate of 0, meaning that when all other vehicles are non-autonomous, the safety and stability of these vehicles remain unaffected. The optimization of decision-making for autonomous vehicles at unsignalized roundabouts and selection of routes to enhance bicycle passage efficiency through such intersections are explored [13].

Autonomous driving decision-making encompasses both rule-based and data-driven strategies [14]. While rule-based approaches face challenges in addressing problems encountered by autonomous vehicles due to the complexity and dynamic nature of road conditions, data-driven strategies can optimize vehicle decision-making through the interaction between target action behaviors and the environment.

The proximal policy optimization algorithm is a data-driven decision algorithm, which plays an important role in decision-making research [15]. Qian et al. proposed a near-end strategy optimization algorithm based on dual observation and matching reward, in order to study air combat autonomous decision-making, effectively solving the problems of high information redundancy and slow convergence of traditional reinforcement-learning air combat autonomous decision-making algorithms [16]. Based on the traditional PPO algorithm, She et al. introduced the network structure of a long- and short-term memory network and used the PPO–LSTM algorithm to solve the problem of online trajectory planning in the ascending phase of high-speed aircraft [17]. Chen et al. proposed an intersection signal phase and timing optimization method based on the hybrid proximal policy optimization algorithm to solve the signal control problem [18]. Zhao et al. proposed a multi-agent dynamic spectrum resource allocation scheme based on the proximal policy optimization algorithm, which improved the channel transmission rate and the success rate of vehicle information transmission [19]. The PPO algorithm can be competent in multiple, continuous action, space environment decision-making regarding vehicle control, and its training stability, high robustness, and easy-to-adjust parameters are suitable for simulation research [20].

The present study proposes an innovative approach to address the issue of social vehicles dominating urban roads during the initial stage of autonomous vehicle implementation. Specifically, we introduce a coordinate convolution dynamic multi-objective reward mechanism into the proximal policy optimization (PPO) algorithm. Our research focuses on enhancing the interaction between autonomous vehicles and their surrounding environment at intersections by incorporating gap acceptance theory [21]. To achieve this, we integrate CoordConv network and coordinate convolutional network into our algorithm model to capture and store low-dimensional environmental information. By optimizing the dynamic multi-objective reward mechanism model, we aim to enhance safety, stability, and driving efficiency of autonomous vehicles. We select PPO algorithm for optimization purposes. In an unsignalized roundabout scenario, our proposed approach enables effective interaction between autonomous vehicles and other vehicles as well as traffic signs in the scene. We define a reward mechanism function for training purposes with the objective of establishing a decision model throughout various scenes and obtain an optimal decision strategy for autonomous vehicles.

2. PPO Algorithm Model

2.1. PPO Optimization Algorithm

Deep reinforcement learning (DRL), an artificial intelligence method that combines deep learning and reinforcement learning, exhibits a cognitive resemblance to human decision-making processes. Deep learning, as a multi-layer neural network, possesses exceptional perceptual capabilities in machine learning.

Reinforcement learning involves the interaction between an agent and its environment, where the agent learns from feedback information in the form of reward values to optimize its strategy for maximizing rewards. By integrating deep learning’s ability for end-to-end nonlinear fitting with reinforcement learning’s problem-solving capacity, deep reinforcement learning algorithms are formed. The actor–critic algorithm framework is widely employed in these algorithms due to its combination of value function estimation and strategy search methods, effectively addressing both high bias and high variance issues associated with traditional approaches. This framework enables the design of agents capable of generating strategies while simultaneously evaluating their effectiveness through value functions. The utilization of deep networks within deep reinforcement learning facilitates the accurate fitting of output value functions and strategies over iterative updates towards optimal performance (Figure 1).

PPO is a strategy optimization algorithm for reinforcement learning, which was proposed and published by OpenAl researchers and is widely used in various reinforcement learning tasks. The PPO algorithm utilizes strategy gradient-based optimization to maximize the expected cumulative return of the strategy function. In comparison with other strategy optimization algorithms, PPO ensures improved stability and convergence speed through constraint optimization, thereby controlling the update amplitude within each iteration.

The PPO algorithm is enhanced by incorporating the CoordConv network and a dynamic model multi-objective reward mechanism. The CoordConv network establishes an aerial perspective of the vehicle to acquire spatial perception information, capturing low-dimensional details such as road conditions and surrounding vehicles. In autonomous driving simulations, data processing plays a crucial role. By utilizing an aerial view, the data processing procedure can be simplified, facilitating easier analysis and enhancing simulation efficiency and accuracy. The global overview provided by the aerial view enables better comprehension of the surrounding environment and obstacles for self-driving systems, extracting features from lower-dimensional data. These feature data are stored, with the CoordConv network employed to extract relevant information from the aerial view’s features such as vehicle speed, distance to preceding vehicles, and their speeds—all serving as inputs for decision-making in the algorithm [22,23]. This approach effectively reduces dimensionality in state feature representation within the algorithm.

The optimal reward function model is established based on a multi-objective reward mechanism to optimize the decision-making strategy of autonomous vehicles in unsignalized roundabout scenarios, which differ from normal roads. The multi-objective reward mechanism encompasses lane offset, lateral speed, longitudinal speed, vehicle collision, brake and throttle rewards, workshop distance reward, and critical clearance reward to enhance the driving efficiency, safety, and stability of autonomous vehicles. Figure 2 illustrates the structural diagram of PPO algorithm.

The gap acceptance theory dictates that a circular vehicle can enter the roundabout when its headway exceeds a critical gap, while it must wait if the headway is less than this threshold. By calculating the maximum number of vehicles that can enter per unit of time, capacity calculation models based on gap acceptance theory accurately reflect the traffic flow characteristics of roundabouts.

The critical gap is a crucial research topic in autonomous vehicle navigation at roundabouts, as it involves determining the appropriate gap for the vehicle to enter. This refers to the minimum distance required for a branching vehicle to safely merge into the circular traffic flow. The critical gap serves as a decision criterion for vehicles approaching the roundabout entrance. Once an acceptable gap is identified, the vehicle accelerates to its desired speed and seamlessly merges with existing traffic while maintaining a safe following distance.

The time required for vehicles to enter the intersection is mainly divided into three sections: the first section is the driver’s reaction time t₁; the second section is the time t₂ from moving the foot to pressing the accelerator pedal to the start of the vehicle movement; and the third section is the time t₃ required for the vehicle to enter the circular lane after the vehicle starts to move. When the vehicle enters the intersection, it must maintain a certain distance from the vehicle in front, so it is necessary to calculate the on-board clearance t₄ with the vehicle in front, and t_c is the critical clearance. Therefore, the calculation of critical gap is shown in Equation (1):

$$t_c=t_1+t_2+t_3+t_4$$

(1)

The entry function is a function established according to the critical reception theory, which represents the number of vehicles allowed to pass by the time headway. The entry function is used to reflect the overall traffic efficiency of the unsignalized roundabout. The formula g(t) of the entry function is:

$$\mathrm{g}\left(\mathrm{t}\right)=\left\{\begin{array}{l}\frac{\mathrm{t}-\mathrm{t}0}{tf},t>t0\\ 0,t\le t0\end{array}\right\}$$

(2)

$${\mathrm{t}}_0{=\mathrm{t}}_c+\frac{t_f}{2}$$

(3)

Among them, t₀ is the minimum acceptable gap; t_f is the vehicle spacing; t_c is the critical gap.

In the process of building and training deep reinforcement learning models, there is a continuous optimization of the decision models. However, during the initial stage of training, the reward value remains close to zero, making it ineffective to repeatedly employ poor historical strategies that hinder training efficiency improvement. Therefore, optimizing the experience pool of the PPO algorithm becomes necessary. The PPO algorithm discards previously collected experiences after each policy update as they become irrelevant to the new policy. One advantage is that this ensures consistency between the policy and value function, enabling the accurate estimation of current policy performance. To store and reuse historical experiences requiring correction, PPO algorithms can utilize experience playback buffers.

In the research experiments conducted here, prioritized experience replay was employed which assigns priority based on data value or strength; specifically favoring completed experiment data and high-reward values. This approach allows for more frequent sampling of valuable or advantageous data, thereby enhancing learning efficiency while reducing collisions with autonomous vehicles.

2.2. State Model

The simulation model of the unsignalized roundabout is utilized to establish a research experiment that encompasses the autonomous vehicle’s traversal through the unsignalized roundabout under various traffic flow conditions. The driving process of an autonomous vehicle involves both state space and action space.

The state space, acquired through the CoordConv network, incorporates essential information such as vehicle route details, driving direction, relative position, speed, and surrounding vehicle data. This provided state space is defined as follows:

$$A=\left\{\tau ,l,\phi ,v,d\right\}$$

(4)

τ is the characteristic vector; l is the distance between the vehicle center point and the lane line; φ is the angle between the vehicle and the lane line; v is the current speed of the vehicle; and d is the distance between the vehicle and the vehicle in front. A is input as a vector to the policy and evaluation network.

The motion state space includes lateral control and longitudinal control. The lateral control is controlled by the steering wheel steer, whose value is [−1, 1], and the right steering wheel is positive. Longitudinal control is controlled by the vehicle throttle and brake break, set to acc, whose value is [−1, 1], and vehicle acceleration is positive. Among them, the giving action state is defined as:

$$B=\left\{steer,acc\right\}$$

(5)

2.3. The Reward and Punishment Function of an Unsignalized Roundabout

Studies have improved the reward mechanism model of the PPO algorithm in deep reinforcement learning to enhance training efficiency and reduce meaningless simulations [24,25,26]. The experiments are terminated when the vehicle’s stopping time reaches 2 min due to congestion or impassable roads, followed by starting the next training session. Additionally, encountering obstacles leads to a vehicle stop for a record duration of 30 seconds before initiating subsequent training sessions to increase repetition and improve overall efficiency [27,28,29].

The reward function design is optimized, and the gap acceptance theory is considered to enhance the success rate of autonomous vehicles entering the roundabout and improve their efficiency. Furthermore, variables and characteristics of autonomous vehicles in terms of safety, stability, and driving efficiency are taken into account (see

Table 1. Driving state evaluation index.

Category	Variable	Characteristics
Security	Lateral offset	Left–right offset, Offset range
	Lane change time	Average time
	Vehicle collision	Collision or not
	Vehicle spacing	Minimum spacing
Comfort level	Lateral acceleration	Maximum value, average value
	Longitudinal acceleration	Maximum value, average value
	Vehicle deviation angle	Offset angle and offset range
Driving efficiency	Speed	Maximum value, average value
	Vehicle acceleration	Maximum value, average value
	Transit time	Average value

).

The PPO algorithm of deep reinforcement learning optimizes the training model of the autonomous driving vehicle and measures the quality of vehicle decisions in relation to the environment, which plays an important role in vehicle reinforcement learning training. The reward function model considers the evaluation indicators and the following vehicle state and environment variables: lane offset (r_line, r_out), vehicle lateral speed (r_{speed_lat}), longitudinal speed (r_{speed_lon}, r_fast), vehicle collision (r_collision), brake (r_brake), throttle (r_throttle), workshop distance (S_spacing), and roundabout critical clearance (t_s). Set the c constant to −0.1. The reward function design requires the autonomous vehicle to drive at the roundabout, and it needs to run smoothly and steadily along the lane to ensure the safety of the vehicle.

The initial configuration of the training model for autonomous vehicles, such as the maximum speed and acceleration range, is relatively broad, leading to unreasonable decisions regarding vehicle speed, throttle control, and braking during early-stage simulation training. As a result, the autonomous vehicle is prone to collisions or lane deviations, ultimately terminating the experiment and resulting in low efficiency in early-stage training. To enhance overall training efficiency, we propose a dynamic reward function model that considers the varying weights of longitudinal speed, throttle control, brake control, and distance from other vehicles.

P_x is a dynamic coefficient, and its value changes according to the value of the reward function. P₁ is the dynamic coefficient of longitudinal velocity. When the reward function approaches 0, the weight of longitudinal velocity is minimum. As the value of the reward function increases, the weight of the reward function of the longitudinal speed is increased, and the initial value is 1 to ensure that the vehicle can start normally. P₁ formula is a piecewise function:

$$P_1=\left\{\begin{array}{cc}\frac{2}{\pi }\arctan \frac{R}{10}& ,ifR>0\\ 1& ,ifR\le 0\end{array}\right\}$$

(6)

The initial R value is 0, and R is the reward value obtained in the last experiment result of each experiment.

P₂ is a decreasing function, giving the dynamic coefficients of brake, throttle and workshop distance. When the reward function is 0, the brake weight is the largest. As the value of the reward function increases, the weight of the reward function decreases, and the final value approaches 1. The dynamic coefficient of the decreasing function enables important parameters in the early stages of training to play a role as much as possible, speeding up training efficiency and improving training success rate. The formula P₂ is as follows:

$$P_2={\left(\frac{1}{e}\right)}^{\frac{R}{10}}+1$$

(7)

After several rounds of experimental tests, the following reward mechanism is obtained:

$$\begin{array}{ll}R=& k_1\times r_{line}+k_2\times r_{out}+k_3\times r_{speed\_lat}+k_4\times r_{speed\_lon}\\ & +P_1\times k_5\times r_{fast}+k_6\times r_{collision}\\ & +P_2\times \left(k_7\times r_{brake}+k_8\times r_{throttle}+k_9\times S_{spacing}\right)+k_{10}\times t_s+c\end{array}$$

(8)

The research requires the vehicle to drive along the center line of the road; the deviation of the center lane is 0; the left side of the front is negative; the larger the offset, the smaller the value; the right side of the front is positive; the larger the offset, the larger the value. The function $e^{-{\left|x\right|}^2+1}$ is used as a reward function, and its characteristics are utilized, that is, the function image is axially symmetric with x = 0. When the value is 0, the function value is the largest, and the larger the value is; the smaller the value is, the smaller the value of the function is, so the function can punish vehicle lane offset. According to the importance of lane offset to the overall reward mechanism, r_line is the penalty value. Set its coefficient k₁ to 1.

$$r_{line}=e^{-{\left|x\right|}^2+1}$$

(9)

Lane offset has the maximum threshold l_thr; when l_thr > 2 m, the vehicle offset is too large and leaves the lane, which directly ends the training. When the offset threshold is set, the vehicle is prone to vehicle collision when the threshold is too large, which is not conducive to training the model. When the threshold is too small, there is no room for fault tolerance, the learning cost is high, and it is difficult to obtain an effective training model. r_out is the deviation penalty of the vehicle; when the vehicle deviates from the exit lane and exceeds the threshold l_thr, the reward value is −1, and in other cases it is 0. The coefficient k₂ is 5, which strongly penalizes lane departures.

$${\mathrm{r}}_{\mathrm{out}}=\left\{\begin{array}{l}-1,\mathrm{if}\mathrm{l}>{\mathrm{l}}_{\mathrm{thr}}\\ 0,otherwise\end{array}\right\}$$

(10)

Preventing the vehicle from driving unstably due to excessive lateral speed during the test. The penalty is set to r_{speed_lat} based on vehicle corner steer and vehicle speed v_{speed_lon}, and the coefficient k₃ is 0.2.

$$r_{(speed\_lat)}=-\left|steer\right|\times v_{speed\_lon}^2$$

(11)

The longitudinal speed v_{speed_lon} reward is set to r_{speed_lon}, and the higher the longitudinal speed, the greater the reward value. The coefficient k₄ is 1, as setting the reward too high will cause the vehicle to crash ahead, and setting the reward value too low will make the vehicle unable to move forward.

$$r_{speed\_lon}=v_{speed\_lon}$$

(12)

The vehicle speed should be given a speed threshold according to the road speed limit requirements, and punishment should be given if the threshold is exceeded. The penalty value is r_fast. This time, the threshold is set as v_thr = 8 m/s. When the vehicle speed is greater than the threshold, r_fast = −1 is given; in other cases, it is 0, and the coefficient k₅ is 10.

$${\mathrm{r}}_{\mathrm{fast}}=\left\{\begin{array}{l}-1,\mathrm{if}\mathrm{v}>{\mathrm{v}}_{\mathrm{thr}}\\ 0,otherwise\end{array}\right\}$$

(13)

When the self-driving vehicle is driving, its own speed is too fast, the vehicle brakes slowly, other vehicles suddenly change lanes, and other factors will lead to vehicle collision. During the training process, if there is a vehicle collision, immediately terminate the current test training and start the next simulation. r_collision is the penalty for a vehicle collision, with a reward value of −1 if the vehicle collides and 0 in other cases. The collision directly affects the safety of the autonomous vehicle. The experiment requires that the vehicle does not collide, and the set coefficient k₆ has a large weight. The coefficient k₆ of r_collision is 200.

$${\mathrm{r}}_{\mathrm{collision}}=\left\{\begin{array}{l}-1,\mathrm{if}{\mathrm{collision}}_{\mathrm{r}}\\ 0,otherwise\end{array}\right\}$$

(14)

When the current party encounters an obstacle or the vehicle speed is greater than the threshold, the autonomous vehicle will brake and slow down. In this paper, the simulated environment perceives the obstacle distance ahead as d = 15 m. The braking reward value is r_brake. When there is an obstacle in front or the speed of the vehicle is greater than the threshold value, the braking behavior of the vehicle will be rewarded, and the setting coefficient k₇ is 5.

$${\mathrm{r}}_{\mathrm{brake}}=\left\{\begin{array}{l}\mathrm{brake},\mathrm{if}\mathrm{d}<{\mathrm{d}}_{\mathrm{thr}}\mathrm{or}\mathrm{v}>{\mathrm{v}}_{\mathrm{thr}}\\ 0,otherwise\end{array}\right\}$$

(15)

In this scenario, the maximum speed is 8 m/s or 30 km/h. When the vehicle speed is lower than 8 m/s, the vehicle acceleration is rewarded. When the vehicle speed is higher than 8 m/s, the vehicle acceleration penalty is given; the penalty for speeding up when the vehicle deviates from the lane. The throttle reward value is r_throttle and the set factor k₈ is 5.

$${\mathrm{r}}_{\mathrm{throttle}}=\left\{\begin{array}{l}-\mathrm{throttle},\mathrm{if}\mathrm{v}>{\mathrm{v}}_{\mathrm{thr}}\mathrm{or}\\ \mathrm{acceleration}\mathrm{after}\mathrm{crossing}\mathrm{the}\mathrm{line}\\ 0,otherwise\end{array}\right\}$$

(16)

In this scenario, the maximum road speed is 30 km/h, and the minimum vehicle spacing is S_thr = 3 m. The research requires the self-driving vehicle to maintain the minimum vehicle spacing from the front vehicle as far as possible, and uses the exponential function as the reward function, as shown in formula 18. When the value is 0, the function value is the largest, and the larger the value is; the smaller the value is, the function value also becomes smaller. This function can reward the self-driving vehicle for maintaining a relatively close distance from the front vehicle. Vehicle spacing is the penalty value and the coefficient k₉ is set to 5.

$$S_{spacing}=e^{-{\left|x\right|}^2+1}$$

(17)

According to the gap acceptance theory, the critical gap t_c can be obtained. When there is a critical gap at the roundabout, when the entry function g(t) is ≥1, the autonomous vehicle is allowed to enter the roundabout. When the time difference between the two vehicles in the roundabout is less than the critical gap, that is, when the entry function g(t) is <1, the autonomous vehicle brakes. The critical gap ts is the penalty value, and the coefficient k₁₀ is set to 30. Gap acceptance is the most critical evaluation index for roundabout optimization. If the setting is too large, the influence of other factors on vehicles entering the roundabout is not obvious. If the setting is too small, the influence of gap acceptance theory on vehicles entering the roundabout cannot be reflected.

$${\mathrm{t}}_{\mathrm{s}}=\left\{\begin{array}{l}\mathrm{brake},\mathrm{if}{\mathrm{t}}_{\mathrm{s}}<{\mathrm{t}}_{\mathrm{c}}\\ 0,otherwise\end{array}\right\}$$

(18)

3. Results and Discussion

3.1. Experimental Environment

In this study, the proximal policy optimization (PPO) algorithm is employed to optimize the decision-making behavior of autonomous vehicles at unsignalized roundabouts. The simulation environment consists of a 64-bit Ubuntu 20.04 operating system, Carla 0.9.6 simulation platform, and Roadrunner R2020b software for creating autonomous driving scenarios.

The simulated training scenarios encompass urban roads, including unsignalized roundabouts and signalized intersections. Specifically, the decision scenario involves a straight road segment of 100 meters in both front and rear directions, as well as an unsignalized roundabout with a diameter of 20 meters. The simulated scene includes dual lanes for the entrance lane, the inner lane of the roundabout, and the exit lane. At the entrance of the unsignalized roundabout, appropriate measures such as speed reduction signs, speed limits, and traffic signs indicating a roundabout should be implemented; moreover, sensors on autonomous vehicles should respond accordingly upon detecting these relevant traffic signs [30]. The designed scene aims to evaluate various vehicle action decisions such as turning movements, avoidance maneuvers, and acceleration/deceleration behaviors during following or overtaking other vehicles.

The experimental groups utilized the proximal policy optimization (PPO) algorithm, the coordinate convolution multi-reward function under PPO (PPO+CCMR), and an optimized version of the PPO algorithm. It is important to note that the deep reinforcement learning PPO algorithm requires a relatively long time for establishing the automatic driving model. Therefore, in order to ensure a fair comparison, all other factors were kept constant throughout the experiment. Furthermore, it is crucial to optimize the training efficiency of deep reinforcement learning; thus, each of the three experimental groups were given equal training time for accurate comparison.

The independent variables in our experiments included traffic flow at unsignalized roundabouts with four different conditions as follows: (1) a scenario with 20 social vehicles where there are minimal other vehicles present; (2) a scenario with 100 social vehicles causing interference at unsignalized roundabouts; (3) a scenario with 200 social vehicles resulting in increased interference at unsignalized roundabouts without signals; and finally, (4) a scenario with heavy traffic congestion at unsignalized roundabouts due to the presence of 300 social vehicles.

Figure 3 illustrates the program execution process.

The stability evaluation criteria for autonomous vehicles encompass attitude stability, lateral and longitudinal stability, driving safety, and obstacle avoidance capability. In the simulation results, the one-step reward value function is employed as the evaluation metric to compare the one-step reward function curves in identical environments. A higher value indicates better stability.

During the simulation experiment, the driving efficiency of autonomous vehicles is evaluated based on their travel time. The simulation scenarios include linear roads and unsignalized roundabouts where travel time is recorded to calculate time reduction efficiency that directly reflects traffic efficiency.

The safety evaluation of autonomous vehicles encompasses vehicle control accuracy, sensor performance, road adaptability, emergency handling capability, and compliance with traffic regulations. In the simulation experiment, apart from the decision-making algorithm of the autonomous vehicle, all other vehicles exhibit identical performance characteristics. Additionally, the perception system comprising cameras and lidar is configured uniformly. The simulation environment is trained using urban roads and unsignalized roundabouts. The safety gap in autonomous vehicles primarily manifests in the control accuracy of their decision-making algorithms and their ability to handle emergency situations effectively. Success rate serves as a benchmark for comparing algorithms in simulation experiments.

3.2. Simulation and Analysis

The simulation scenario includes a total of 20 social vehicles, and the automatic driving vehicle obtained simulation data after undergoing 600,000 steps of training in an unsignalized roundabout.

The results are presented in Figure 4, where it can be observed that the reward value function of the optimized PPO algorithm stabilizes around 500,000 steps, while the reward value function of the PPO+CCMR algorithm stabilizes at approximately 540,000 steps. Notably, compared to the PPO+CCMR algorithm, there is a significant increase in reward value of approximately 2.8%, indicating an improvement in simulation training efficiency by about 11.1% for the optimized PPO algorithm.

As depicted in Figure 5, the optimized PPO algorithm for vehicle average speed optimization exhibits a gradual increase with the simulation step size, surpassing both the PPO+CCMR curve and the standard PPO algorithm. The initial average training speed of the optimized PPO algorithm is relatively low but improves progressively during simulation experiment training, demonstrating smoother and more stable performance compared to the PPO+CCMR algorithm.

The simulation scenario includes 100 social vehicles, and the automatic driving vehicle obtained simulation data after 600,000 training steps in an unsignalized roundabout.

As depicted in Figure 6, the reward value function of the optimized PPO algorithm tends to stabilize at around 500,000 steps, while the reward value function of the PPO+CCMR algorithm stabilizes at approximately 560,000 steps. Compared to the PPO+CCMR algorithm, there is a significant increase of 7.17% in the reward value achieved by utilizing only the optimized PPO algorithm. Moreover, this improvement results in a remarkable enhancement of training efficiency by approximately 13.8%.

The average vehicle speed of the optimized PPO algorithm and PPO+CCMR curve is demonstrated as being in close proximity, yet significantly higher than that of the conventional PPO algorithm, as depicted in Figure 7. Moreover, it is observed that the average speed growth curve of the optimized PPO algorithm exhibits enhanced smoothness and stability compared to that of the PPO+CCMR algorithm.

The simulation scenario involves 200 social vehicles, and the automatic driving vehicle obtained simulation data after undergoing 600,000 steps of training in an unsignalized roundabout.

As depicted in Figure 8, within this environment of 200 social vehicles, autonomous vehicles are influenced by their own driving status as well as surrounding vehicles. The average reward value achieved by autonomous vehicles utilizing the optimized PPO algorithm surpasses that of the PPO+CCMR algorithm in an unsignalized roundabout. Moreover, decision-making performed by the optimized PPO algorithm is more rational and yields higher rewards. The reward value function of the optimized PPO algorithm tends to stabilize at approximately 540,000 steps, while for the PPO+CCMR algorithm it stabilizes around 560,000 steps. Compared to the PPO+CCMR algorithm, there is a significant increase of 19.58% in the reward value when employing the optimized PPO algorithm; additionally, simulation training efficiency improves by approximately 7.4%.

The initial average speed of the optimized PPO algorithm is observed to be lower than that of the PPO+CCMR algorithm, as depicted in Figure 9. However, with the progressive increment of the speed weight during training, a steady rise in average speed is witnessed, leading to continuous enhancement in autonomous vehicles’ efficiency while navigating unsignalized roundabouts. Upon reaching 600,000 steps, it becomes evident that the average speed surpasses that achieved by training under the PPO+CCMR algorithm. Consequently, self-driving vehicles exhibit a more significant increase in their average speeds.

To compare the passing efficiency of autonomous vehicles, our focus lies on two specific periods: when the total step length of the training model reaches 300,000 and 600,000. We then proceed to analyze and compare the average passing time in various simulation scenarios, as presented in

Table 2. Simulation average pass time.

Training Step Size		300,000			600,000
Scenes		20	100	200	20	100	200
Average transit time (s)	PPO+CCMR algorithm	34.42	34.86	41.81	31.61	31.97	39.61
	Optimized PPO algorithm	33.62	33.95	36.51	30.86	31.13	34. 08
Reduce efficiency through time		2.32%	2.61%	12.67%	2.37%	2.62%	13.96%

.

The average passing time does not take into account the PPO algorithm, which exhibits poor training performance in unsignalized roundabouts due to limited simulation experiments and lack of reference for the average passing time. As shown in Table 2, with 300,000 training steps, the optimized PPO algorithm reduces the passing time by 2.32%, 2.61%, and 12.67% compared to the PPO+CCMR algorithm when social vehicles are increased on the scene. With 600,000 training steps, the optimized PPO algorithm further reduces the passing time by 2.37%, 2.62%, and 13.96% compared to the PPO+CCMR algorithm under similar conditions of increasing social vehicles on the scene.

The enhanced efficiency of traffic achieved by optimizing the PPO algorithm is positively correlated with both simulation step length and vehicle density within a given scenario at an unsignalized roundabout where autonomous vehicles navigate through inner loop lanes or outer loop lanes with varying route selection strategies as indicated in

Table 3. The inner and outer loop traffic schedule.

Training Step Size		300,000			600,000
Scenes		20	100	200	20	100	200
Traffic ratio (Inner: outer)		5:5	5:5	6:4	9:1	9:1	8:2
Average time (s)	Inner	31.30	31.55	35.15	30.48	30.74	33.52
	outer	35.94	36.35	38.55	34.26	34.62	36.31

.

As presented in Table 3, when employing an optimized PPO algorithm for the simulation training of autonomous vehicles, the average passing time of inner ring vehicles is observed to be lower than that of outer ring vehicles. Moreover, as the training duration increases, autonomous vehicles tend to prefer traveling within the inner ring. The path selection of autonomous driving vehicles in unsignalized roundabout scenarios is influenced by the number of social vehicles present, with this influence becoming less pronounced as the number of social vehicles increases.

To assess the safety performance of autonomous vehicles, particular attention is given to two specific periods during which the total step size reaches 300,000 and 600,000 in the training model. Around these periods, data from 20 experimental groups are collected to determine whether or not successful simulations were achieved.

The primary shortcomings observed in the simulation experiments were vehicle collisions and off-lane misalignment, with 20 scenarios primarily involving off-lane misalignment and 200 scenarios mainly focusing on vehicle collisions.

Table 4. Simulation success rate table.

Training the Total Step Size		300,000			600,000
Scenes		20	100	200	20	100	200
PPO Algorithm	Successes	2	2	1	4	4	3
	Failures	18	18	19	16	16	17
	Success rate	10%	10%	5%	20%	20%	15%
PPO+CCMR Algorithm	Successes	9	9	7	16	15	13
	Failures	11	11	13	4	5	7
	Success rate	45%	45%	35%	80%	75%	65%
Optimized PPO Algorithm	Successes	9	9	8	16	15	14
	Failures	11	11	12	4	5	6
	Success rate	45%	45%	40%	80%	75%	70%

presents the success rate of these simulation experiments. For both training the PPO+CCMR algorithm and optimizing the PPO algorithm over 300,000 steps, as well as for training over 600,000 steps, the success rates ranged between 35% and 45%, and between 65% and 80%, respectively. However, when synchronizing the PPO+CCMR algorithm with the optimized PPO algorithm for an extended period of time, there was a decrease in the success rate of simulation experiments as scene vehicles increased; this decrease was significantly higher than that observed when only using the PPO algorithm. In scenarios involving identical vehicles, increasing the length of training steps led to an improvement in simulation experiment success rates while enhancing autonomous vehicle safety.

The simulation experiment demonstrates that the optimized PPO algorithm effectively enhances the efficiency of simulation training. However, in the case of 20 and 100 vehicles, the optimization algorithm exhibits marginal improvements in terms of driving efficiency, stability, and safety for autonomous vehicles at unsignalized roundabouts. Conversely, when confronted with a scenario involving 200 vehicles, the optimization algorithm significantly enhances both stability and driving efficiency for autonomous vehicles entering unsignalized roundabouts, albeit with limited safety enhancements.

4. Conclusions

This paper investigates the decision-making process of autonomous driving in unsignalized roundabouts and proposes a proximal policy optimization (PPO) learning method based on the PPO algorithm, which is trained using the Carla simulation environment. The proposed algorithmic framework utilizes an aerial view and an automatic driving perception system to acquire road environment information, while the CoordConv network extracts and integrates feature data into the decision-making network. A dynamic multi-objective reward mechanism and dynamic behavioral decision-making are employed to adjust the weight of autonomous vehicle states during training, enabling dynamic adjustment of important indices at different stages.

The results demonstrate that the optimized algorithm trained with this approach exhibits no significant improvement compared to the PPO+CCMR algorithm; however, it outperforms the PPO algorithm when there are fewer vehicles present. As other vehicles increase on the road, both the improved optimization algorithm’s dynamic model multi-objective reward mechanism and unsignalized roundabout decision-making optimization module exhibit noticeable effects, leading to better decisions regarding path selection and effectively enhancing training efficiency as well as vehicle stability and safety. In scenarios involving 20, 100, and 200 social vehicles, respectively, compared with the PPO+CCMR algorithm, our optimized PPO algorithm shows improvements in reward value function by 2.85%, 7.17%, and 19.58%; simulation training efficiency improves by 11.1%, 13.8%, and 7.4%; traversal time is reduced by 2.37%, 2.62%, and 13.96%.

While our proposed algorithms have achieved certain results in unsignalized roundabout experiments, further research can be combined with intelligent transportation systems considering actual road conditions along with increasingly complex road environments.