Generalized Single-Vehicle-Based Graph Reinforcement Learning for Decision-Making in Autonomous Driving

Abstract

In the autonomous driving process, the decision-making system is mainly used to provide macro-control instructions based on the information captured by the sensing system. Learning-based algorithms have apparent advantages in information processing and understanding for an increasingly complex driving environment. To incorporate the interactive information between agents in the environment into the decision-making process, this paper proposes a generalized single-vehicle-based graph neural network reinforcement learning algorithm (SGRL algorithm). The SGRL algorithm introduces graph convolution into the traditional deep neural network (DQN) algorithm, adopts the training method for a single agent, designs a more explicit incentive reward function, and significantly improves the dimension of the action space. The SGRL algorithm is compared with the traditional DQN algorithm (NGRL) and the multi-agent training algorithm (MGRL) in the highway ramp scenario. Results show that the SGRL algorithm has outstanding advantages in network convergence, decision-making effect, and training efficiency.

1. Introduction

In autonomous driving, the decision-making system is mainly used to produce advanced actions of vehicles, such as lane changing, acceleration, braking, and so on. Tactical decision-making for autonomous driving is challenging due to the diversity of environments, the uncertainty in the sensor information, and the complex interaction with other road users [1,2]. Traditional vehicle trajectory modeling and tracking control methods, such as genetic algorithms, neural networks, and their optimizations, have played a positive role in the research of decision-making [3].

The operational space of an autonomous vehicle (AV) can be diverse and vary significantly. Due to this, formulating a rule-based decision-maker for selecting driving maneuvers may not be ideal [4]. With the development of deep learning, the domain of reinforcement learning (RL) has become a robust learning framework now capable of learning complex policies in high-dimensional environments [5]. Therefore, using reinforcement learning to solve decision-making problems has gradually become the mainstream of research. Carl-Johan Hoel et al. introduce a method based on deep reinforcement learning for automatically generating a general-purpose decision-making function [6]. They trained an RL agent to handle a truck–trailer combination’s speed and lane change decisions in a simulated environment. Hongbo Gao et al. solved sequential decision optimization problems based on the inverse reinforcement learning algorithm, and the proposed method was verified in terms of efficiency [7]. Some algorithms for planning and decision-making are based on analyzing driver behavior [8,9].

The realization of the decision-making is based on the understanding and analysis of high-dimensional environmental information. However, the traditional reinforcement learning algorithm only has a good capacity for decision-making based on low-dimension features [10]. It will have the problem of insufficient understanding in the face of more complex scenario information. The deep neural network (DNN) has a strong ability to learn representations and for the generalization of matching patterns from high-dimensional data. Therefore, deep reinforcement learning (DRL) algorithms are effective in tasks requiring feature representation and policy learning, e.g., autonomous driving decision-making [11]. Using the functional approximation ability of a deep neural network (DNN), an intelligent controller integrating artificial intelligence technologies such as deep learning (DL) and reinforcement learning (RL) is designed to maintain and avoid obstacles in lanes [12,13,14], decision-making [15,16,17,18,19,20,21], longitudinal control [22], merger maneuvers [23], human-like driving strategies [24,25,26], and other large-scale autonomous driving control tasks. Yingjun Ye et al. put forward a framework for decision-making training and learning. It consists of a deep reinforcement learning (DRL) training program and a high-fidelity virtual simulation environment [27]. Compared with the DQN algorithm based on a value function, the deep deterministic policy gradient (DDPG) algorithm based on an action policy can solve the continuity problem of the action space. Haifei Zhang et al. used the DDPG algorithm to solve the control problem of automatic driving based on a reasonable reward function, deep convolution network, and exploration policy [28].

Interaction between vehicles in common public transport scenarios is necessary and pervasive. However, there are relatively few studies on how autonomous vehicles interact in public environments with reinforcement learning. Realizing coordination between vehicles in a shared environment is challenging due to the unique feature of vehicular mobility, which makes it infeasible to apply the existing reinforcement learning methods directly. Chao Yu et al. proposed using a dynamic coordination graph to model the continuously changing topology during vehicles’ interactions and developed two basic learning approaches to coordinate the driving maneuvers for a group of vehicles [29].

Cooperative vehicle–infrastructure systems and automatic driving technology are developing rapidly [30]. The graph neural network (GNN) has gained increasing popularity in various domains, including social network analysis [31,32]. GNN can extract the relational data representations and generate useful node embeddings on the node features and the features from neighboring nodes. The interactions between the ego vehicle and other surrounding vehicles can also be represented by the dynamic potential field (DPF) and embedded in the gap acceptance model to ensure safety and personalization during driving [33]. Jiqian Dong et al. proposed a novel deep reinforcement learning (DRL)-based approach combining the graphic convolution neural network (GNN) and deep Q network (DQN), namely the graphic convolution Q network, as the information fusion module and decision processor [34]. The proposed model can aggregate the information obtained by collaborative perception and output collaborative lane change decisions for multiple vehicles. Even in the case of highly dynamic and partially observed mixed traffic, the intention can be satisfied.

However, the above multi-agent training-based GNN reinforcement learning (MGRL) has the following problems in the actual verification process (the scenario of highway ramp exiting):

1.

Multi-agent training simultaneously increases the computational network complexity, resulting in a higher overall training time cost. Therefore, each parameter modification requires a more prolonged verification time, which is not conducive to the development and adjustment of the algorithm.

2.

The reward and punishment offset each other in multi-agent overall training, resulting in poor network convergence in the training process. Due to the mutual influence of multiple agents’ reward values, it cannot accurately evaluate the current state, resulting in the very unstable fluctuation of the loss curve.

3.

Through the test of the final training model, the final task success rate of the GCQ algorithm is maintained at around $50\%$, which cannot meet the basic driving needs of vehicles.

4.

The GCQ decision-making model stays in the lateral lane change and cannot control the longitudinal behavior of the vehicle at the same time. This is an incomplete driving control model, which leads to a low success rate, serious collisions, and low traffic efficiency.

Given the above problems, this paper proposes an improved single-agent GNN-based RL algorithm (SGRL algorithm). This paper has the following contributions:

1.

Based on retaining interactive feature extraction (GNN), the trained object is transferred from multi-agent to single-agent. Through the internal processing of the network model, the training results of single-agent can be applied to the application scenarios of multi-agent. This training method can significantly reduce the time cost of model training and eliminate the interaction of rewards and punishments between agents to improve the training effect.

2.

An inductive reward function is designed to improve the convergence speed of the model. The reward function incorporates driving intention, collision, lane change frequency, and vehicle speed into the calculation. The trained model simultaneously performs well in terms of task success rate, safety, driving stability, and traffic efficiency.

3.

The dimension enhancement of the action space is realized by changing the network structure. Adding vehicle longitudinal velocity control gives the model a more robust control ability for task success rate, safety, and traffic efficiency.

4.

In the training process, the real-time screening of stored data can improve the training speed. The random generation mechanism of vehicles’ number, type, and position in the training scenario can improve the model’s generalization ability and avoid overfitting.

The paper is organized as follows. Section 2 introduces the proposed SGRL algorithm in detail. Section 3 shows the model training and testing of the SGRL algorithm and comparison algorithms (MGRL and NGRL). Section 4 shows the results of training and testing and analyzes the comparison. Finally, Section 5 derives the conclusions and proposes future improvement directions.

2. Method

The proposed SGRL methods are used to model the Markov Decision Process (MDP). The agents can explore the environment by observing states, taking actions, and receiving rewards, as shown in Figure 1.

The implementation core of the SGRL method is based on the preprocessing of input data, the structure of the deep neural network, and the setting of the output end.

2.1. Network Input

In each time step in the training process, the vehicles in the scenario can be divided into human-driven vehicles (HVs) and autonomous vehicles (AVs). At each step t, the input of the training model can be divided into two parts: feature matrix $X_t$ and correlation matrix $C_t$. The state $s_t=(X_t,C_t)$. Concerning the feature matrix $X_t$, the necessary basic information based on highway scenarios is included. The total number of human-driven vehicles (HVs) and autonomous vehicles (AVs) is set to $N=N_{HV}+N_{AV}$. Then, the specification of the matrix is $N\times 8$. The eight characteristic parameters of each vehicle describe the speed, lateral position, longitudinal position, and destination information, denoted as $x_i=(v_i,p_i,l_i,I_i)$, To serialize multiple data at the same scale, the parameters are set as follows:

• $v_i=\frac{v_{i-current}}{v_{max}}$ is the relative speed, where $v_{max}=max(v_{max-vehicle},v_{max-highway})$ is the maximum speed for the current vehicle.
• $p_i=\frac{x_i}{l_{highway}}$ is the relative longitudinal position normalized by the entire length of the highway. (The algorithm belongs to local planning. Its application scenarios are specific ramps and expressway sections within short distances, so the normalization of location information is more favorable for network computing.)
• $l{}_i$ is the lateral lane position for the vehicle i. The position of the vehicle is represented by three-bit binary coding. For example, vehicles in the rightmost lane are denoted as $[1,0,0]$, the middle lane $[0,1,0]$ and the leftmost lane $[0,0,1]$.
• $I_i$ is the destination intention feature for the vehicle i. The data type is similar to $l_i$. The three destinations are expressed as $[1,0,0]$ (merge out from the first ramp), $[0,1,0]$ (merge out from the second ramp) and $[0,0,1]$ (go straight along the highway).

Based on GNN, we can introduce a correlation matrix $C_t$ to calculate the relationship between vehicle nodes. Considering that the sensors installed on autonomous vehicles have fixed sensing ranges, only vehicles within the sensor sensing range are recorded in the correlation matrix. $C_t$ is a square matrix with the specification $N\times N$. The row data i of the matrix represent the relationship between vehicle i and other vehicles through binary data. The value $C_{ij}$ of the vehicle j within the set distance of the target vehicle i will be set to 1, as shown in Figure 2.

Considering that the number of vehicles in the scene is dynamic, this situation has been considered when setting the feature matrix and the correlation matrix. The vehicles in the environment are divided into HVs and AVs, located in the feature matrix’s upper and lower parts. When the number of vehicles is less than N, the positions in the matrix are occupied by 0.

To ensure the authenticity of the simulation process, all vehicles appear and disappear successively in the scenario, so the feature information and correlation information obtained at the step t cannot reach the predetermined matrix size. To ensure the standard calculation of GNN, matrix data filling is needed. Matrix segmentation prevents data confusion and error correspondence caused by random packing, as shown in Figure 3.

2.2. Network Structure

In the whole network structure, the function of graph convolution is to obtain the interaction information between vehicles. The function of the full connection layer is to parse the matrix information. Each newly obtained matrix in the network needs to be input into the full connection layer for recording and information analysis. In addition, the number of nodes in each layer also plays a decisive role in the model’s performance. The optimal number of nodes is selected through comparative experiments.

Data records for comparative tests are shown in

Table 1. Training effect for different numbers of nodes.

N of Nodes	Training Time for 1000 Episodes	Convergence Effect
32	1.5179 h	Poor (large fluctuation)
64	2.6438 h	Acceptable (occasional large fluctuations)
128	3.4756 h	Good (small fluctuation)
256	6.0987 h	Good (small fluctuation)
512	10.9542 h	Good (small fluctuation)

.

At each step t, the feature matrix $X{}_t$ is first fed to a Fully Connected Network (FCN) encoder $\psi$ and then the matrix $H_t\in {}^{N\times 128}$ is obtained (Equation (1)). The original eight characteristic values will be recoded to 128 values. FCN will be executed twice consecutively.

$$H_t=\psi \left(X_t\right)\in {}^{N\times 128}$$

(1)

Next is the calculation of graph convolution. The input of the convolution function is the newly obtained feature matrix $H_t$ and correlation matrix $C_t$. The function output matrix $K_t$ has the same specification as the original matrix $H_t$ (Equation (2)).

$$K_t=\chi (H_t,C_t)\in {\mathbb{R}}^{N\times 128}$$

(2)

The original feature information and the new information obtained by graph convolution are fused by matrix stitching online as the total input of the subsequent neural network $\eta$.

The feature data density changes at each stage are as follows:

• Fully Connected Network (FCN): $Dense\left(8\right)\to Dense\left(128\right)\to Dense\left(128\right)$;
• Graph Neural Network (GNN): $Dense\left(128\right)\to Dense\left(128\right)\to Dense\left(128\right)$;
• Q Network: $Dense\left(256\right)\to Dense\left(128\right)\to Dense\left(128\right)$;
• Output: $Dense\left(128\right)\to Dense\left(33\right)$.

The overall structure of the SGRL algorithm is shown in Figure 4.

2.3. Network Output

Since the vehicle longitudinal control model built into the simulation environment is still rule-based, it cannot incorporate the interaction between vehicles into the control model. The SGRL algorithm fuses longitudinal and lateral control into the same model by increasing the output dimension, which significantly simplifies the complexity of the control process and solves the coupling problem of two directions.

The SGRL algorithm is trained based on DQN, so the output action space can only be discrete. For AVs, the longitudinal control is mainly reflected in the acceleration, and the lateral control is primarily reflected in the lane change direction. The improvement of the output action resolution is conducive to improving the control sensitivity, but it also increases the computational complexity and reduces the control frequency. In the algorithm of this paper, a compromise solution is selected. The longitudinal control is set to 11 discrete values in the interval $[-5,5]$, and the lateral control is set to three actions: keeping, turning left and turning right. Thus, the output matrix of the model is set as $A_t\in {\mathbb{R}}^{N\times 33}$, as shown in Figure 5.

2.4. Reward Function

The setting of the reward function needs clear guidance and a strong correlation with training objectives. In the SGRL algorithm, the task success rate, security, and traffic efficiency should be considered simultaneously. The corresponding reward values are set as intention reward, crash reward, and speed reward.

2.4.1. Intention Reward

To guide autonomous vehicles to complete driving tasks from the corresponding ramps out of the highway, the SGRL algorithm constructs reward gradients for different lanes, as shown in Figure 6. Merge_0 and merge_1, respectively, refer to the autonomous vehicles that need to drive away from ramp_0 and ramp_1 according to the task requirements. In the simulation experiment, the driving route of all autonomous vehicles is entirely determined by the reinforcement learning controller. Therefore, vehicles belonging to the merge_0 category will not necessarily exit from ramp_0; that is, they cannot complete the driving task.

To ensure practical guidance for all locations, the reward value $R_{I-t}$ for the fixed area is set to a constant value. To ensure the interaction between various rewards and to pass it to the training process, it is necessary to pay attention to the consistency of the numerical scale when setting $R_{I-t}$. Moreover, the $R_{I-t}$ needs to distinguish between positive and negative values, which is more conducive to the training. In addition, if the autonomous vehicle completes the task, it can obtain greater $R_{I-t}$, and if the task fails, it will also obtain a greater negative $R_{I-t}$.

2.4.2. Crash Reward

The safety of autonomous driving is the basis of all other characteristics. The simulation platform SUMO can detect collisions at step t and output the number $N_{collision-t}$ of vehicles involved in collisions. The collision reward is calculated according to $N_{collision-t}$ (Equation (3)).

$$R_{C-t}=-N_{collision-t}/2$$

(3)

2.4.3. Speed Reward

To control the consistency of the reward scale, the speed reward value $R_{S-t}$ needs to be normalized. $v_{max}=max(v_{max-vehicle},v_{max-highway})$ is the max speed for the AV. To ensure the positive and negative values of $R_{S-t}$, the calculation is shown as Equation (4).

$$R_{S-t}=\frac{v_{i-t}}{v_{max}}-0.3$$

(4)

2.4.4. Total Reward

The general method to calculate the total reward is a weighted summation of each component. Direct addition will lead to mutual coverage of rewards, resulting in poor training effects. The SGRL algorithm uses a new aggregation method, as shown in Equation (5).

$$R_t=\left\{\begin{array}{c}{\omega }_I\times R_{I-t}\times R_{S-t}+{\omega }_C\times R_{C-t}\\ {\omega }_I\times R_{I-t}+{\omega }_C\times R_{C-t}\end{array}\right.\begin{array}{c}(R_{I-t}>0)\\ (R_{I-t}\le 0)\end{array}$$

(5)

This calculation method takes the task success rate and safety as the primary considerations and considers traffic efficiency simultaneously. The problem of vehicle parking for intention rewards can be completely avoided. Moreover, the weight relationship between rewards can be adjusted by parameters ${\omega }_I$ and ${\omega }_C$. The parameter settings of this paper are ${\omega }_I=1,{\omega }_C=2$.

2.5. Model Training and Testing

The most prominent feature of the SGRL algorithm is training for a single agent, but the obtained model can be applied to a multi-agent environment. For trained agents, the vehicles around them can be considered the same category—surrounding vehicles. Therefore, the work done by the GNN-based reinforcement learning model can be interpreted as planning and decision-making based on the characteristics and relationships of the target agent and its surrounding vehicles. The model obtained by single-agent training can be directly transplanted to other agents in the same scenario, as shown in Figure 7.

To prevent the overfitting of the reinforcement learning process, we randomize the distribution of training vehicles and the task of autonomous vehicles in each episode based on fixed rules, as shown in Figure 8.

Algorithm 1 shows the detailed steps of training.

Algorithm 1 SGRL Q Learning Steps.
Initialize the reply memory R to capacity N
Initialize the weights for the SGRL Net ($\psi ,\chi ,\eta \dots \dots$)
Jointly Current Network $\stackrel{\wedge }{Q_{\theta }}$ and Target Network $\stackrel{\wedge }{Q_t}=\stackrel{\wedge }{Q_{\theta }}$
## Warming up ##
For step $t=1$ to $T_0$(warming up steps) do
	Choose random action for agent i: $a_{t-r}=np.random.choice(np.arrange\left(3\right),N)$
	Get the transition $(s_t,a_t,r_t,s_{t+1})$
	Store in the buffer
## Training step ##
For step $t=T_0+1$ to T(total steps) do
	With the probability e choose random action for agent i: $a_{t-r}$
	With the probability $1-e$ do:
		State decoding: $(X_t,C_t)=s_t$
		Double FCN: $H_{t-1}={\psi }_0\left(X_t\right)\in {\mathbb{R}}^{N\times 128}$; $H_{t-2}={\psi }_1\left(H_{t-1}\right)\in {\mathbb{R}}^{N\times 128}$
		GNN + FCN: $K_t=\chi (H_{t-2},C_t)\in {\mathbb{R}}^{N\times 128}$; $K_{t-1}={\psi }_2\left(K_t\right)\in {\mathbb{R}}^{N\times 128}$
		Feature Stitching: $F_t=(H_{t-2},K_{t-1})\in {\mathbb{R}}^{N\times 256}$
		Double FCN: $F_{t-1}={\psi }_3\left(F_t\right)\in {\mathbb{R}}^{N\times 128}$; $F_{t-2}={\psi }_4\left(F_{t-1}\right)\in {\mathbb{R}}^{N\times 128}$
		Compute Q values: $\stackrel{\wedge }{Q_{\theta }}\left(s_t\right)={\psi }_5\left(F_{t-3}\right)\in {\mathbb{R}}^{N\times 33}$
		Select $a_t^{*}=argmax\stackrel{\wedge }{Q_{\theta }}\left(s_t\right)$
	Execute $a_t^{*}$ and get a new state $s_{t+1}$
	Store in the buffer
	Set $s_t=s_{t+1}$
## Training model at training step ##
Sample a batch from the buffer and calculate the Q target:
	$Q_{target}=\left\{\begin{array}{cc}{r}_t+\gamma \underset{a}{max\stackrel{\wedge }{Q_{\theta }}\left(s_t\right)}& done=0\\ r_t& done=1\end{array}\right.$
Get average $Loss$
Update parameters $\theta$ of the model $\stackrel{\wedge }{Q_{\theta }}$
## Updating Target Net every n steps ##
$\stackrel{\wedge }{Q_{target}}=\stackrel{\wedge }{Q_{\theta }}$

Algorithm 2 shows the detailed steps of testing.

Algorithm 2 SGRL Testing Steps.
Initialize the simulation environment
$\#\#$ Testing step $\#\#$
For step $t=1$ to T(total test steps) do
	State decoding: $(X_t,C_t)=s_t$
	For AV $i=1$ to $N_{AV}$ do
		Move other AV’s features to the back of HV: $X_{t0}=\lambda \left(X_t\right)\in {\mathbb{R}}^{N\times 8}$
		Calculate Q based on trained model: $\stackrel{\wedge }{Q_{\theta }}\left(s_t\right)={\pi }_{SGRL}(X_{t0},C_t)\in {\mathbb{R}}^{N\times 33}$
		Select $a_{t-i}^{*}=argmax\stackrel{\wedge }{Q_{\theta }}\left(s_t\right)$
	Get the action matrix $a_t^{*}=(a_1^{*},a_2^{*},\dots \dots a_{N_{AV}}^{*})$
	Execute $a_t^{*}$ and get a new state $s_{t+1}$
	Set $s_t=s_{t+1}$

3. Simulation

3.1. Baseline Models

Two baseline models are introduced for comparative analysis in the simulation: the traditional DQN algorithm (NGRL) and the multi-agent training algorithm (MGRL). In the NGRL algorithm, the GNN part is removed, and the correlation matrix is used as input by splicing with the feature matrix.

The model of the MGRL algorithm is consistent with the SGRL proposed in this paper in terms of network structure, but its training process is based on multi-agent environment interaction.

By comparing the three models, the specific effects of the GNN structure and single-agent training of SGRL can be effectively analyzed. The simulation scenario is shown in Figure 9.

3.2. Simulator Parameters

The simulation scenario is a long highway with three lanes. There are exit ramps at one-third and two-thirds of the total length respectively. The speed limit for the whole road is set as 20 m/s (76 km/h) for all the vehicles. The AVs and HVs are put into the scenario at the probability of 0.1 and 0.4 from the left side of the road. And the initial speed and lane position are random.

There are six types of vehicles in the scenario, including HVs that travel straight through the highway, vehicles (HVs and AVs) that want to exit from ramp_0 and vehicles (HVs and AVs) that want to exit from ramp_1. The simulation environment controls the driving of HVs, and the AVs are completely controlled by the reinforcement learning model SGRL in real time.

The number of vehicles in the experiment is set as shown in

Table 2. Number setting of experimental vehicles.

	Algorithm Type	N of Vehicles
		AVs		HVs
		Merge_0	Merge_1
Training Process	SGRL	1		19
	MGRL	5	5	10
	NGRL	5	5	10
Testing Process	SGRL	5	5	10
	MGRL	5	5	10
	NGRL	5	5	10

.

4. Results and Discussion

4.1. Training Results

All three models were trained for 1000 episodes. The symbolic data of the training process are the reward, average Q value and loss value. Average rewards are obtained by averaging rewards against steps. The specific changes are shown in Figure 10, Figure 11, Figure 12 and Figure 13.

For the comparison of reward values in the training process, under the same reward value calculation, the reward values and average reward values of the SGRL algorithm can converge faster and have better final convergence.

The changing trend of the Q value and loss value of the SGRL algorithm in the training process is consistent with the learning process. According to the loss results, SGRL has a faster convergence speed.

To compare the task success rate, security, and traffic efficiency, the average velocity, number of collisions, success rate, and average steps per episode must be collected. The results of the above training data are shown in Figure 14, Figure 15, Figure 16 and Figure 17.

The SGRL algorithm has obvious advantages regarding task success rate and average vehicle speed. In terms of collision times and average training step length, SGRL also meets driving safety requirements.

In addition, SGRL has apparent advantages in training efficiency under the premise of the same training episodes. The specific hardware parameters and training time are shown in

Table 3. Computer hardware information.

Item	Type
CPU	Intel I9 10980XE
GPU	NVIDIA RTX3090(24G)
RAM	Crucial DDR4 3200MHz 32G × 4
SSD	SAMSUNG 970 EVO Plus 1T × 2
OS	Ubuntu 20.04

. The comparison of the training time is shown in

Table 4. Training time statistics (ten experiments for each algorithm; time unit is hours).

	SGRL	MGRL	NGRL
1	3.335665	66.40787	3.251174
2	3.687935	78.94237	4.812508
3	3.782653	71.37449	3.737191
4	3.763133	82.16632	4.360045
5	3.38374	66.463	3.259155
6	3.286891	72.25948	3.890604
7	3.874574	67.38896	3.840993
8	3.043649	67.68016	3.668797
9	3.368947	65.51202	3.017189
10	3.832845	69.72951	4.072374
Mean	3.536003	70.79242	3.791003

.

4.2. Testing Results

The trained model needs to be verified by the test process. To fully verify the algorithm’s effectiveness, we adjust the total length of the road under the premise of the same traffic flow (20 vehicles per episode). The three algorithms are tested on 1000 m, 750 m and 500 m roads to simulate different traffic flow and congestion levels.

Each test process includes 1000 episodes. In the test process, the reward value can still be used as an essential evaluation of model performance. The simulation results of reward value and average reward value are shown in Figure 18 and Figure 19.

For the most complex and congested 500 m highway scenario, the longitudinal motion spatial distribution of the three algorithms throughout the test cycle is as shown in Figure 20.

The longitudinal control of autonomous vehicles will become more and more complex with the increase in road congestion, so the test results of the 500 m scene are the optimal reference. The test results show that the action output of the SGRL algorithm is mainly concentrated near 0, and the probability of large acceleration is acceptable.

To compare the task success rate, security, and traffic efficiency, the average velocity, number of collisions, success rate, and average step per episode must be collected.

The data of different methods are listed in

Table 5. Performance comparison for different models.

Model	Road Length	Average_V	N_Collisions	Success_Rate	Average_Steps
SGRL	1000 m	9.55588	1.66888	0.94068	601.1014
	750 m	8.45047	1.73714	0.92385	494.9505
	500 m	7.50548	2.06911	0.91493	307.8586
MGRL	1000 m	3.60231	2.90205	0.54129	2478.409
	750 m	3.33545	2.96804	0.53864	1836.932
	500 m	3.20753	3.02719	0.47095	1334.704
NGRL	1000 m	8.92665	1.23373	0.53529	204.3344
	750 m	7.16045	1.83906	0.52708	166.5145
	500 m	6.70472	2.31418	0.4839	111.9219

, and the mean of the above data is shown in Figure 21, Figure 22, Figure 23 and Figure 24.

It can be seen from the figure and table that SGRL has apparent advantages in task success rate and average velocity. In terms of the number of collisions and the test steps, although the SGRL value is not the best, it is consistent with the best value.

4.3. Results Discussion

It can be seen that the SGRL algorithm has outstanding advantages over the MGRL algorithm and the NGRL algorithm. The SGRL algorithm can converge faster during the training process and achieve better data performance. In the testing process, the SGRL algorithm has outstanding data performance in terms of task success rate, average vehicle speed, and security.

The MGRL algorithm directly sums the reward value of all autonomous vehicles in the process of reward value calculation, so there is a phenomenon in which the excellent performance of vehicle behavior and the poor performance of vehicle behavior offset each other, which is not conducive to the adequate updating of parameters, and also causes the final convergence speed to be slow, and thus the convergence effect is not good.

The NGRL algorithm lacks the calculation consideration of the interaction process between vehicle individuals. Therefore, in the decision-making process, due to the relatively simple understanding of the environment, it cannot obtain sufficient data support, so the data performance is poor.

The SGRL algorithm considers and solves the above problems and optimizes the network structure. According to the comparison of data, it can be verified that the improvement of SGRL is obviously effective.

5. Conclusions

This paper proposes a generalized single-vehicle-based graph neural network reinforcement learning algorithm (SGRL algorithm). This algorithm combines GNN with deep reinforcement learning to solve the vehicle planning problem in the scenario of highway driving out of the ramp. The SGRL model is trained for a single agent and can be tested in multi-agent scenarios. At the same time, the algorithm sets up an improved reward function to provide a clear direction for training.

Comparing the three algorithms, the conclusions are as follows:

• Firstly, a training mode for single-agent training extended to multi-agent scenarios is proposed and verified in terms of training effectiveness and performance. The algorithm improves the analytical ability of the DRL by increasing the number of network nodes, thereby increasing the control dimension of vehicle decision-making to longitudinal and lateral dimensions.
• Secondly, the proposed SGRL algorithm simplifies the training mode and improves the training efficiency without affecting the training effect. This helps to adjust complex parameters and reduce time costs.
• Thirdly, SGRL is more sufficient in the training process to achieve a better convergence effect. The fluctuation is the smallest after the training data are stable. This shows that the proposed SGRL algorithm has outstanding training ability and is more suitable for decision-making based on reinforcement learning in multi-agent scenarios.
• Finally, the newly designed reward function effectively solves the problem of mutual influence between longitudinal and lateral control. SGRL can achieve higher task success rates and average velocity in the training and testing process. This shows that the new reward function, the training method for a single agent, and the incorporation of GNN effectively improve the decision performance of the model.

In future research, the continuity of model action space can be added to this algorithm, which will effectively improve the driving fluency of vehicles. In addition, the relationship between multiple agents in the scenario should not be limited to physical characteristics: decision-making, driving intention, and task priority can also be incorporated into the calculation process.