SGD-TripleQNet: An Integrated Deep Reinforcement Learning Model for Vehicle Lane-Change Decision

Abstract

With the advancement of autonomous driving technology, vehicle lane-change decision (LCD) has become a critical issue for improving driving safety and efficiency. Traditional deep reinforcement learning (DRL) methods face challenges such as slow convergence, unstable decisions, and low accuracy when dealing with complex traffic environments. To address these issues, this paper proposes a novel integrated deep reinforcement learning model called “SGD-TripleQNet” for autonomous vehicle lane-change decision-making. This method integrates three types of deep Q-learning networks (DQN, DDQN, and Dueling DDQN) and uses the Stochastic Gradient Descent (SGD) optimization algorithm to dynamically adjust the network weights. This dynamic weight adjustment process fine-tunes the weights based on gradient information to minimize the target loss function. The approach effectively addresses key challenges in autonomous driving lane-change decisions, including slow convergence, low accuracy, and unstable decision-making. The experiment shows that the proposed method, SGD-TripleQNet, has significant advantages over single models: In terms of convergence speed, it is approximately 25% faster than DQN, DDQN, and Dueling DDQN, achieving stability within 150 epochs; in terms of decision stability, the Q-value fluctuations are reduced by about 40% in the later stages of training; in terms of final performance, the average reward exceeds that of DQN (by 6.85%), DDQN (by 6.86%), and Dueling DDQN (by 6.57%), confirming the effectiveness of the proposed method. It also provides a theoretical foundation and practical guidance for the design and optimization of future autonomous driving systems.

1. Introduction

Autonomous driving systems, as cutting-edge technologies in the field of transportation, are centered around improving vehicle safety and decision-making efficiency [1,2]. The decision-making system, acting as the bridge between perception and control, not only needs to accurately understand complex road traffic environments but also must plan the optimal driving strategies for the vehicle [3]. Through artificial intelligence technologies such as deep reinforcement learning, the adaptability of the decision-making system in dynamic traffic scenarios can be significantly enhanced, reducing the risk of traffic accidents, optimizing traffic flow, and alleviating congestion issues [4]. Additionally, the research findings can be directly applied to improve existing driver assistance systems and promote the industrialization of autonomous driving technologies, bringing significant social benefits and economic value. With the rapid development of autonomous driving technology, lane-change decision-making has become an increasingly critical component in ensuring both safety and efficiency on the road [5,6]. Existing research indicates that behavioral decision-making models for autonomous vehicles can be divided into two main categories based on their underlying principles: rule-based methods and deep learning-based driving behavior decision models [7,8,9,10].

Rule-based driving behavior decision models guide vehicle behavior through human-designed rules or strategies. Before the advent of artificial intelligence, the implementation of lane-change decision (LCD) tasks was primarily solved using feedback control techniques [11]. These techniques typically used information collected from sensor measurements as input, employing classical control algorithms such as fuzzy logic models and LQR to stabilize vehicle behavior in simple lane-change tasks. With deeper research into the LCD problem, a new lane-change model was proposed that considers uncertainties or fuzzy factors and their relationships [12], which included simulations of mixed traffic flows at different penetration rates and demonstrated the impact of autonomous vehicle penetration and certain fuzziness on lane-change behavior. In the LQR approach, an improved LQR lateral motion control method for vehicle lane-changing with feedforward control was introduced, which enhanced control accuracy and adaptability [13]. However, these classical control methods generally require accurate mathematical models of the system, are better suited for simpler traffic scenarios, and often require the manual adjustment of tools and parameters. In dynamic or extreme environments, these methods may exhibit unstable behavior and cannot fully handle all unseen scenarios, thus posing safety risks [14,15].

With advances in technology and the continuous development of deep reinforcement learning (DRL), deep learning-based driving behavior decision models have become mainstream in research. These models generate decisions by learning from historical data or interacting with the environment, training deep neural networks [16]. In 2013, the DeepMind team introduced DRL through the Atari game platform, combining deep learning and reinforcement learning to train agents that automatically learn from environmental feedback (rewards), which sparked widespread interest in academia [17]. Researchers began using onboard sensors and communication network systems to gather scene information, focusing primarily on strategies to improve safety, comfort, and energy optimization for autonomous driving in complex environments. However, the research found that its generalization ability was poor, and it was difficult to implement in real-world environments. However, deep reinforcement learning-based lane-change decision frameworks were proposed to find the optimal decision-making strategy that minimizes the risks of autonomous driving [18,19]. However, compared to earlier methods, the end-to-end learning approach can more effectively handle complex and dynamic task environments [20]. Researchers then began studying integrated deep reinforcement learning models, combining Deep Deterministic Policy Gradient (DDPG) reinforcement learning with Long Short-Term Memory (LSTM) trajectory prediction models [21], significantly improving lane-change safety and efficiency. An improved Grey Wolf Optimization Deep Belief Network (DBN) model was introduced, effectively predicting the longitudinal and lateral positions of vehicles and enabling efficient lane-change prediction [22], but its stability was poor and lacked application in real-world traffic environments [23]. Subsequently, an integrated deep reinforcement learning-based driving behavior decision model for autonomous vehicles was proposed [24]. However, this model only used a simple voting mechanism to assign weights to a single model based on the current environmental state, and the data used were not from real-world environments, making it less capable of handling complex and dynamic traffic scenarios [25].

This paper proposes an ensemble deep reinforcement learning model, called SGD-TripleQNet. The model integrates three deep Q-learning networks and dynamically adjusts their individual weights through the Stochastic Gradient Descent (SGD) optimization algorithm to optimize the decision-making process. The weights are updated dynamically based on the loss functions of each model, enabling effective collaboration and performance enhancement between the models.

2. Materials

In this section, we introduce the advantages of the NGSIM dataset for vehicle LCD and describe the steps of data preprocessing, data filtering, and smoothing. These processes aim to enhance the model’s training performance and generalization ability, enabling a more accurate simulation of lane-change behavior in real-world traffic scenarios.

2.1. Environment Description

In LCD tasks, data quality plays a crucial role in ensuring the accuracy and reliability of decisions. A comprehensive dataset should encompass a wide range of real-world driving scenarios, including vehicle trajectories for straight driving, left turns, right turns, acceleration, and deceleration. Therefore, this study selects the NGSIM (Next Generation Simulation) dataset, a high-quality traffic data collection and simulation project initiated by the U.S. Department of Transportation, and specifically utilizes highway data without intersections to validate the vehicle lane-change model [26]. The NGSIM dataset provides detailed motion information for each vehicle on monitored road segments, recorded at 0.1-s intervals (such as speed, acceleration, and position). The dataset includes highway data collected at various locations and times, including segments of California’s US-101 and I-80, as well as urban street data from Peachtree Street in Atlanta and Lankershim Boulevard in Los Angeles. The US-101 and I-80 datasets are important sources of data for analyzing vehicle driving behavior. The data from the US-101 highway in California cover three time periods (each 15 s) between 7:50 and 8:35 a.m., reflecting different traffic flow and speed conditions. The data from the I-80 highway in California span three time periods (each 15 s) between 4:00 and 5:30 p.m., focusing on complex traffic scenarios such as highway merging and lane changing [27].

Both datasets are collected from multi-lane, bidirectional highways and provide detailed information on vehicle position, speed, acceleration, and headway distance. They include three different traffic flow conditions: sparse, medium-density, and dense. The I-80 dataset offers greater diversity in terms of traffic density and the number of lanes, and it features numerous merging sections. In contrast, the US-101 highway dataset is from a straight road environment with simpler scenarios, lacking complex highway merges or diverging lanes. This makes the US-101 dataset more suitable for early-stage research, as it allows the focus to be on lane-change behavior itself without the interference of complex traffic patterns. Therefore, this paper selects the US-101 as the lane-change environment for the study [28,29].

2.2. Data Filtering

The NGSIM (Next Generation Simulation) US-101 dataset includes data from three time periods: 7:50 a.m.–8:05 a.m., 8:05 a.m.–8:20 a.m., and 8:20 a.m.–8:35 a.m. These periods cover different traffic flow densities and dynamic variations during the morning rush hour. The data record traffic scenes ranging from sparse to medium-density to dense flow conditions. This paper analyzes vehicle behavior during these different time periods, as shown in the Figure 1.

As shown in the figure above, the data from 8:05 a.m.–8:20 a.m. demonstrate optimal conditions in terms of traffic flow, stability, and lane-change behavior.

The lane utilization curve for the 8:05 a.m.–8:20 a.m. period is similar to other time intervals but exhibits a more balanced distribution of data, which is beneficial for analyzing vehicle behavior across different lanes. The number of unique vehicles and total records during this period are at moderate levels, and the moderate traffic flow is conducive to studying lane-change behavior. It helps to avoid the issue of sparse or overly congested data, which could make lane-change behavior less evident.

Furthermore, the standard deviations of average speed (Velocity) and average acceleration (Acceleration) are relatively small, indicating that vehicle behavior during this period is more consistent with less fluctuation in the data. This makes it an ideal time segment for the research and modeling of vehicle lane-change decision-making.

2.3. Data Preprocessing

We performed outlier handling and smoothing on the NGSIM dataset. The specific preprocessing steps include handling missing values, outliers, and data transformation, followed by smoothing using the Savitzky–Golay filter [30,31].

In real-world traffic data, raw position data often exhibit high volatility due to measurement errors and sensor noise. The Savitzky–Golay filter effectively preserves the trend of the signal while reducing the impact of noise on the results. To minimize the effect of noise and extract meaningful vehicle motion information from the traffic data, we applied the Savitzky–Golay filter for smoothing [32,33]. The following Figure 2 illustrates the smoothed curve when the filter window size is 25.

3. Background

In this section, we describe three individual deep reinforcement learning (DRL) models: DQN, DDQN, and Dueling DDQN, and provide a detailed introduction to their algorithmic principles and model structures.

3.1. DQN Model

The Q-learning algorithm is the predecessor of the DQN (Deep Q-Network) algorithm. Q-learning was proposed by Watkins in 1989, while DQN is an algorithm that combines deep learning with reinforcement learning, introduced by the Google DeepMind team in 2015 [34]. DQN is a reinforcement learning algorithm based on Q-learning, enhanced by the incorporation of a deep convolutional neural network to approximate the value function, a simulation replay mechanism, and an independent target network. DQN is primarily designed to address issues in high-dimensional state spaces [35,36]. Reinforcement learning algorithms typically use Markov Decision Processes (MDP) to model reinforcement learning tasks. The specific implementation process is shown in Figure 3.

Where $\alpha$ is the learning rate, $\gamma$ is the discount factor, $r_t$ is the immediate reward obtained by the agent, and $s_t$ and $a_t$ represent the new state and the action taken in that state, respectively. In the lane-change decision-making task, DQN can estimate the future rewards for each lane-change option (such as left lane, middle lane, or right lane) by considering both the historical and current states, thereby guiding the decision-making process. In the model training process, the DQN model updates the parameters $\theta$ using gradient descent to optimize the loss function, which represents the difference between the target value network and the current value network outputs. The loss function is typically expressed by Equation (1):

$$L\left(\theta \right)=\mathbb{E}\left[{\left(y_t-Q(s_t,a_t;\theta )\right)}^2\right]$$

(1)

where

$L\left(\theta \right)$ is the loss function, representing the mean squared error between the target value network output and the current value network output. $y_t$ is the target value, given by $y_t=r_t+\gamma {max}_{a^{\prime }}Q(s_{t+1},a^{\prime };{\theta }^{-})$, where $r_t$ represents the immediate reward from the environment at time t, $\gamma$ is the discount factor, $s_{t+1}$ is the next state of the environment at time $t+1$, $a^{\prime }$ represents the possible actions that can be taken in the next state $s_{t+1}$, $Q(s_{t+1},a^{\prime };{\theta }^{-})$ is the Q-value for the next state–action pair under the target value network parameters ${\theta }^{-}$, and ${\theta }^{-}$ refers to the parameters of the target value network.

3.2. DDQN Model

DDQN (Double Deep Q-Network) is an improved version of DQN designed to address the overestimation problem inherent in DQN. The overestimation issue can lead to biased learning, which negatively impacts the performance of the policy. DDQN resolves this by splitting the maximization process in the target network into two parts: action selection using the current Q-network and action evaluation using the target Q-network. This approach effectively reduces the overestimation bias in DQN, thereby enhancing the model’s stability [36].

Although the two networks have different structures in the reinforcement learning task, they share the same input and output data types, making it possible to integrate them into a unified model. The target value calculation formula is shown in Equation (2).

$$y_t=r_t+\gamma Q(s_{t+1},arg\underset{a^{\prime }}{max}Q(s_t,a^{\prime };\theta );{\theta }^{-})$$

(2)

where $arg{max}_{a^{\prime }}Q(s_t,a^{\prime };\theta )$ is the action chosen by the current online network (Online Network) for the next state $s_{t+1}$, $Q(s_{t+1},a^{\prime };{\theta }^{-})$ is the Q-value output by the target network (Target Network) for the next state $s_{t+1}$ and the selected action $a^{\prime }$, $\gamma$ is the discount factor, typically in the range of [0, 1], and $r_t$ is the immediate reward received at time step t.

This target value formulation reduces overestimation bias by using the online network to select the action and the target network to evaluate it.

3.3. Dueling DDQN Model

Dueling DDQN (Dueling Double Deep Q-Network) is a further improvement upon DDQN, combining the Dueling Network Architecture with the action selection and evaluation separation mechanism of DDQN. This mechanism enables more efficient learning in certain states, as it avoids the excessive optimization of irrelevant actions and addresses the learning efficiency issues faced by traditional Q-learning methods in some states [37]. This improves both the performance and stability of the model [38].

The core idea of Dueling Network Architecture is to decompose the Q-value into two components; this is shown in Equation (3).

$$Q(S_t,A_t)=V\left(s\right)+A(s,a)$$

(3)

The final state–action $Q(s,a|\theta )$ in Dueling DDQN can be expressed as shown in Equation (4):

$$Q(s,a|\theta )=V\left(s\right|\theta ,\alpha )+(A(s,a|\theta ,\lambda )-{\displaystyle \frac{1}{\left|A\right|}}\sum _{a^{\prime }}A(s,a^{\prime }|\theta ,\lambda ))$$

(4)

where $V\left(s\right|\theta ,\alpha )$ represents the state value, $A(s,a|\theta ,\lambda )$ represents the advantage function, $\theta$ represents the hidden layer network parameters, $\left|A\right|$ represents the number of actions, $\alpha$ represents the state value network parameters, and $\lambda$ represents the action advantage network parameters.

4. Methods

In this section, we first define the state space using six features: vehicle speed, X/Y coordinates, current lane, and the distances to the front and rear vehicles. Next, we define the action space, which includes five actions: left lane change, right lane change, keep lane, accelerate, and decelerate. We then consider various reward and penalty factors for LCD and design a reward function that incorporates driving safety, traffic efficiency, driving comfort, lane-change behavior, and lane-keeping performance to ensure a comprehensive evaluation of the vehicle’s behavior under different traffic conditions. Finally, we provide a detailed introduction to the proposed SGD-TripleQNet model and its principles.

4.1. Set State Space

State Space is a core concept in reinforcement learning, used to describe the environmental state of an intelligent agent at a specific time [32]. The state space S represents the state information perceived by an autonomous driving vehicle while driving on the road, including the position (X, Y), velocity v, and acceleration a of both the ego vehicle and surrounding vehicles. Each state represents the current status of the vehicle, which serves as an input for the reinforcement learning model. The model selects actions based on the current state (State) and learns through a reward (Reward) function.

The state space consists of six key features, defined as a six-dimensional vector. Its function is represented by Equation (5):

$$(Presence,x,y,v,a,P)$$

(5)

where Presence indicates whether a vehicle exists; x represents the lateral position of the vehicle, indicating its position relative to the road laterally; and y represents the longitudinal position of the vehicle, indicating its position relative to the road longitudinally. v represents the real-time velocity of the vehicle in feet/second, reflecting the current dynamic state of the vehicle. P represents the state information of surrounding vehicles. The design of these state features can comprehensively describe the actual position and dynamic behavior of vehicles in the traffic environment.

4.2. Set Action Space

The action space is designed with five choices. In the autonomous vehicle behavior decision MDP model, the action space A (6) includes five driving behaviors to adapt to different driving scenarios: changing lanes to the left ($a_0$), changing lanes to the right ($a_1$), maintaining the current lane ($a_2$), accelerating in the same lane ($a_3$), and decelerating in the same lane ($a_4$):

$$A=\{a_0,a_1,a_2,a_3,a_4\}$$

(6)

This design ensures that the agent can consider actual traffic rules when making decisions and avoid selecting invalid lane-changing actions.

4.3. Design Reward Function

4.3.1. Basic Safety Reward

This reward is primarily based on evaluating inter-vehicle distance. A positive reinforcement is given when the headway exceeds the safety threshold, with the reward proportional to the margin above the safe distance; a negative punishment is given when the distance falls below the safety threshold, with the penalty proportional to the safety deficit. This design aims to encourage vehicles to maintain safe distances and avoid rear-end collision risks [34,35]. Its reward function is represented by Equation (7).

$$r_1=\left\{\begin{array}{cc}5.0·(SH-SD)\hfill & \mathrm{if}SH>SD\hfill \\ -1.0·(SD-SH)\hfill & \mathrm{if}SH\le SD\hfill \end{array}\right.$$

(7)

where, Space_Hdwy (SH) represents the current headway distance, and SAFE_DISTANCE (SD) is the preset safety distance.

4.3.2. Traffic Efficiency Reward

This reward incorporates two factors: speed deviation, which is the absolute difference between current and target speeds, and time step penalty. By applying a negative reward to speed deviation, it guides vehicles to maintain speeds near the desired velocity while considering time costs to promote efficient traffic flow. Its reward function is represented by Equation (8)

$$r_2=-0.5·|V_{\mathrm{vel}}-V_t|-\mathrm{\Delta }t$$

(8)

where $V_{\mathrm{vel}}$ is the vehicle velocity, $V_t$ is the target velocity, and $\mathrm{\Delta }t$ is the time step penalty.

4.3.3. Comfort Reward

The driving comfort is measured by evaluating vehicle acceleration and jerk (rate of change in acceleration), suppressing dramatic speed changes to enhance driving comfort. This paper designs the reward function using an inverse form, so that the smaller acceleration and jerk values result in larger rewards. Its reward function is represented by Equations (9) and (10).

$$\mathrm{jerk}={\displaystyle \frac{A_{vel}(t+1)-A_{vel}\left(t\right)}{\mathrm{\Delta }t}}$$

(9)

$$r_3=\rho {\displaystyle \frac{1}{1.0+\left|\mathrm{jerk}\right|+A_{vel}(t+1)}}$$

(10)

where $A_{vel}\left(t\right)$ is the acceleration at time t, $A_{vel}(t+1)$ is the acceleration at the next time step, $\mathrm{\Delta }t$ is the time step interval, jerk represents the rate of acceleration change, and $\rho$ is the comfort balance coefficient.

4.3.4. Lane-Change Behavior Reward

This component introduces a lane-change cooling mechanism. The agent receives a penalty for lane changes during the cooling period (T) to suppress frequent lane changes, provides a larger positive reward for lane changes outside the cooling period to support necessary lane-changing behavior, and gives a moderate positive reward for maintaining the current lane to encourage stable driving. If the agent chooses to change lanes or maintain the current lane, each action receives a reward of 5. The formula for its reward function is (11)

$$r_4=\left\{\begin{array}{cc}-5\hfill & \mathrm{if}\mathrm{\Delta }t<T\hfill \\ 10\hfill & \mathrm{if}\mathrm{\Delta }t\ge T\hfill \\ 5\hfill & \mathrm{if}\mathrm{stay}\hfill \end{array}\right.$$

(11)

where $\mathrm{\Delta }t$ represents the time elapsed since the last lane change and T is the cooling period threshold. A penalty is applied for lane changes during the cooling period ($\mathrm{\Delta }t<T$), a larger positive reward is given for lane changes after the cooling period ($\mathrm{\Delta }t\ge T$), and a moderate positive reward is given for maintaining the current lane (stay).

4.3.5. Violation Penalty

When the vehicle drives outside the normal lane boundaries, a significant negative penalty is applied to ensure that the vehicle always drives within the designated lanes. Its reward function is represented by Equation (12):

$$r_5=\left\{\begin{array}{cc}-5.0\hfill & \mathrm{if}{L}_{ID}<1\cup L_{ID}>4\hfill \\ 10.0\hfill & \mathrm{if}\mathrm{stay}\hfill \end{array}\right.$$

(12)

4.3.6. Final Total Reward

The final comprehensive reward can be expressed as the sum of the above five sub-rewards. This design enables the reward function to comprehensively evaluate vehicle lane-changing decision behavior. It balances traffic efficiency and driving comfort, effectively regulates vehicle lane-changing behavior, and weights each reward component to achieve optimal results, all while ensuring driving safety. The finall rewards function is represented by Equation (13).

$$R_{\mathrm{total}}=\sum _{i=1}^5{w}_i{r}_i$$

(13)

where $w_i$ is the weighting coefficients for each sub-reward component $r_i$.

4.4. SGD-TripleQNet Model

The core objective of this paper is to integrate three network models—DQN, DDQN, and Dueling DDQN—to collaboratively optimize lane-changing decisions through their respective advantages. At the start of model training, the initial weights $\alpha$, $\beta$, and $\gamma$ are assigned to each model. Initially, $\alpha =\beta =\gamma =1/3$, which is used to distribute weights among different models, and an SGD optimizer is used to update these weights with a certain step size. Then, the Q-values for each model are calculated independently, denoted as $Q_{DQN}$, $Q_{DDQN}$, and $Q_{DuelingDDQN}$. The Q-value output of TripleQNet can be expressed in Equation (14).

$$Q_{Triple}(s,a)=\alpha Q_{DQN}(s,a)+\beta Q_{DDQN}(s,a)+\gamma Q_{DuelingDDQN}(s,a)$$

(14)

where s represents the current state, and a represents the current chosen action (such as “lane left”, ”lane left”, “keep lane”, etc.). This method leverages the strengths of different models to achieve a more stable and robust policy.

After each training round, the gradient of the loss function is calculated, and the weights of each model are dynamically updated with a learning rate to minimize the total model loss. To prevent any model’s weight from becoming too large, an L2 regularization term is added to TripleQNet’s loss function (15) and (16).

$$L_{\mathrm{TripleQNet}}=\alpha L_{\mathrm{DQN}}+\beta L_{\mathrm{DDQN}}+\gamma L_{\mathrm{DuelingDDQN}}+L2$$

(15)

$$=\alpha L_{\mathrm{DQN}}+\beta L_{\mathrm{DDQN}}+\gamma L_{\mathrm{DuelingDDQN}}+\lambda ({\alpha }^2+{\beta }^2+{\gamma }^2)$$

(16)

The weights, shown in Equation (17), are updated according to the loss gradient during each epoch:

$$\{\alpha ,\beta ,\gamma \}\leftarrow \mathrm{SGD}(\alpha ,\beta ,\gamma ,\nabla L_{\mathrm{TripleQNet}})$$

(17)

The updated weights are used for Q-value calculation in the next round, allowing the model to gradually optimize its performance. The architecture diagram of SGD-TripleQNet is shown in the Figure 4.

4.5. DQN, DDQN, Dueling DDQN and SGD-TripleQNet Pseudocode

SGD-TripleQNet integrates the training processes of DQN, DDQN, and Dueling DDQN into a unified framework, leveraging the SGD algorithm to optimize the weights, thereby minimizing the total loss. Additionally, an L2 regularization method is incorporated to prevent any single model’s weight from becoming excessively large. The pseudocode for the SGD-TripleQNet algorithm, the individual DQN, DDQN, and Dueling DDQN is shown in Algorithm 1 below.

Algorithm 1 DQN, DDQN, Dueling DDQN, and SGD-TripleQNet Training

1: Initialize: state s, Input:$\alpha =\beta =\gamma =1/3$, learning rate $\eta$, L2 coefficient $\lambda$, initialized Q-learning networks: DQN, DDQN, Dueling DDQN, training dataset D, target network update frequency: T, batchsize. 2:    Neural networks: DQN, DDQN, Dueling DDQN 3:    Optimizers: Adam for each model 5:    Action space: $A=\{a_0,a_1,\dots ,a_4\}$ 6:    State space: $S=\{s_0,s_1,\dots ,s_m\}$ 7:    Replay buffer: $\mathcal{D}$ 8: For each episode = 1 to max_episodes: 9:    Initialize state $s_0$ at time step $t=0$ 10:    For each step $t=0$ to $max\_steps$: 11:       Choose action $a_t$ based on policy (e.g., epsilon-greedy) 12:       Execute action $a_t$, observe next state $s_{t+1}$, and reward $r_t$ 13:       Store transition $(s_t,a_t,r_t,s_{t+1})$ in $\mathcal{D}$ 14:       If episode > warmup_steps: 15:          Compute target $y_t$: 16:             For DQN: $y_t=r_t+\gamma ·{max}_{a}Q(s_{t+1},a)$ 17:             For DDQN: $y_t=r_t+\gamma ·Q_{\mathrm{target}}(s_{t+1},arg{max}_a{Q}_{\mathrm{online}}(s_{t+1},a))$ 18:             For Dueling DDQN: Similar to DDQN 19:          Compute loss: $\mathcal{L}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{(Q(s_i,a_i)-y_i)}^2$ 20:          Perform gradient descent to minimize $\mathcal{L}$ 21:          Update target network every T steps 22:    End if 23:    End for 24:    Update learning rates using SGD for SGD-TripleQNet: 25:       Adjust weights for each model based on performance 26: Update weights using SGD: $$\{\alpha ,\beta ,\gamma \}\leftarrow \mathrm{SGD}(\alpha ,\beta ,\gamma ,\nabla L_{\mathrm{TripleQNet}},\eta )$$ 27:    Ensemble Model Q-value and loss: 28:       $Q_{\mathrm{TripleQNet}}=\alpha ·Q_{\mathrm{DQN}}+\beta ·Q_{\mathrm{DDQN}}+\gamma ·Q_{\mathrm{Dueling}\mathrm{DDQN}}$ 29:      $L_{\mathrm{TripleQNet}}=\alpha ·L_{\mathrm{DQN}}+\beta ·L_{\mathrm{DDQN}}+\gamma ·L_{\mathrm{DuelingDDQN}}+L2$ 30: End for 31: End

5. Results

This section verifies the performance of the three individual models, DQN, DDQN, and Dueling DDQN, on lane-change decision (LCD) through training. It also implements an integrated model with optimized weight distribution using the SGD optimization algorithm, and discusses the training results, considering factors such as driving safety, lane change necessity, and comfort.

In the training process, we conducted model training over 1000 epochs and evaluated the model performance using various metrics including the loss function, Q-values, and rewards. We designed and conducted comparative experiments across four algorithms: traditional DQN, DDQN, Dueling DDQN, and the proposed SGD-TripleQNet. The stability of the SGD-TripleQNet model was evaluated, and the average rewards obtained by the agent under different policies were compared. In order to evaluate and compare the learning capabilities and stability characteristics of these algorithms, we conducted comprehensive training of three models: DQN, DDQN, and Dueling DDQN. Throughout the training process, we monitored and recorded both the evolution of loss values and weight convergence patterns, which are summarized in the subsequent table. The models were trained for 1000 epochs, and their final loss values and weight distributions are presented in Figure 5.

The training loss curves illustrate the convergence characteristics of three deep reinforcement learning models: DQN, DDQN, and Dueling DDQN. Initially, all three models start with a loss value of approximately 8.5. However, during the first 50 epochs, the loss rapidly decreases, indicating efficient initial learning and parameter optimization. Around the 100th epoch, the loss curve shows a clear inflection point, after which the rate of change gradually slows down. From epoch 200 onward, the loss stabilizes at around 2.0, with minimal fluctuations, indicating that the models have reached a stable convergence state. During the later stages of training (epochs 400 to 1000), the curves remain stable, validating the effectiveness of the chosen hyperparameters, gradient descent optimization method, and training strategy. All three models successfully converge and find their optimal parameters.

Finally, after 1000 epochs, the final loss values and the weights of the three models are shown in the

Table 1. Final loss values of DQN, DDQN, and Dueling DDQN after 1000 epochs.

Model	Loss	Weight
DQN	8.9710	0.3215
DDQN	8.9701	0.3319
Dueling DDQN	8.9958	0.3466

below.

We plot the average Q-value curve and the average action–reward curve for four models during the training process in Figure 6 and Figure 7. The average Q-value curve represents the model’s estimate of the expected cumulative future reward, while the average action–reward curve reflects the agent’s average reward per action throughout the task. From the average Q-value curve, we can observe that in the early stages of training (0–50 epochs), all models exhibit a consistent rapid learning pattern, with Q-values quickly rising from 0 to a peak of 11. In the optimization phase (50–200 epochs), the SGD-TripleQNet model shows smaller oscillations (±0.3), while the single models exhibit relatively larger fluctuations (±0.5).

Between 200 and 1000 epochs, the Q-value for SGD-TripleQNet stabilizes at around 9.7 ± 0.1, which is approximately 2.1% higher than the single models, which stabilize at 9.5 ± 0.15. All four models eventually converge and stabilize, reflecting an accurate estimation of state–action values and validating the model’s strong generalization ability and decision-making performance. However, the performance of Dueling DDQN is similar to that of DQN, indicating that the separation of state value and advantage function does not provide a significant advantage in this LCD task.

The average reward curve clearly demonstrates the model’s performance in real-world decision-making. Starting from an initial random policy level (around 3.7), the model rapidly improves within the first 50 epochs. SGD-TripleQNet reaches an average reward of 9.8 at epoch 75, while the three single models stabilize around 9.3. Ultimately, SGD-TripleQNet maintains the highest average reward of 10.0 ± 0.05, with minimal fluctuation, showcasing its superior stability in decision-making and performance optimization.

The cumulative average reward curve represents the running average of rewards accumulated over each training epoch, reflecting the model’s long-term learning performance. In Figure 8, we plotted the cumulative average reward curves for DDQN, Dueling DDQN, DQN, and SGD-TripleQNet. Over the first 800 epochs, their performance steadily improves as the models learn. From epochs 800 to 1000, the curves stabilize with minimal fluctuations, indicating that the models have learned a relatively stable strategy. Notably, the SGD-TripleQNet model consistently achieves higher and more stable reward values than the individual models, suggesting that SGD-TripleQNet has the best learning performance.

Based on the comprehensive analysis of the average Q-value, average reward curves, and cumulative average reward, SGD-TripleQNet demonstrates clear advantages in terms of convergence speed, decision stability, and generalization. This also confirms that the ensemble deep reinforcement learning strategy not only effectively overcomes the limitations of single models but also validates the effectiveness of dynamic weight adjustment mechanisms in deep reinforcement learning. Furthermore, it provides reliable technical support for the practical application of autonomous driving decision systems. We also recorded the average Q-value, average reward, and cumulative average reward after 1000 epochs in

Table 2. The average Q-value, average reward, and cumulative average reward after 1000 epochs.

Model	Average Q-Value	Average Rewards	Cumulative Average Reward
DQN	8.9710	8.9713	8.0875
DDQN	8.9701	8.9707	8.0696
Dueling DDQN	8.9958	8.9965	8.1487
SGD-TripleQNet	9.1292	9.5863	8.6507

below.

6. Conclusions

This study proposes an integrated vehicle lane-change decision-making model based on deep reinforcement learning, named SGD-TripleQNet. The model leverages an innovative dynamic weight adjustment mechanism to successfully integrate the complementary advantages of DQN, DDQN, and Dueling DDQN, thereby improving the accuracy and reliability of lane-change decisions while achieving balanced collaboration among the sub-models. Compared to single models, SGD-TripleQNet demonstrates superior performance, addressing common issues of slow convergence and instability in traditional methods. Additionally, the integrated model achieves a good exploration–exploitation balance during the training phase and maintains consistently high performance in later stages. Through the design of a multidimensional reward function that incorporates dynamic safety distance and speed control mechanisms, the model effectively balances efficiency and safety, improving decision-making safety while optimizing travel efficiency.

However, there are limitations to our work that require further improvement. First, the training and inference processes of the integrated model demand significant computational resources. Due to resource constraints, our experiments were conducted on a subset of NGSIM data, which may result in certain limitations in the decision-making outcomes. Second, our research is confined to highway scenarios. The model’s performance in more complex environments, such as intersections, or under extreme conditions like adverse weather, needs further validation to assess its generalizability. Furthermore, the data used in this study are limited to a specific time period during the morning rush hour on a single highway. The lack of data from other time periods, such as nighttime or dusk, restricts the model’s ability to adapt to more diverse real-world traffic conditions. To address these issues, we plan to incorporate a multimodal mechanism and expand the dataset in future work, including more scenarios to enhance the model’s robustness in handling complex and dynamic road environments.