Dual-Layer Path Planning Model for Autonomous Vehicles in Urban Road Networks Using an Improved Deep Q-Network Algorithm with Proportional–Integral–Derivative Control

Abstract

With the continuous progress of intelligent transportation systems and automated driving technologies, complex urban road environments put forward higher requirements on the real-time characteristic and accuracy of path planning algorithms. Traditional single-layer path planning methods struggle to effectively handle the complexity of road and lane networks, leading to high computational complexity and suboptimal planning outcomes. To address this issue, we propose a dual-layer path planning model. First, at the road level, we employ the A* algorithm to efficiently determine the optimal macroscopic route, reducing computational overhead. At the lane level, we introduce a Proportional–Integral–Derivative Q-network (PIDQN) based on deep reinforcement learning, which leverages PID control mechanisms to enhance lane selection accuracy and adaptability. By incorporating proportional, integral, and derivative control, PIDQN effectively handles dynamic environments and avoids local optima, ensuring stable and faster convergence. Compared with traditional Deep Q-Network (DQN) and Q-learning algorithms, PIDQN demonstrates significant improvements in success rate and convergence speed in path planning tasks. Using high-precision maps in real-world environments and Python for simulation experiments, we verify the superiority of this approach in complex urban road networks, and we compare the performance of traditional A* algorithms and two-layer planning algorithms. The results show that the two-layer planning algorithm outperforms the traditional A* algorithm and provides a more robust and efficient solution for self-driving car navigation.

1. Introduction

With the rapid growth of urbanization and the surge in vehicle ownership, modern cities face significant challenges related to traffic congestion and road safety [1]. The complexity of urban road networks, characterized by intricate intersections, densely populated areas, and varying road conditions, necessitates advanced path planning algorithms that can navigate these environments efficiently. Traditional path planning methods, which typically rely on single-layer models, struggle to account for the detailed structure of urban roads, where both road-level and lane-level information are crucial for optimal navigation. As a result, these methods often suffer from high computational complexity and fail to provide real-time solutions [2].

Many classical planning algorithms have been proposed to solve the problem of how to plan rational urban roads. These algorithms are broadly categorized into several types, the first is the deterministic algorithms represented by Dijkstra [3] and Floyd [4], which find the optimal path by traversing all the nodes from the node to the end point and calculating the value of the generation needed to reach each node, which is computationally intensive but guarantees that an efficient path can be found to reach the goal point and therefore is still a practical and effective method.

However, due to the large number of nodes in urban roads, traversing each node leads to a decrease in real-time performance. For this problem, the second class of heuristic algorithms, represented by A* [5], D* [6], and D* Lite [7], provides an innovative solution. These algorithms are based on deterministic methods and utilize heuristic functions to estimate the distance between the current node and the target point. This estimation serves as a guide, directing the algorithm toward nodes closer to the target point. By doing so, they significantly reduce the number of nodes traversed while ensuring an appropriate path is found, thereby improving search efficiency. Among these, the A* algorithm is widely used in the field of autonomous driving to provide global path planning for intelligent vehicles. However, in complex and dynamic urban road environments, the algorithm requires recalculating the path every time there is a change in the road environment, which introduces significant limitations.

Heuristic algorithms and deterministic algorithms depend on the cost function, but it is difficult to design a reasonable cost function in complex urban road environments, so Nature-inspired metaheuristic methods are used such as ant colony optimization (ACO) algorithms [8] and artificial bee colony algorithm (ABC) [9]; they explore the optimal path by mimicking the characteristics of animals in nature. ACO induces ants to search for paths by setting the concentration of pheromones, and the ABC algorithm uses nectar sources and messaging to explore reasonable paths. These methods are practical, but the computation time increases as the complexity of the environment increases.

Reinforcement learning (RL) algorithms are able to make dynamic decisions in complex urban road environments by interacting with the environment; thus, this approach has been widely used in recent years to deal with path planning problems. The most representative of these is the Q-Learning algorithm based on value learning, which is a model-free RL method. Q-Learning plans a reasonable path through a Q-table by interacting with the environment to obtain rewards in a finite state space. Ref. [10] successfully realized a reasonable path to reach the destination in a raster map environment using Q-Learning algorithm. Ref. [11] proposed a cooperative Q-Learning to deal with unexpected situations encountered in urban environments and to plan a safe path to reach the target point. However, state spaces in real-world environments are often very complex, and the Q-table is not applicable to complex state spaces. Ref. [12] proposed the classical DQN algorithm, which utilizes neural networks instead of a Q-table to solve this challenge. Ref. [13] solved the path planning problem for autonomous vehicle in unknown environment on urban roads through DQN. Ref. [14] input radar-collected data as state information into a DQN neural network and applied it to path planning for urban roads. Most of the existing studies [15,16,17,18] adopt a two-tier search strategy based on traditional methods but focus mainly on finding the shortest paths in the urban road network and tend to ignore the complexity of road geometry and lane-specific behaviors in dense urban environments’ details, leading to unsatisfactory planning results. To address this problem, we propose a two-layer path planning model. By dividing the planning process into a road layer and a lane layer, the model is able to effectively handle the complex interactions between roads and lanes. With this two-layer structure, the model not only improves the accuracy and efficiency of path planning but also exhibits greater adaptability and flexibility in complex urban road networks.

This paper utilizes high-precision urban maps as prior information to construct a two-layer path planning model. In the road level, different optimization objectives are established for both the road and lane levels. The traditional A* algorithm is employed at the road level to address the computational efficiency and connectivity issues in global path planning, reducing the number of lane nodes and thereby decreasing the computational complexity of lane-level planning. At the lane level, the planning process is further optimized to effectively avoid common issues in global planning, such as frequent lane changes and dead-end selections, thereby improving the comfort and safety of the planned paths. By combining Deep Q-Network (DQN) algorithms with PID [19] control mechanisms, we enhance the decision-making ability of the DQN algorithm, making lane selection and lane-change decisions more intelligent and enabling dynamic adaptation to the complex urban traffic environment.

This paper is organized as follows. In Section 2, we introduce a dual-layer path planning model for urban roads, and an improved DQN algorithm is presented in Section 3. Finally, our experiments and conclusions are provided in Section 4 and Section 5, respectively.

2. Urban Road Double-Layer Planning Model

2.1. Urban Network Environment Modeling

In the urban road environment, we construct the connection relationship between nodes in the urban road grid based on high-precision maps, as shown in Figure 1, which is a part of the high-precision map of the urban road. As shown in Figure 2, we construct the directed graph $G\left(V,E\right)$ based on the road identifier $\mathit{Road}\mathit{ID}$ and the lane identifier $\mathit{Lane}\mathit{ID}$, where the set of nodes V represents all nodes in the graph, and each node is represented by a binary $\left(r_i,l_i\right)$, where $r_i$ is a road identifier and $l_i$ is a lane identifier. Thus, the set of nodes can be represented as:

$$V=\left\{(r_i,l_i)\right|r_i\in Road\_IDs,l_i\in Lane\_IDs\}$$

(1)

The set of edge E represents the connectivity between nodes, each edge $e=((r_i,l_i),(r_j,l_j))$ represents a directed path from $l_i$ on $r_i$ to $l_j$ on $r_j$, and the direction of the edge indicates the traveling direction from the start node to the target node.

2.2. Double-Layer Planning Model

In the complex urban road environment, there are a large number of nodes and diverse topologies. Traditional single-level path planning methods have difficulties dealing with a large number of road and lane nodes effectively, which often leads to high computational complexity and inefficient path planning, and traditional single-level planning algorithms ignore the complex structure of roads and lanes. To solve this problem, we propose a two-stage topology modeling approach based on the urban road environment. Our model effectively integrates road geometry, lane selection, and lane-changing behavior by modeling roads and lanes hierarchically. By treating road planning and lane planning separately, we are able to significantly reduce the complexity of planning and improve the efficiency and accuracy of the navigation of self-driving vehicles in urban environments.

We split the original road directed graph G into two simpler directed subgraphs, the road-level-based directed graph $G_1(V_1,E_1)$ and the lane-level-based directed graph $G_2(V_2,E_2)$. In the road hierarchy $G_1(V_1,E_1)$, the node set $V_1$ denotes all road segments, and each node $v_i\in V_1$ corresponds to a specific road, while the set of edges $E_1$ represents the connectivity between roads, each edge $e_{ij}\in E_1$ represents a directed path from road segment $v_i$ to road segment $v_j$. In the lane hierarchy, the directed graph $G_2(V_2,E_2)$ is an optimal subset of $G_1(V_1,E_1)$, the set of nodes $V_2$ represents the start and end points of each lane of the optimal road sequence computed from the directed graph $G_1(V_1,E_1)$, each node corresponds to a unique lane $l_i\in V_2$, and each edge $e_{ij}\in E_2$ represents a directed path from lane to lane containing the feasibility of lane changing, merging, and splitting.

The optimization objective at the road level is to obtain the optimal set of road nodes $V_2$ that minimizes the objective function $F_1$, thereby constructing the subgraph $G_2(V_2,E_2)$ of the lane level, which contains all lane nodes in the optimal road nodes $V_2$ as edges of the subgraph $G_2(V_2,E_2)$ in $E_2$. At the road level, the optimal path problem can be expressed as the following optimization objective:

$$F_1=min\sum _{(i,j)\in E_1}{w}_1(i,j)$$

(2)

where $w_1$ is the composite cost function at the road level, consisting of the following components:

$$w_1(i,j)=\alpha d(i,j)+\beta \kappa (i,j)+\tau C_{\mathit{turn}}(i,j)$$

(3)

The length $d(i,j)$ of each road is determined by the Euclidean distance between the start and end points:

$$d(i,j)=\sqrt{{\left(x_j-x_i\right)}^2+{\left(y_j-y_i\right)}^2}$$

(4)

where $i,j$ denote the starting and ending points of the road, and $x,y$ are the corresponding geographic coordinates, weighting factor $\alpha$ used to adjust the weight of the impact of road length in total costs. Selecting shorter road segments helps to reduce travel time and energy consumption, so minimizing this part of the cost is usually desirable. We introduce the road segment curvature $\kappa (i,j)$ to quantify the degree of road curvature:

$$\kappa (i,j)=\sum _{p=1}^{n-1}\left|{\displaystyle \frac{\Delta {\theta }_p}{\Delta d_p}}\right|$$

(5)

where $\Delta {\theta }_p$ denotes the change in direction angle between two neighboring road points in the current road segment:

$$\Delta {\theta }_p=arctan\left({\displaystyle \frac{y_{p+1}-y_p}{x_{p+1}-x_p}}\right)-arctan\left({\displaystyle \frac{y_p-y_{p-1}}{x_p-x_{p-1}}}\right)$$

(6)

where $(x_{p-1},y_{p-1}),(x_p,y_p),(x_{p+1},y_{p+1})$ are the coordinates of three consecutive waypoints on the path. $\Delta d_p$ denotes the arc length or Euclidean distance between two neighboring waypoints, calculated as:

$$\Delta d_p=\sqrt{{\left(x_{p+1}-x_p\right)}^2+{\left(y_{p+1}-y_p\right)}^2}$$

(7)

$\kappa (i,j)$ is an accumulation of curvature values for each pair of neighboring road points along the entire road segment, and this accumulation reflects the overall curvature of the road, not just the curvature change at a local location. $\beta$ is used to adjust the importance of curvature costs.

Turning costs $C_{turn(i,j)}$ are considered in two main parts, the first is the difference $\Delta \theta =\left|{\theta }_i-{\theta }_j\right|$ between the heading ${\theta }_i$ of the road currently located and the heading ${\theta }_j$ of the extended road as part of the turning cost, and the other part is the consideration of the road extension cost $L_{ij}$:

$$L(i,j)=\sum _{p=1}^{n-1}\Delta d_p$$

(8)

From these two components, we define the turning cost function $f_{\mathit{turn}}(\Delta \theta ,L_{ij})$:

$$\begin{array}{c}{f}_{\mathit{turn}}(\Delta \theta ,L_{i,j})=\Delta \theta ·L_{i,j}\end{array}$$

(9)

$$\begin{array}{c}{C}_{\mathit{turn}}(i,j)=f_{\mathit{turn}}(\Delta \theta ,L_{i,j})\end{array}$$

(10)

Weighting factor $\tau$ adjusts the degree of influence of the cost of turning. In complex urban road environments, frequent turns may increase accident risk and fuel consumption, so this cost is usually scaled up to avoid frequent turns.

The optimization objective of the lane layer is to find the set of lane sequences in the optimal road subgraph $G_2(V_2,E_2)$ that minimizes the objective function $F_2$. At the lane level, the optimization objective of the optimal path can be expressed as:

$$F_2=min\sum _{(k,l)\in E_2}{w}_2(k,l)$$

(11)

where $w_2$ is the composite cost function at the lane level and includes the following components:

$$w_2(k,l)=\delta d_{\mathit{lane}}(k,l)+\mu C_{\mathit{speed}}(k,l)+\eta C_{\mathit{change}}(k,l)+\phi C_{\mathit{expansion}}(k,l)$$

(12)

In the cost function, lane length cost $d_{lane}(k,l)$ is defined as the cumulative distance of all discrete waypoints between lane node k and lane node l, which is defined by the equation:

$$d_{lane}(k,l)=\sum _{p=1}^{n-1}\Delta d_p$$

(13)

Similar to road segment length, it is often desirable to minimize this cost to reduce travel distance. Weighting factor $\delta$ adjusts the weight of the cost of lane lengths. The speed cost $C_{\mathit{speed}}(k,l)$ is used to indicate the speed limit for traveling on the lane segment k to the lane segment l:

$$C_{\mathit{speed}}(k,l)=\left|v_{\mathit{current}}\left(k\right)-v_{max}\left(l\right)\right|$$

(14)

where $v_{\mathit{current}}\left(k\right)$ denotes the current lane speed, and $v_{max}\left(l\right)$ denotes the maximum speed limit of the extended lane. A larger difference between the extended lane and the current speed indicates that the vehicle can pass through the lane more quickly, but high speeds will bring safety risks at the same time. Weighting factor $\mu$ adjusts for the effect of the speed cost. High weights $\mu$ mean that the model prefers lanes with high speed limits.

Lane change cost $C_{\mathit{change}}(k,l)$ indicates the cost of switching from the current lane to another lane, taking into account factors such as the frequency of lane changes $f_{\mathit{frequency}}$ and the length of lane changes $f_{\mathit{length}}$. We define a nonlinear combinatorial exponential function $f_{\mathit{lane}\_\mathit{change}}(k,l)$ to represent the lane change cost:

$$\begin{array}{c}{f}_{\mathit{lane}\_\mathit{change}}(f_{\mathit{frequency}},f_{\mathit{length}})={\mathrm{e}}^{{\rho }_1·f_{\mathit{frequency}}}+{\mathrm{e}}^{{\rho }_2·f_{\mathit{length}}}\end{array}$$

(15)

$$\begin{array}{c}{C}_{\mathit{change}}(k,l)=f_{\mathit{lane}\_\mathit{change}}(f_{\mathit{frequency}},f_{\mathit{length}})\end{array}$$

(16)

where $f_{\mathit{frequency}}=n_{\mathit{change}}$ is the cumulative number of lane changes during lane hierarchical planning, with more lane changes representing more frequent lane changes. The lane change cost $f_{\mathit{length}}$ is related to the length of the lane change; the shorter the length of the lane changed, the greater the risk of lane change. ${\rho }_1$ and ${\rho }_2$ are adjustable parameters that control the relative importance of the frequency of lane change and the target lane length cost. A higher weight $\eta$ encourages fewer lane changes in path planning and avoids unnecessary lane changing operations.

Lane expansion cost $C_{\mathit{expansion}}(k,l)$ is used to measure the ease or flexibility of expanding from the current lane to other lanes, which reflects the diversity and the amount of choice space available to reach other subsequent lanes from the current lane. Higher expandability means more possibilities to reach other lanes from that lane and more flexibility; lower expandability means fewer choices and limited flexibility in path planning. We assume that m subsequent lanes can be extended from lane k. The extension probability of each subsequent lane is $p_i$, $i\in (1,2,\cdots ,m)$. We introduce the scalability entropy $H\left(p_i\right)$ to characterize the diversity of subsequent lane choices:

$$H\left(p_i\right)=-\sum _{i=1}^m{p}_{i}log\left(p_i\right)$$

(17)

where $p_i$ denotes the probability of expanding from lane k to the subsequent lane l. We assume that the probability of lane selection follows a uniform distribution:

$$p_i={\displaystyle \frac{1}{m}}$$

(18)

A higher expansion entropy means that there are more options for lane expansion, meaning that the lane is more scalable:

$$C_{\mathit{expansion}}(k,l)={\displaystyle \frac{1}{H\left(l_i\right)}}$$

(19)

In complex urban road environments, some lanes may have fewer paths to choose from due to restrictions or congestion. By considering the cost of lane expansion, path planning can avoid selecting those poorly expandable lanes, thus reducing possible traffic bottlenecks or restrictions. $\phi$ is the weight used to adjust the expansion cost.

By minimizing two objective functions $F_1$ and $F_2$, we convert the urban road path planning problem into a multi-objective optimization problem to obtain the optimal global path. Road hierarchy planning focuses on macroscopic path selection, while lane hierarchy planning further optimizes specific lane selection.

3. Urban Path Planning

3.1. Road Level Planning

In urban road networks, the number of lane nodes is large and complex, and the computational complexity of direct path planning on all lane nodes is too high to meet real-time requirements. In order to efficiently find the optimal path, we first use the A* algorithm [5] to quickly find a feasible road sequence at the road level and then further optimize the lane selection at the lane level. The goal of the A* algorithm is to find the optimal path on the directed graph $G_1(V_1,E_1)$ from the starting road segment node $n_{begin}$ to the ending road segment node $n_{end}$, thus minimizing the integrated cost function $F_1$.

The cost function of the traditional A* algorithm is defined as follows:

$$f(n,n^{\prime })=g\left(n^{\prime }\right)+h\left(n^{\prime }\right)$$

(20)

where $g\left(n^{\prime }\right)$ denotes the actual cost of expanding a road node $n^{\prime }$, and $h\left(n^{\prime }\right)$ denotes the heuristic cost of expanding a node $n^{\prime }$ to reach the end point generally denoted by Euclidean distance.

The traditional A* algorithm only considers the intrinsic length of the road, ignoring the geometry and scalability of the road. In Algorithm 1, we improve the cost function of the A* algorithm based on the objective function at the road level. This modification makes the planned paths more reasonable and more adaptable to the geometric characteristics and comfort requirements of the road. In Algorithm 1, we initialize two lists, an open list and a closed list, and the total cost function of the A* algorithm is derived from the road-level planning model:

$$f(n,n^{\prime })=w_1(n,n^{\prime })$$

(21)

We select the node with the minimum total cost from the open list as the current node n. If the current node is the goal node, the path search is complete, and the optimal path is returned. Otherwise, we remove the current node from the open list and add it to the closed list. For each neighboring node of the current node, we compute the actual cost increment from the current node to the successor node $n^{\prime }$:

$$\Delta g(n,n^{\prime })=\alpha d(n,n^{\prime })+\beta \kappa (n,n^{\prime })+\tau C_{turn}(n,n^{\prime })$$

(22)

If $n^{\prime }$ is in the closed list and the newly computed $\Delta g(n,n^{\prime })$ is lower than its already existing cost, we update its cost and replace $n^{\prime }$ from the closed list back to the open list. If the node $n^{\prime }$ is not in the open list or its cost can be updated, we update its cost and parent node, and then add or update the neighboring node in the open list. By utilizing the structure of the urban road network represented by the directed graph $G_1(V_1,E_1)$ and using the A* algorithm for preliminary path planning at the road layer, the optimal sequence of roads from the start to the end point can be efficiently found.

Algorithm 1 A* algorithm

Initialize: the open list with start road node $n_{\mathit{begin}}\in {\mathbf{G}}_1$ and closed list as empty 1: while open list is not empty: do 2:     Select current road = n with the lowest $f(n,n^{\prime })$ from the open list 3:     if $n==n_{\mathit{end}}$ then 4:         return 5:     else 6:         Remove n from the open list and add it to the closed list 7:         for each neighbor $n^{\prime }$ of n: do 8:            Calculate the actual cost increment: $\Delta g(n,n^{\prime })$ 9:            if $n^{\prime }$ is in closed list then 10:                Continue 11:            else if $n^{\prime }$ is in open list and $\Delta g(n,n^{\prime })<f(n,n^{\prime })$ then 12:                Update $f(n,n^{\prime })=\Delta g$ 13:                Move $n^{\prime }$ from closed list back to open list 14:            else 15:                Add $n^{\prime }$ in the open list 16:            end if 17:         end for 18:     end if 19: end while

3.2. Lane-Level Planning

In an urban road environment, the road hierarchy provides the optimal road segment paths from the starting point to the end point using the optimal sequence of roads found by the A* algorithm on the road-level directed graph $G_1$. However, to further refine the planning to specific lane levels, we need to perform lane hierarchy planning on each selected road segment.

In order to select the optimal path in a given lane network, this paper proposes a Proportional–Integral–Differential Q-Network (PIDQN) lane hierarchical path planning method based on an improved Deep Q-Network (DQN). The method guides the vehicle to learn the optimal path selection strategy on the lane hierarchical directed graph $G_2$ by minimizing the composite cost function $F_2$. In DRL [20], we model the problem as a Markov Decision Process (MDP) [21] represented by a quintuple $(S,A,P,R,\gamma )$. The state space S is the set of states of a vehicle in a lane hierarchical directed graph $G_2$. The current $s_t\in S$ consists of the road number $Road\_ID$, the lane number $Lane\_ID$ and the speed of the vehicle in which the vehicle is currently located:

$$s_t=(Road\_ID,Lane\_ID,speed)$$

(23)

Action space A is the set of actions that a vehicle can take in each state, in order to describe the actions that a smart vehicle can take in each state, we define the action space of a smart vehicle as all the possible successor lane nodes in the directed graph $G_2$. In state $s_t$, the smart vehicle selects an action $a_t\in A$, which denotes a transfer from the current lane node k to the successor lane node l.

The state transfer probability $P(s_{t+1}\mid s_t,a_t)$ is the smart vehicle taking action $a_t$ in state $s_t$ and then transfers to the next state $s^{\prime }$ with probability. The reward function $R(s_t,a_t)$ is used to guide the intelligent vehicle towards minimizing the integrated cost function $F_2$. The reward function is designed as follows:

$$\begin{array}{c}R\left(s_t,a_t\right)=-w_2(s_t,a_t)\end{array}$$

(24)

$$\begin{array}{c}R\left(s_t,a_t\right)=-(\delta d_{\mathit{lane}}(s_t,a_t)+\mu C_{\mathit{speed}}(s_t,a_t)+\eta C_{\mathit{change}}(s_t,a_t)+\phi C_{\mathit{expansion}}(s_t,a_t))\end{array}$$

(25)

Intelligent vehicles aim to minimize the integrated cost function through iterative training and learning:

$$\underset{\pi }{min}\mathbb{E}[\sum _{t=0}^T{\gamma }^t·w_2(s_t,a_t)]$$

(26)

where $\pi$ is the policy function that defines the probability of action selection in each state $s_t$, and $\gamma$ is the discount factor used to balance current and future rewards.

DQN is a DRL method based on value learning [22], which evaluates the value of each action through a Q-value function $Q^{*}(s_t,a_t;\theta )$ based on the Bellman equation:

$$Q^{*}(s_t,a_t;\theta )=\mathbb{E}\left[r+\gamma \underset{a_{t+1}}{max}Q(s_{t+1},a_{t+1};\theta )|s_t,a_t\right]$$

(27)

where $\theta$ is the neural network parameter used to fit the value of the action, and $r_t$ is the value of the immediate reward that the smart car receives by interacting with the environment. In PIDQN, we introduce the idea of PID control to improve the Q-value update in DQN, and the specific Q-value update formula is as follows:

$$Q_{\mathit{PIDQN}}(s_t,a_t;\theta )=Q(s_t,a_t;\theta )+P_t+I_t+D_t$$

(28)

where $P_t=K_p·e_t$ is the difference between the immediate reward and the estimated Q value. It is calculated as follows:

$$e_t=r_t+\gamma \underset{a_{t+1}}{max}Q(s_{t+1},a_{t+1};\theta )-Q(s_t,a_t;\theta )$$

(29)

A direct proportional response through the current error $e_t$ can help the algorithm respond quickly to changes in the environment. Integral control $I_t$ helps the algorithm to avoid decision instability due to short-term fluctuations and to ensure long-term decision consistency by accumulating past errors, which is calculated as follows:

$$I_t=K_I·\sum _{i=0}^t{e}_i$$

(30)

Differential control $D_t$ adjusts the Q value in advance to better cope with future path selection by predicting the trend of the error change, which is calculated as follows:

$$D_t=K_D·\left(e_t-e_{t-1}\right)$$

(31)

where $K_p,K_I$, and $P_D$ are the proportional, integral, and differential coefficients of the PID controller for the speed of response, cumulative effect, and future trend prediction of the regulation error, respectively. Ultimately, the Q-value update formula for the PIDQN algorithm synthesizes the three parts of the PID control into the following formula:

$$Q_{\mathit{PIDQN}}(s_t,a_t;\theta )=Q(s_t,a_t;\theta )+K_P·e_t+K_I·\sum _{i=0}^t{e}_i+K_D·(e_t-e_{t-1})$$

(32)

Neural network parameter $\theta$ uses value iteration for the parameter update:

$$\theta \leftarrow \theta +\alpha [r_t+\gamma ·Q_{\mathit{PIDQN}}(s_{t+1},a_{t+1};\theta )-Q(s_t,a_t;{\theta }^{-})]{\nabla }_{\theta }Q(s_t,a_t;{\theta }^{-})$$

(33)

The policy iteration uses gradient descent to obtain the optimal policy ${\pi }^{*}$ that minimizes $F_2$, and the policy selection uses a dynamically decreasing greedy policy:

$$\begin{array}{c}{a}_t=\left\{\begin{array}{cc}\underset{a}{max}{Q}_{\mathit{PIDQN}}(s_t,a_t;\theta )\hfill & {\chi }_t<\epsilon \hfill \\ A\left\{a_1,a_2...\right\}\hfill & other\hfill \end{array}\right.\end{array}$$

(34)

$$\begin{array}{c}{\chi }_t={\chi }_{min}+({\chi }_{max}-{\chi }_{min})e^{-kt}\end{array}$$

(35)

where ${\chi }_t$ is the exploration rate that decreases step by step, the decay rate k determines how fast the rate of exploration decreases. Figure 2 and Algorithm 2 demonstrate the algorithm-specific iterative flow of the PIDQN. An intelligent vehicle begins interacting with its environment from an initial state, which includes a road network composed of various road segments and lanes. For each vehicle position, the PID controller provides proportional (P), integral (I), and derivative (D) feedback to assist the vehicle in making immediate adjustments to its actions. The PIDQN optimizes the vehicle’s decision-making process by integrating the feedback from the PID controller with the Q-value update rules of the DQN. This combination enables more accurate and dynamic decision-making, improving the vehicle’s ability to adapt to its environment.

Algorithm 2 PIDQN algorithm

Initialize: neural network weights $\theta$ and target network weights ${\theta }^{\prime }$ Initialize: Initialize experience replay buffer D Initialize: Initialize PID controller parameters $K_P,K_I,K_D$ Initialize: Set initial exploration rate ${\chi }_{m}ax$ 1: for episode in range(episodes) do 2:     Initialize state s 3:     Initialize PID terms: $\mathrm{integral}=0,\mathrm{previous}\_\mathrm{error}=0$ 4:     for $t=1$ to MaxSteps do 5:         if $\mathrm{random}\left(\right)<\chi$ then 6:            Choose random action a 7:         else 8:            Choose action $a=max\left(Q\right(s,a;\theta \left)\right)$ 9:         end if 10:         Execute action a 11:         Observe next state $s_{t+1}$ and reward $r_t$ 12:         Store $(s_t,a_t,r_t,s_{t+1})$ in D 13:         Sample a random batch of experiences $(s_j,a_j,r_j,s_j^{\prime })$ from D 14:         for each experience $(s_j,a_j,r_j,s_j^{\prime })$ do 15:            if $s_j^{\prime }$ is terminal state then then 16:                $\mathrm{target}=r_j$ 17:            else 18:                $\mathrm{target}=r_j+{\gamma }^{*}max\left(Q(s_j^{\prime },a^{\prime };{\theta }^{\prime })\right)$ 19:            end if 20:            $\mathrm{error}=\mathrm{target}-Q(s_j,a_j;\theta )$ 21:            $\mathrm{proportional}=K_P\ast \mathrm{error}$ 22:            $\mathrm{integral}+=K_I\ast \mathrm{error}$ 23:            $\mathrm{derivative}=K_D\ast (\mathrm{error}-\mathrm{previous}\_\mathrm{error})$ 24:            adjustment = proportional + integral + derivative 25:            adjusted_target = $Q(s_j,a_j;\theta )$ + adjustment 26:            previous_error = error 27:         end for 28:         Update Q-network weights $\theta$ to minimize loss 29:         Update $\chi$ 30:     end for 31: end for

4. Experiments and Results

4.1. Experimental Setup

Introduction: In our study, we processed the road network by collecting maps of specific open test sections in our country. The path planning method proposed in this paper was validated by simulation based on the collected maps. Figure 3 shows the labeled urban roads with 600 road nodes and a total length of about 25.65 km.

Parameter Setting: the parameters of the two-layer planning model based on the design of the urban road environment in this paper are shown in

Table 1. Parameters of the two-tier planning model.

Parameter	Value
$\alpha$	0.0001
$\beta$	0.001
$\tau$	1
$\delta$	0.25
$\mu$	1
$\eta$	0.2
$\phi$	1
${\rho }_1$	0.008
${\rho }_2$	0.002

, and

Table 2. Parameters of the PIDQN algorithm.

Parameter	Value
$\gamma$	0.99
$K_P$	0.1
$K_I$	0.002
$K_D$	0.5
${\chi }_{min}$	0.01
${\chi }_{max}$	0.99

demonstrates the specific parameter settings of the PIDQN algorithm.

Test scenarios: To validate the accuracy of the algorithm, we utilized the urban road depicted in Figure 3. The performance of the algorithm was compared with two reinforcement learning algorithms: DQN and Q-Learning. Multiple path comparisons were performed with the A* algorithm as a benchmark for path comparisons. All experiments were conducted on a Lenovo Legion Y9000P laptop equipped with a CPU i9-13900HX, 16 GB RAM, a 1 TB SSD, and an RTX4060. The operating system used was Ubuntu 18.04, and the program was implemented in Python (version 3.13.1).

4.2. Experiment on Urban Roads

To validate our planning algorithm, we selected two high-traffic areas featuring schools and hospitals. In

Table 3. Latitude and longitude coordinates of the origin and destination.

Start Position	End Position
$(31.423$ N, 120.633 E)	$(31.419$ N, 120.638 E)
$(31.421$ N, 120.639 E)	$(31.419$ N, 120.637 E)
$(31.419$ N, 120.633 E)	$(31.424$ N, 120.634 E)
$(31.420$ N, 120.641 E)	$(31.425$ N, 120.643 E)

, we show the four sets of starting point locations that were selected for lane-level planning and path comparison based on the A* algorithm for urban roads, respectively.

In Figure 4a–h, we visualize the four sets of planned paths. As can be seen from the figure, our algorithm integrated lane-changing risk, road scalability, road length, and comfort. PIDQN favored straight lanes to reduce the number of lane changes and the risk of changing lanes when planning the paths, and since our algorithm took road scalability into account, we advanced the lane-changing process on the road before the end point instead of choosing to always change lanes at the road’s end point. The A* algorithm only considered the intrinsic length cost of the path and the end-point inspired distance so it favored the continuous lane-changing strategy to arrive at the lane closest to the end point first. This strategy is very risky when driving on urban roads, so we should try to avoid continuous lane changing in the path planning. Obviously, under the incentive of our designed reward function, PIDQN chose a more reasonable lane-changing decision for path planning.

Figure 5a–d show the variation in the reward value (Reward) for three reinforcement learning path planning algorithms (PIDQN, DQN, and Q-learning) over 3000 training cycles. The number of training cycles (Episode) is represented by the horizontal axis and the reward value (Value) by the vertical axis, and the performance of the PIDQN, DQN, and Q-learning algorithms are represented by blue, orange, and green curves, respectively.

As can be observed from Figure 5a–d, the PIDQN algorithm compared favorably with DQN and Q-learning in terms of both convergence speed and final reward value. Specifically, the PIDQN algorithm showed a significant reward boost after about 500 cycles, rapidly increasing the reward level of the corresponding value and converging at close to 2500 cycles. This shows that PIDQN has a clear advantage in terms of convergence speed and strategy optimization. In contrast, the DQN algorithm converged slowly but showed a steady increase after about 1000 cycles, eventually converging to a relatively stable reward value after 2500 cycles. The Q-learning algorithm performed poorly, with a large increase in reward value during training and a significantly slower convergence than the first two algorithms due to the Q-table fitting approach.

In the two-layer planning model, the PIDQN algorithm achieved more robust and efficient path planning through the combination of global path planning and local path optimization. The figure shows the number of planning successes of the three algorithms PIDQN, DQN, and Q-learning in four experiments, which demonstrate the superiority of PIDQN. In Figure 6, the number of path planning successes of PIDQN is significantly higher than that of DQN and Q-learning, especially in the first and second experiments, where the number of successes of PIDQN is close to 2500, which is far more than the 2000 successes of DQN and 1500 successes of Q-learning. This indicates that PIDQN can better cope with complex urban roads in global path planning and local optimization, ensuring that the vehicle completes the path planning task more efficiently.

PIDQN was able to better balance the relationship between exploration and utilization by introducing the PID control mechanism. The coordinated work of the proportional, integral, and differential parts of the PID control made PIDQN not only able to respond quickly to the environmental changes in the path planning process but also able to avoid converging to the local optimal solution prematurely, which enhanced the ability to explore the globally optimal path. This was also verified by the performance of the number of successes in Figure 6. PIDQN consistently maintained a high success rate under different experimental conditions.

Table 4. Performance comparison.

Algorithm	Length	Curvature	Lane-Changing Frequency	Extended Entropy
${\mathrm{A}}^{*}$	2526.45	8.556	4	9.667
	2411.24	10.432	3	10.812
	3411.25	16.215	2	8.117
	2251.64	9.826	4	11.102
${\mathrm{A}}^{*}+\mathrm{PIDQN}$	2397.64	7.981	2	11.208
	2378.14	8.656	2	13.208
	3407.75	15.565	1	12.117
	2251.64	7.216	3	13.012

presents a performance comparison between the A* algorithm and the A* + PIDQN algorithm across four key metrics: path length (length), curvature, lane-change frequency (Lane change frequency), and extended entropy. The results highlight the relative strengths and limitations of each approach in addressing the challenges of autonomous path planning.

Path length: The A* + PIDQN algorithm demonstrated a reduction in path length across all cases compared to A*. For instance, in the shortest scenario, A* + PIDQN achieved 2251.64 units, while A* required 2526.45 units. This reduction indicates that the incorporation of PIDQN facilitates more efficient pathfinding.

Curvature: In terms of curvature, A* + PIDQN consistently achieved lower values, suggesting smoother paths. For example, the highest curvature value in A* + PIDQN was 15.565, whereas A* reached 16.215.

Lane-change frequency: The frequency of lane changes was reduced in most cases when using A* + PIDQN, demonstrating the algorithm’s ability to enhance driving comfort by minimizing unnecessary maneuvers. The maximum frequency was three for A* + PIDQN, compared to four for A*.

Extended entropy: The extended entropy metric, which reflects the adaptability and diversity of the solution, was higher for A* + PIDQN in all scenarios. For example, the highest entropy achieved by A* + PIDQN was 13.208, compared to 11.102 for A*.

Overall, the results showed that A* + PIDQN performed better in terms of path efficiency, smoothness, driving comfort, and adaptability compared to the standard A* algorithm. This suggests that a two-layer path planning algorithm that integrates the deep reinforcement learning algorithm PIDQN into the classical path planning algorithm has the potential to improve path planning performance.

The parameter sensitivity analysis in Figure 7 illustrates that the PIDQN algorithm achieved robust performance across a wide range of PID parameter configurations (Kp, Ki, Kd). Although these parameters influenced the convergence speed, stability, and initial reward performance, all tested configurations ultimately converged to a similar reward level (approximately 90–100) after 3000 episodes. This demonstrates that PIDQN exhibits strong adaptability to varying parameter settings and does not require extensive parameter tuning to achieve optimal results.

Specifically, the proportional term (Kp) significantly affected the initial learning phase, with higher Kp values (Kp = 10) accelerating early reward growth. The derivative term (Kd) primarily influenced the stability of the training process, where lower Kd values (Kd = 0.01) yielded smoother convergence curves. The integral term (Ki) had a comparatively minor impact, contributing to incremental improvements over time. Despite these variations, the final reward performance remained consistent, highlighting the algorithm’s robustness.

5. Conclusions and Discussion

In this paper, we proposed a two-layer planning algorithm based on an urban road environment, which improved the extended cost function of the A* algorithm based on the model at the road layer, while at the lane layer it was based on a path planning algorithm combining Proportional–Integral–Differential (PID) control and deep reinforcement learning (DQN), the PIDQN algorithm. Through the two-layer planning model, the global planning layer generated the optimal road sequences in a wide range, and the local planning layer further performed lane layer planning based on multiple factors such as lane changing, comfort, and expandability. On this basis, the PIDQN algorithm utilized the proportional, integral, and differential mechanisms of the PID controller to finely regulate the estimation of the value function in the deep Q-learning process, which effectively improved the algorithm’s environment exploration capability and convergence speed and avoided the local optimality problem caused by overestimation in the traditional DQN algorithm.

Through multiple experiments using high-precision maps of real urban environments in a simulation environment, the two-layer planning algorithm showed significant advantages over the traditional A-algorithm. Specifically, in the road layer, we improved the path generation of the A-algorithm, so that it could not only effectively consider the path length but also comprehensively incorporate the geometric characteristics of the road, which provided a more reasonable initial path selection for lane layer planning. At the lane layer, the PIDQN algorithm, which combined proportional–integral–derivative (PID) control with deep reinforcement learning (DQN), further optimized the path planning, taking into full consideration a variety of factors, such as lane transformation, driving comfort, and scalability.

In addition, we conducted comparative experiments on different reinforcement learning algorithms, and the results showed that the PIDQN algorithm exhibited significant advantages in terms of success rate, convergence speed, and path smoothness. This was mainly due to the introduction of the PID control mechanism, which effectively enhanced the algorithm’s ability to explore the environment while improving the stability of the training process. Compared with traditional reinforcement learning algorithms, PIDQN is more suitable for dynamic and complex urban environments by finely regulating the estimation of the value function and avoiding the local optimization problem caused by overestimation in DQN. Through the combination of a two-layer planning framework, our algorithm achieved a good balance between road-level path rationality and lane-level path fine optimization and could better meet the path planning needs of intelligent vehicles in real urban environments.

Although the two-layer planning algorithm in this study demonstrated significant advantages, there are still many areas that deserve further research and improvement. Considering the computational complexity of the PIDQN algorithm, a future study could further explore how to improve the operation efficiency of the algorithm on low-resource devices, such as hardware acceleration optimization using GPUs or FPGAs, to ensure real-time performance in practical applications.