Dynamic Path Planning for Vehicles Based on Causal State-Masking Deep Reinforcement Learning

Abstract

Dynamic path planning enables vehicles to autonomously navigate in unknown or continuously changing environments, thereby reducing reliance on fixed maps. Deep reinforcement learning (DRL), with its superior performance in handling high-dimensional state spaces and complex dynamic environments, has been widely applied to dynamic path planning. Traditional DRL methods are prone to capturing unnecessary noise information and irrelevant features during the training process, leading to instability and decreased adaptability of models in complex dynamic environments. To address this challenge, we propose a dynamic path-planning method based on our Causal State-Masking Twin-delayed Deep Deterministic Policy Gradient (CSM-TD3) algorithm. CSM-TD3 integrates a causal inference mechanism by introducing dynamic state masks and intervention mechanisms, allowing the policy network to focus on genuine causal features for decision optimization and thereby enhancing the convergence speed and generalization capabilities of the agent. Furthermore, causal state-masking DRL allows the system to learn the optimal mask configurations through backpropagation, enabling the model to adaptively adjust the causal features of interest. Extensive experimental results demonstrate that this method significantly enhances the convergence of the TD3 algorithm and effectively improves its performance in dynamic path planning.

1. Introduction

Path-planning technology aims to generate safe and efficient routes for vehicles, enabling them to reach target locations without collisions. Achieving optimal paths is crucial for ensuring that vehicles complete tasks accurately and swiftly [1,2,3]. Path planning includes two mainstream strategies: global and reactive planning. Global planning adopts an offline approach utilizing predefined environmental maps, typically represented by grid or graph structures, and determines the most effective path through search algorithms such as breadth-first search [4] and rapidly exploring random trees (RRTs) [5]. These methods are referred to as global planning because they consider the entire environment when planning the path, formulating a comprehensive route from the starting point to the destination.

Reactive planning, on the other hand, is applied in online scenarios, dynamically navigating by continuously processing real-time data from sensors, emphasizing local path planning, and focusing on immediate obstacle avoidance and adaptation to environmental changes. Common reactive planning methods include the vector field histogram (VFH) [6], dynamic window approach (DWA) [7], simultaneous localization and mapping [8], and model predictive control [9]. These techniques enable vehicles to make rapid decisions based on current sensor inputs, achieving excellent performance in unpredictable and dynamic environments.

In recent years, machine learning has played a significant role in global and reactive path planning, substantially enhancing the intelligence level of vehicle navigation [10,11,12]. In addition to traditional supervised learning algorithms, such as support vector machines (SVMs) [13] and decision trees, used for predicting optimal paths based on labeled data, reinforcement learning (RL) [14,15], a key branch of machine learning, is rapidly becoming a research hotspot in the field of path planning. RL autonomously optimizes decision-making strategies by interacting with the environment and utilizing reward mechanisms, enabling vehicles to achieve adaptive navigation in dynamic and complex environments. Common RL algorithms include Q-learning, deep Q-networks (DQNs) [16], policy gradient methods, and proximal policy optimization [17]. These methods not only improve the efficiency and flexibility of path planning but also reduce the dependence on large amounts of labeled data. Deep RL (DRL) [18], because of its excellent performance in handling high-dimensional state spaces and complex decision-making tasks, has become a valuable method in the field of motion planning.

However, traditional DRL methods are prone to capturing task-irrelevant features during the training process. These spurious correlations often arise from environmental noise, sensor errors, or dynamic obstacles, leading to a significant decline in the model’s generalization ability and robustness in new environments. To address this challenge in path planning, this study proposes an innovative Causal State-Masking Twin-Delayed Deep Deterministic Policy Gradient (CSM-TD3) method. This method, based on the twin-delayed deep deterministic (TD3) policy gradient algorithm, integrates a causal inference mechanism by introducing state masks and intervention mechanisms to effectively filter out features irrelevant to decision making. CSM-TD3 generates dynamic state masks to ensure that the policy network relies solely on state features with genuine causal relationships.

The main contributions of this study are summarized as follows:

• Enhanced model robustness and transferability: By combining the causal consistency loss term and dynamic state masks, the policy network is compelled to rely solely on state features with genuine causal relationships for decision making. This addresses the issue of DRL models capturing spurious correlations during training.
• Improved convergence and generalization capabilities of DRL in dynamic path planning: Introducing encoding configurations optimized through backpropagation learning enables the model to adaptively adjust the causal features of each state dimension.
• Algorithm validation: The effectiveness of CSM-TD3 was validated through multiple simulation experiments. The experimental results demonstrate that this method performs excellently in terms of convergence and generalization capabilities in dynamic path planning.

The remainder of this paper is structured as follows. The related work is introduced in Section 2. The problem formulation is presented in Section 3. The implementation of the improved CSM-TD3 is elaborated in Section 4. The simulation results are compared in Section 5. The conclusions are summarized in Section 6.

2. Related Work

In the field of path planning, with the continuous advancement of automation technologies, particularly in autonomous driving and robotics, researchers have proposed various path-planning methods. These methods can primarily be categorized into traditional path-planning approaches and reinforcement learning (RL)-based methods. Traditional path-planning techniques, such as the A* algorithm and RRT, exhibit certain advantages in specific environments. On the other hand, RL-based path-planning methods demonstrate enhanced flexibility and adaptability, enabling online learning and adaptation in dynamic environments. Both of these approaches have been extensively studied in recent years.

2.1. Traditional Path-Planning Methods

Numerous studies on path planning have focused on optimizing classical path-planning algorithms to address increasingly complex environments and specific application requirements. For autonomous-vehicle path planning, Lin et al. [19] proposed an improved A* algorithm that incorporates redundant safety spaces and predictive planning, generating shorter and smoother paths to enhance collision avoidance capabilities. In terms of hybrid methods, Sun et al.’s study combined the enhanced A* algorithm with the DWA, utilizing bidirectional search, adaptive heuristics, and key node filtering to shorten path lengths and improve algorithm reliability [20]. Chen et al. [21] improved the Lifelong Planning A* (LPA*) algorithm by introducing a path priority partitioning mechanism, dividing paths into high- and low-priority categories to maintain the stability of high-priority paths. This method effectively avoids path conflicts by weighting high-priority path nodes, thereby enhancing the efficiency and stability of multi-path planning.

Li et al., in a study on autonomous driving [22], integrated environmental and vehicle constraints to propose an improved RRT-Connect algorithm. By designing appropriate path cost functions and heuristic guidance strategies, the method significantly reduced path lengths and planning times, expanding the path-planning capabilities of intelligent vehicles in complex environments. For mobile vehicles in complex unknown environments, Shi et al. [23] optimized the RRT-Connect algorithm through goal-biased strategies and node selection, improving path smoothness and success rates.

In dynamic environments, Lee et al. [24] combined Artificial Potential Fields (APFs) with RRT methods, rapidly generating collision-free local paths through weighted obstacle configurations, yielding improved replanning efficiency. In other developments, the virtual potential field combined with RRT (VPF-RRT) algorithm [25] and the convex decomposition topology-based RRT (CDT-RRT) algorithm [26] were designed to optimize the path smoothness and the performance in exploring the shortest paths, respectively. For robotic-arm path planning, Wen et al. [27] proposed the PS-RRT algorithm based on partitioned dynamic sampling and node rejection mechanisms, effectively reducing the number of invalid nodes and achieving efficient path planning.

Traditional path-planning methods largely rely on predefined environmental maps, making it difficult to cope with highly dynamic or unknown environmental changes, limiting their real-time application capabilities. Many optimization algorithms exhibit high computational complexity in complex environments, affecting the real-time performance and efficiency of path planning. These methods typically focus on the geometric optimization of paths, such as optimizing path length and smoothness, while neglecting the causal relationships and environmental interference factors in the path-planning process, resulting in insufficient robustness in dynamic environments.

2.2. RL-Based Path-Planning Methods

With the rapid development of deep learning and RL, RL-based path-planning methods have become a research hotspot because of their adaptability and online learning capabilities in dynamic environments. Many researchers have explored how to leverage RL to enhance the effectiveness of path planning. Tan et al. [28] proposed a complete coverage path-planning algorithm based on Q-learning, which achieved 100% coverage by introducing global environmental information and optimizing path strategies near obstacles while significantly reducing path redundancy. For unplugged ground platforms, Zhang et al. [29] improved the Q-learning algorithm to enhance path-planning flexibility and convergence speed, making it suitable for autonomous driving functions. Similarly, Zhang et al. [30] combined deep Q-Learning with path-smoothing techniques, demonstrating their model’s effectiveness in complex environments.

Kong et al. [31] proposed an APF path-planning strategy based on the partially observable Markov decision process (POMDP) model, which integrates sensor information to predict obstacle positions, enabling real-time path planning for uncrewed aerial vehicles in complex environments. In further research [32], the path generation method based on the deep Markov model (RL-PG) could ensure path-tracking safety through a motion fine-tuning module, exhibiting a high success rate. In research on drone path planning [33], the authors combined the A* algorithm with velocity obstacles in an RL approach, improving planning efficiency in dynamic environments. In addition, Jaramillo et al. [34] designed a reward function strategy to encourage agents to choose the shortest paths with fewer turns, thereby optimizing the number of turns and total distance.

For time-constrained path planning in complex environments, Wang et al. [35] integrated prioritized experience replay and improved exploration mechanisms in an off-policy RL method, enhancing sampling efficiency and rescue target achievement rates. Hua et al. [36] introduced the PIOPGRL algorithm based on the pigeon-inspired optimization algorithm, fusing it with RL to optimize spacecraft path planning under multiple attitude constraints. Finally, Feng et al. [37] proposed a path obstacle-avoidance control method based on the deep deterministic policy gradient that maps perception to control through deep neural networks, improving obstacle-avoidance performance.

RL methods are prone to capturing task-irrelevant noise and features in high-dimensional state spaces, leading to unstable policies and decreased generalization capabilities. Moreover, these methods typically require large amounts of training data and computational resources, resulting in lengthy training processes that hinder real-time applications. Additionally, many RL algorithms lack effective causal inference mechanisms when facing environmental disturbances and dynamic changes, affecting the model’s convergence in complex environments and its ability to be generalized to new environments.

3. Problem Formulation

To effectively address the problem of autonomous path planning for vehicles in complex dynamic environments, this study first needed to precisely model the vehicle’s kinematic and dynamic characteristics. This process required not only a comprehensive understanding of the vehicle’s motion behavior in different coordinate systems but also the integration of causal relationship models with RL frameworks to construct a suitable mathematical model for solving the problem. This section details the mathematical formalization process for the problem. Firstly, by establishing a vehicle dynamics model, we can describe the motion characteristics of the vehicle within a two-dimensional plane and clarify the specific description of the dynamic path-planning problem. Subsequently, we formalize the problem as a Markov decision process (MDP), laying the theoretical foundation for the subsequent causal RL algorithms. Finally, we construct a structural equation model (SEM) to identify key state features that directly impact vehicle decision making, introducing the concept of a state mask to enhance the interpretability of the model and the effectiveness of decision-making.

3.1. Dynamic Path-Planning Model

In dynamic path planning, accurately describing the vehicle’s dynamic characteristics is crucial for achieving precise control and path optimization. We model the vehicle as a planar rigid-body dynamic model moving within a two-dimensional plane, as represented in Figure 1. The dynamic equations of the vehicle in the body coordinate system are as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill M({\dot{v}}_x-v_y\omega )& =F_x\hfill \\ \hfill M({\dot{v}}_y+v_x\omega )& =F_y\hfill \\ \hfill I_{\gamma }\dot{\omega }& =M_z\hfill \end{array}\right.\end{array}$$

(1)

where M is the mass of the vehicle; $I_{\gamma }$ is the moment of inertia of the vehicle about the centroid perpendicular to the plane; ${\dot{v}}_x$ and ${\dot{v}}_y$ are the longitudinal and lateral accelerations, respectively; $\dot{\omega }$ is the angular acceleration of the vehicle; and $F_x$, $F_y$, and $M_z$ are the longitudinal force, lateral force, and yaw moment of the vehicle, respectively. To represent the vehicle’s motion state in the global coordinate system, we establish a kinematic model describing the position and heading-angle changes in that system:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill \dot{x}& =v_x\cos \psi -v_y\sin \psi \hfill \\ \hfill \dot{y}& =v_x\sin \psi +v_y\cos \psi \hfill \\ \hfill \dot{\psi }& =\omega \hfill \end{array}\right.\end{array}$$

(2)

where $\dot{x}$ and $\dot{y}$ are the longitudinal and lateral velocities of the vehicle in the global coordinate system, respectively, and $\psi$ is the vehicle’s heading angle. By integrating the vehicle’s dynamic and kinematic Equations (1) and (2), respectively, we can comprehensively consider the vehicle’s motion in its coordinate system and the global coordinate system, as well as the coupling effects of rotational motion on lateral and longitudinal movements. In the design of dynamic path planning for vehicles, the state variables are defined as

$$\begin{array}{c}\hfill {\mathbf{s}}_v=\left[\begin{array}{cccccc}x& y& \psi & v_x& v_y& \omega \end{array}\right],\end{array}$$

(3)

and the control variables are defined as

$$\begin{array}{c}\hfill \mathbf{u}=\left[\begin{array}{ccc}{F}_x& F_y& M_z\end{array}\right].\end{array}$$

(4)

Based on the established vehicle dynamics model, we describe the dynamic path-planning problem as follows: given the vehicle’s initial state ${\mathbf{s}}_0$ and the target position ${\mathbf{s}}_g=[x_g,y_g]$, a control policy ${\pi }_{\theta }$ that enables the vehicle to move from the starting position to the target position while satisfying dynamic and environmental constraints needs to be learned. During motion, the vehicle must avoid dynamic obstacles in the environment while optimizing path length and smoothness to achieve efficient and safe path planning. To formalize the dynamic path-planning problem as an optimization problem, we aim to minimize the cumulative cost from the start point to the target point. The specific optimization objective function is defined as

$$\begin{array}{c}\hfill \underset{{\pi }_{\theta }}{min}J=\sum _{t=0}^T∆t\left[\alpha C_{\mathrm{pat}}\left(t\right)+\beta C_{\mathrm{smo}}\left(t\right)+\lambda C_{\mathrm{col}}\left(t\right)\right]\end{array}$$

(5)

where $∆t$ is the control time step; the coefficients $\alpha$, $\beta$, and $\lambda$ are weight factors used to balance the importance of each cost component; $C_{\mathrm{pat}}\left(t\right)$ represents the path length cost, encouraging the vehicle to choose shorter paths and reducing the travel distance and time; $C_{\mathrm{smo}}\left(t\right)$ denotes the path smoothness cost, encouraging the vehicle to maintain a smoother motion process and reducing motion shocks; and $C_{\mathrm{col}}\left(t\right)$ signifies the collision risk cost, penalizing the vehicle’s proximity to obstacles to ensure the safety of the path.

We define the path length cost $C_{\mathrm{pat}}\left(t\right)$ as the Euclidean distance between the current position and the target position:

$$\begin{array}{c}\hfill C_{\mathrm{pat}}\left(t\right)=\sqrt{{(x\left(t\right)-x_{\mathrm{g}})}^2+{(y\left(t\right)-y_{\mathrm{g}})}^2}\end{array}$$

(6)

where $x\left(t\right)$ and $y\left(t\right)$ represent the vehicle’s longitudinal and lateral positions in the global coordinate system at the current time step, respectively, and $x_{\mathrm{g}}$ and $y_{\mathrm{g}}$ denote the longitudinal and lateral positions of the endpoint in the global coordinate system, respectively.

Because the variations in control variables at adjacent time steps reflect the variation in the vehicle’s state at those times, we define the path smoothness cost $C_{\mathrm{smo}}\left(t\right)$ as related to the variations in control variables:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & C_{\mathrm{smo}}\left(t\right)={\left(F_x\left(t\right)-F_x(t-1)\right)}^2+{\left(F_y\left(t\right)-F_y(t-1)\right)}^2+{\left(M_z\left(t\right)-M_z(t-1)\right)}^2\hfill \end{array}\end{array}$$

(7)

where $F_x\left(t\right)$, $F_y\left(t\right)$, and $M_z\left(t\right)$ represent the vehicle’s current longitudinal force, lateral force, and yaw moment, respectively.

The collision risk cost is a penalty for the vehicle’s proximity to dynamic obstacles in the environment. To quantify this risk, we employ the following formula:

$$\begin{array}{c}\hfill C_{\mathrm{col}}\left(t\right)={\displaystyle \frac{1}{d_{\min }\left(t\right)+ϵ}}\end{array}$$

(8)

where $ϵ$ is a small constant to avoid division by 0 and $d_{\min }\left(t\right)=\sqrt{{(x\left(t\right)-x_{\min })}^2+{(y\left(t\right)-y_{\min })}^2}$ represents the Euclidean distance between the vehicle’s current position and the nearest obstacle’s position, where $x_{\min }$ and $y_{\min }$ denote the longitudinal and lateral positions of the nearest obstacle in the global coordinate system, respectively. When the vehicle approaches an obstacle, the value of this cost function increases significantly, driving the vehicle away from obstacles to establish a safer path.

3.2. Formalization of the Markov Decision Process

Based on the definitions in Section 3.1, we formalize the vehicle’s actual control and dynamic path-planning optimization problem as an MDP to enable effective control decision making and optimization using DRL algorithms [38]. An MDP is a mathematical framework widely used for modeling decision-making problems, especially in the presence of uncertainty and dynamic environments. Within this framework, as represented in Figure 2, the agent’s actions are defined to be the same as the vehicle’s control variables, i.e., $\mathbf{a}=\mathbf{u}$. The state is defined to include both the vehicle’s current state and the environmental state, specifically represented as

$$\begin{array}{c}\hfill \mathbf{s}=\left[\begin{array}{ccc}{\mathbf{s}}_v& d_{\min }& {\mathbf{s}}_g\end{array}\right]=\left[\begin{array}{ccccccccc}x& y& \psi & v_x& v_y& \omega & d_{\min }& x_{\mathrm{g}}& y_{\mathrm{g}}\end{array}\right]\end{array}$$

(9)

To ensure that the agent’s actions are executable and can maintain the vehicle’s stability, we constrain the vehicle’s control inputs within predefined ranges. These constraints not only reflect the vehicle’s physical limitations but also help prevent excessive control inputs that could lead to vehicle instability or loss of control. The specific constraints are as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill F_{x,min}& \le F_x\left(t\right)\le F_{x,max}\hfill \\ \hfill F_{y,min}& \le F_y\left(t\right)\le F_{y,max}\hfill \\ \hfill M_{z,min}& \le M_z\left(t\right)\le M_{z,max}\hfill \end{array}\right.\end{array}$$

(10)

These limitations ensure that the control inputs remain within a safe and executable range for the vehicle, avoiding unnecessary risks caused by excessively large or small forces and moments.

For the vehicle to avoid collisions with any obstacles throughout the path-planning process, we introduce the concept of safety distance, in which the distance $d_i$ between the vehicle and all obstacles must be greater than a preset safety distance $d_s$. This constraint can be expressed as

$$\begin{array}{c}\hfill d_i\left(t\right)=\sqrt{{(x\left(t\right)-x_i\left(t\right))}^2+{(y\left(t\right)-y_i\left(t\right))}^2}\ge d_s\end{array}$$

(11)

where i denotes the number of obstacles and $x_i\left(t\right)$ and $y_i\left(t\right)$ represent the longitudinal and lateral positions, respectively, of the ith obstacle in the global coordinate system at time t. By mandating that the vehicle maintain a distance of at least $d_s$ from all obstacles, we can effectively avoid potential collision risks.

We define the condition for the vehicle to reach the endpoint as the distance between the vehicle’s current position, and the endpoint as less than a predefined distance $d_g$:

$$\begin{array}{c}\hfill d\left(t\right)=\sqrt{{(x\left(t\right)-x_g)}^2+{(y\left(t\right)-y_g)}^2}\le d_g\end{array}$$

(12)

This condition ensures that the vehicle can reach the target position during the path-planning process while allowing a certain tolerance to accommodate uncertainties and dynamic changes in the actual environment.

In summary, the optimization problem for dynamic path planning can be formalized as

$$\begin{array}{cc}\hfill & \begin{array}{cccc}\hfill & {\mathbf{a}}^{*}=\hfill & \hfill & \arg \underset{{\pi }_{\theta }}{\min }\sum _{t=0}^T∆t\left[\alpha C_{\mathrm{pat}}\left(t\right)+\beta C_{\mathrm{smo}}\left(t\right)+\lambda C_{\mathrm{col}}\left(t\right)\right]\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\hfill & \hfill & F_{x,min}\le F_x\left(t\right)\le F_{x,max}\hfill \\ \hfill & \hfill & & F_{y,min}\le F_y\left(t\right)\le F_{y,max}\hfill \\ \hfill & \hfill & & M_{z,min}\le M_z\left(t\right)\le M_{z,max}\hfill \\ \hfill & \hfill & & d_i\left(t\right)\ge d_s,\forall i,\forall t\hfill \\ \hfill & \hfill & & d\left(t\right)\le d_g\hfill \end{array}\hfill \end{array}$$

(13)

The objective of this optimization problem is to minimize the cumulative cost from the start point to the target point by designing an appropriate control policy ${\pi }_{\theta }$ while satisfying dynamic and environmental constraints. Solving this problem provides both theoretical foundations and practical guidance for the autonomous path planning of intelligent vehicles in complex dynamic environments.

To guide the agent’s action decisions and optimize the planned path toward the predefined target, we design the following reward function:

$$\begin{array}{cc}\hfill & r=\left\{\begin{array}{cc}-\left(K_{\alpha }cos{t}_{\mathrm{pat}}\left(t\right)+K_{\beta }cos{t}_{\mathrm{smo}}\left(t\right)+K_{\lambda }cos{t}_{\mathrm{col}}\left(t\right)\right)\hfill & d\left(t\right)\ge d_g,d_i\left(t\right)\ge d_s\hfill \\ K_a\hfill & d\left(t\right)\le d_g\hfill \\ -K_b\hfill & d_i\left(t\right)\le d_s\hfill \end{array}\right.\hfill \end{array}$$

(14)

where $K_{\alpha }$ is a positive gain parameter associated with the distance to the target point, $K_{\beta }$ is a negative gain parameter for action smoothness, $K_{\lambda }$ is a negative gain parameter associated with the distance to the nearest obstacle, $K_a$ is the reward term for reaching the target point, and $K_b$ is the penalty term for collision with obstacles. When the vehicle is in a normal driving state, i.e., the distance to the target point $d\left(t\right)$ is greater than the preset distance $d_g$ and the distance to all obstacles $d_i\left(t\right)$ is greater than the safety distance $d_s$, the reward function is the negative of the weighted sum of the respective costs. This reward function design encourages the agent to choose efficient, safe, and smooth paths by minimizing the path length, path smoothness, and collision risk. When the vehicle successfully reaches the target point, i.e., $d\left(t\right)\le d_g$, a positive reward $K_a$ is provided to encourage the agent to complete the path-planning task. When the vehicle’s distance to any obstacle satisfies $d_i\left(t\right)\le d_s$, a negative penalty $-K_b$ is imposed to strongly discourage the agent from selecting actions that lead to collisions, improving the safety of the path. Through this reward function design, the agent can progressively optimize its policy during the learning process, enabling efficient, safe, and smooth path planning in complex dynamic environments.

3.3. Causal Relationship Model

In complex dynamic environments, the state information perceived by the vehicle may contain a large amount of noise and many redundant features. These task-irrelevant features can lead to decreased model performance and weakened generalization capabilities. To address this, we introduce an SEM to identify key state features that have a direct causal impact on vehicle decision making and propose the concept of a state mask.

In the causal graph model, we use a directed acyclic graph (DAG) to represent the causal relationships between variables [39], as represented in Figure 3. Nodes represent states, actions, and rewards, while directed edges represent causal influences. We assume the following causal relationships:

(1) 

Relationships between state features. $s_i\to s_{i+1}$, indicating that feature $s_i$ has a direct causal influence on feature $s_{i+1}$.

(2) 

The direct influence of state features on control inputs. $s_i\to a_i$, indicating that feature $s_i$ has a direct causal influence on control input $a_i$.

(3) 

The influence of control inputs on the next state features. $a_i\to s_{i+1}$, indicating that control input $a_i$ influences the state feature $s_{i+1}$ at the next time step.

(4) 

States and control inputs jointly determine rewards. $\mathbf{s},\mathbf{a}\to r$.

Based on the above assumptions, we can define an SEM for each variable to provide a quantitative description of causal relationships:

$$\begin{array}{c}\hfill \left\{\begin{array}{cccc}\hfill s_i& =f_i(\mathbf{Pa}\left(s_i\right),n_i),\hfill & \hfill & i=1,2,\dots ,n\hfill \\ \hfill a_l& ={\pi }_{\theta }\left(\tilde{\mathbf{s}}\right),\hfill & \hfill & l=1,2,\dots ,m\hfill \\ \hfill r& =r(\mathbf{s},\mathbf{a})\hfill \end{array}\right.\end{array}$$

(15)

where $\mathbf{Pa}\left(s_i\right)$ denotes the set of parent nodes of $s_i$, i.e., the variables that directly influence $s_i$; $n_i$ represents independent noise terms, that is, unmodeled external influences $n_i\sim P\left(n_i\right)$; and $\tilde{\mathbf{s}}$ represents the set of state features after being masked. To identify the state features that have a direct causal influence on the control inputs $\mathbf{a}$, we partition the set of state features $\mathbf{s}$ into causal and non-causal feature sets. The causal feature set ${\mathbf{s}}^C$ represents the features that directly influence the control inputs. The non-causal feature set ${\mathbf{s}}^N$ represents the features that do not have a direct causal relationship with the control inputs, that is, $\mathbf{s}={\mathbf{s}}^C\cup {\mathbf{s}}^N$. In the causal graph, only the features in ${\mathbf{s}}^C$ point to the control inputs $\mathbf{a}$.

To leverage causal features in DRL, we introduce a state mask vector $\mathbf{m}=[m_1,m_2,$$\dots ,m_n{]}^T$, where $m_i\in 0,1$, which is used to indicate whether each feature is causal:

$$\begin{array}{cc}\hfill & m_i=\left\{\begin{array}{c}1s_i\in {\mathbf{s}}^C\hfill \\ 0s_i\in {\mathbf{s}}^N\hfill \end{array}\right.\hfill \end{array}$$

(16)

After applying the state mask, the masked state is represented as $\tilde{\mathbf{s}}=\mathbf{s}\odot \mathbf{m}$. In this manner, the policy function ${\pi }_{\theta }$ makes decisions solely based on the causal features ${\mathbf{s}}^C$, filtering out noise features irrelevant to decision making and reducing the impact of spurious correlations on the model. To ensure model consistency under intervention conditions, we introduce a causal consistency constraint. Specifically, when intervening on a causal feature $s_i\in {\mathbf{s}}^C$ by setting it to a specific value $\mathrm{do}(s_i={\widehat{s}}_i)$, the conditional distribution of the control input should satisfy

$$\begin{array}{c}\hfill P(\mathbf{a}\mid \mathrm{do}(s_i={\widehat{s}}_i),{\mathbf{s}}^{\setminus s_i})=P(\mathbf{a}\mid s_i={\widehat{s}}_i,{\mathbf{s}}^{\setminus s_i})\end{array}$$

(17)

where ${\mathbf{s}}^{\setminus s_i}$ denotes the set of state features excluding $s_i$. This consistency constraint helps the policy network learn the true causal relationships, preventing it from being misled by spurious correlations. To implement the causal consistency constraint during the RL training process, we introduce a causal consistency loss function $L_c\left(\theta \right)$ to quantify the consistency deviation of the model under intervention conditions:

$$\begin{array}{c}\hfill L_c\left(\theta \right)={\mathbb{E}}_{s_i\in {\mathbf{s}}^C}\left[{∥{\pi }_{\theta }\left(\tilde{\mathbf{s}}\right)-{\pi }_{\theta }\left({\tilde{\mathbf{s}}}_{do(s_i={\widehat{s}}_i)}\right)∥}^2\right]\end{array}$$

(18)

Upon minimizing the causal consistency loss $L_c\left(\theta \right)$, the policy network can better learn the true relationships between causal features and decision making.

4. Causal State-Masking TD3 Algorithm

To enhance the convergence and generalization capabilities of DRL in path planning within complex dynamic environments, we integrate a causal state-masking mechanism into the TD3 algorithm, resulting in the CSM-TD3 algorithm. The TD3 algorithm is an advanced deterministic policy gradient method in DRL that effectively mitigates the problem of overestimation bias and improves the convergence speed and stability of the algorithm through the mechanisms of dual-value networks and delayed policy updates [40]. Building upon TD3, CSM-TD3 enhances the model’s causal inference capabilities by introducing state masks and intervention mechanisms, enabling the agent to focus more on causal features that have the greatest impact on path-planning decisions, exhibiting greater efficiency and robustness in complex environments.

In the standard actor–critic architecture [41], the actor network is responsible for generating policies, while the critic network evaluates the value of policies. Based on this framework, we design and modify the critic and actor networks by incorporating causal inference theory into the TD3 algorithm. The following sections detail the design of the critic and actor networks, as well as the overall steps of the algorithm.

4.1. Critic Networks

The main function of the critic network is to evaluate the value of a given state and action, that is, to estimate the state–action value function $Q(\mathbf{s},\mathbf{a})$. The design of the critic network follows the standard TD3 algorithm structure. We utilize a twin-value network architecture, employing two independent critic networks $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$, represented by parameters ${\varphi }_1$ and ${\varphi }_2$, respectively [42]. The purpose of this is to reduce the bias in value estimation by using two independent estimates, enhancing algorithm performance. The inputs to the critic networks include the original state $\mathbf{s}$ and action $\mathbf{a}$, and the outputs are the corresponding Q-values $Q_{{\varphi }_j}(\mathbf{s},\mathbf{a})$, $j=1,2$. During the update process of the value networks, we define target value networks $Q_{{\varphi }_j^{\prime }}$, whose parameters are obtained from the main network parameters ${\varphi }_j$ through a soft update method. We define the loss function of the value network $L_{\mathrm{critic}}^j\left({\varphi }_j\right)$ as

$$L_{\mathrm{critic}}^j\left({\varphi }_j\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(Q_{{\varphi }_j}({\mathbf{s}}_i,{\mathbf{a}}_i)-y_i\right)}^2$$

(19)

where $y_i$ is the target Q-value:

$$y_i=r_i+\gamma \underset{j=1,2}{min}{Q}_{{\varphi }_j^{\prime }}({\mathbf{s}}_i^{\prime },{\tilde{\mathbf{a}}}_i^{\prime })$$

(20)

where $\gamma$ is the discount factor and ${\tilde{\mathbf{a}}}_i^{\prime }$ is the target action. The target Q-value $y_i$ is based on the Bellman equation, which combines the current immediate reward $r_i$ with the discounted cumulative future return, forming a robust learning signal. By taking the minimum value from the two target value networks, we further reduce the overestimation of Q-values.

The update of the value network aims to minimize the value estimation error so that the critic network evaluates the actor network’s policy accurately and reliably. To update the critic network’s parameters ${\varphi }_j$, we use gradient descent to minimize the loss function $L_{\mathrm{critic}}^j\left({\varphi }_j\right)$:

$${\varphi }_j\leftarrow {\varphi }_j-{\eta }_{\varphi }{\nabla }_{{\varphi }_j}{L}_{\mathrm{critic}}^j\left({\varphi }_j\right)$$

(21)

where ${\eta }_{\varphi }$ is the learning rate of the critic network, which determines the step size for each parameter update.

4.2. Actor Networks

The actor network is responsible for generating optimal actions based on the current state. The design of the policy network directly affects the agent’s decision-making capability. To enhance the model’s focus on key causal features and improve convergence and generalization capabilities in path-planning tasks, we introduce state masks and intervention mechanisms into the actor network. Specifically, a trainable state-mask vector $\mathbf{m}=[m_1,m_2,\cdots ,m_n]$, where $m_i\in [0,1]$, is added before the input layer of the actor network. The original state ${\mathbf{s}}_t$ is processed through the mask to obtain the masked state ${\tilde{\mathbf{s}}}_t$.

The state-mask vector $\mathbf{m}$ highlights causal features directly related to path-planning decisions while suppressing non-causal features and noise interference [43]. In complex dynamic environments, the state space typically contains a large number of features, only a subset of which directly influences path-planning decisions. By introducing state masks, the model can automatically identify and reinforce these key features while suppressing irrelevant or interfering features, enhancing the efficiency and stability of the policy. During the training process of the agent, the mask vector is treated as a trainable parameter and optimized through backpropagation. This means that the optimization of the mask vector is synchronized with the optimization of the policy network parameters, allowing the mask to adaptively adjust to accommodate different environments and task requirements.

To enable the actor network to better learn the true relationship between causal features and path-planning decisions, we introduce intervention mechanisms and causal consistency loss during the training process. The specific operations are as follows:

(1) 

Intervention feature selection. For each training sample, randomly select a causal feature $s_{t,k}$. This process simulates possible causal disturbances in real environments, forcing the model to learn robust policies under different causal contexts.

(2) 

Intervention value generation. Apply perturbations to the selected causal feature to generate an intervention value ${\widehat{s}}_{t,k}$:

$${\widehat{s}}_{t,k}=s_{t,k}+\delta ,\delta \sim \mathcal{N}(0,{\sigma }_{\mathrm{int}}^2)$$

(22)

where $\delta$ follows a Gaussian distribution with a mean of zero and variance ${\sigma }_{\mathrm{int}}^2$. This perturbation simulates unpredictable changes or noise in the environment, helping the model learn robustness when causal features change.

(3) 

Intervention state construction. Generate the intervened state ${\mathbf{s}}_{t,\mathrm{do}}$:

$${\mathbf{s}}_{t,\mathrm{do}}=[s_{t,1},\cdots ,s_{t,k-1},{\widehat{s}}_{t,k},s_{t,k+1},\cdots ,s_{t,n}]$$

(23)

Replacing the selected causal feature with the intervention value constructs a new state, reflecting the environmental state when the feature is disturbed.

(4) 

Intervention state masking. Apply mask processing to the intervened state to obtain ${\tilde{\mathbf{s}}}_{t,\mathrm{do}}={\mathbf{s}}_{t,\mathrm{do}}\odot \mathbf{m}$. Through state masks, adjust the intervened state features, allowing the model to focus on key causal features under intervention conditions.

(5) 

Policy network output calculation. Employ the policy network to generate actions ${\mathbf{a}}_t$ and ${\mathbf{a}}_{t,\mathrm{do}}$ on ${\tilde{\mathbf{s}}}_t$ and ${\tilde{\mathbf{s}}}_{t,\mathrm{do}}$, respectively. By comparing the actions generated under original and intervened states, the model can learn the true impact of causal feature changes on decisions.

(6) 

Causal consistency loss calculation. Compute the causal consistency loss $L_c\left(\theta \right)$. This loss term encourages the policy network to maintain consistency in action outputs when facing causal feature interventions, thereby reducing dependence on non-causal features and enhancing the model’s causal inference capabilities.

By minimizing $L_c\left(\theta \right)$, the policy network is compelled to respond appropriately to changes in causal features under intervention conditions, reducing dependence on non-causal features. Additionally, to encourage the sparsity of the mask vector, we introduce a mask sparsity regularization term $L_m\left(\mathbf{m}\right)$ in the loss function:

$$L_m\left(\mathbf{m}\right)={∥\mathbf{m}∥}_1=\sum _{i=1}^n\left|m_i\right|$$

(24)

L1 regularization helps drive the weights of unnecessary features in the mask vector toward zero, achieving sparsity in feature selection. This reduces the complexity of the model and further reinforces the focus on key causal features. Upon combining the above designs, the loss function of the policy network $L_{\mathrm{actor}}(\theta ,\mathbf{m})$ can be expressed as

$$L_{\mathrm{actor}}(\theta ,\mathbf{m})=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N{Q}_{{\varphi }_1}({\mathbf{s}}_i,{\pi }_{\theta }\left({\tilde{\mathbf{s}}}_i\right))+{\lambda }_c{L}_c\left(\theta \right)+{\lambda }_m{L}_m\left(\mathbf{m}\right)$$

(25)

where N is the batch size and ${\lambda }_c$ and ${\lambda }_m$ are hyperparameters used to balance the causal consistency loss and the mask sparsity regularization, respectively, so that the policy network considers both causal relationships and feature selection during the optimization process. By performing gradient descent on $L_{\mathrm{actor}}(\theta ,\mathbf{m})$, we update the policy network parameters $\theta$ and the mask vector $\mathbf{m}$:

$$\begin{array}{c}\hfill \theta \leftarrow \theta -{\eta }_{\theta }{\nabla }_{\theta }{L}_{\mathrm{actor}}(\theta ,\mathbf{m})\end{array}$$

(26)

$$\begin{array}{c}\hfill \mathbf{m}\leftarrow \mathbf{m}-{\eta }_m{\nabla }_{\mathbf{m}}{L}_{\mathrm{actor}}(\theta ,\mathbf{m})\end{array}$$

(27)

where ${\eta }_{\theta }$ and ${\eta }_m$ are the learning rates for the policy network and the mask vector, respectively. To ensure that $m_i\in [0,1]$, the updated mask vector is clipped so that each element of the mask vector remains within a reasonable range. This prevents negative values or values exceeding 1, resulting in the effectiveness and interpretability of the mask when weighting features.

$$m_i\leftarrow \mathrm{clip}(m_i,0,1)$$

(28)

With the above design, the actor network can generate high-quality policies and adaptively select and reinforce key causal features, enhancing the overall algorithm’s convergence and generalization capabilities in path planning within complex dynamic environments.

4.3. CSM-TD3 Architecture

Based on the above design of the critic and actor networks, the pseudo-code for the overall training and optimization steps of the CSM-TD3 algorithm is as described in Algorithm 1.

Algorithm 1 CSM-TD3

1: Initialize the critic networks $Q_{{\varphi }_1},Q_{{\varphi }_2}$ and the policy network ${\pi }_{\theta }$ with random parameters ${\varphi }_1,{\varphi }_2,\theta$; 2: Initialize the target networks $Q_{{\varphi }_1}^{\prime },Q_{{\varphi }_2}^{\prime },{\pi }_{\theta }^{\prime }$, setting their parameters to ${\varphi }_1^{\prime }\leftarrow {\varphi }_1$, ${\varphi }_2^{\prime }\leftarrow {\varphi }_2$, ${\theta }^{\prime }\leftarrow \theta$; 3: Initialize the state mask vector $\mathbf{m}$; 4: Initialize the experience-replay buffer Ω; 5: for each $episode=1$ to M do 6:       Initialize the random process $R_N$ for action exploration; 7:       Receive the initial observation state ${\mathbf{s}}_1$; 8:       for each time step $t=1$ to E do 9:             Compute the masked state ${\tilde{\mathbf{s}}}_t={\mathbf{s}}_t\odot \mathbf{m}$; 10:           Select action ${\mathbf{a}}_t={\pi }_{\theta }\left({\tilde{\mathbf{s}}}_t\right)+{\mathcal{N}}_t$, where ${\mathcal{N}}_t$ is the exploration noise; 11:           Execute action ${\mathbf{a}}_t$, observe reward $r_t$ and next state ${\mathbf{s}}_{t+1}$; 12:           Store transition $({\mathbf{s}}_t,{\mathbf{a}}_t,r_t,{\mathbf{s}}_{t+1})$ into the experience replay buffer Ω; 13:           Randomly sample a batch of samples ${\{{\mathbf{s}}_i,{\mathbf{a}}_i,r_i,{\mathbf{s}}_i^{\prime }\}}_{i=1}^N$ from the buffer; 14:           Compute the target Q-value $$y_i=r_i+\gamma \underset{j=1,2}{min}{Q}_{{\varphi }_j^{\prime }}({\mathbf{s}}_i^{\prime },{\tilde{\mathbf{a}}}_i^{\prime })$$ 15:           Update the critic network parameters ${\varphi }_j$ by minimizing the loss function: $$L_{\mathrm{critic}}^j\left({\varphi }_j\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(Q_{{\varphi }_j}({\mathbf{s}}_i,{\mathbf{a}}_i)-y_i\right)}^2,j=1,2$$ 16:           if $t \mathrm{mod} d_{\mathrm{delay}}==0$ then 17:              Randomly select a causal feature $s_{t,k}$; 18:              Generate an intervention value ${\widehat{s}}_{t,k}=s_{t,k}+\delta$, where $\delta \sim \mathcal{N}(0,{\sigma }_{\mathrm{int}}^2)$; 19:              Construct the intervened state ${\mathbf{s}}_{t,\mathrm{do}}$; 20:              Compute the masked intervened state ${\tilde{\mathbf{s}}}_{t,\mathrm{do}}={\mathbf{s}}_{t,\mathrm{do}}\odot \mathbf{m}$; 21:              Compute the actions ${\mathbf{a}}_t$, ${\mathbf{a}}_{t,\mathrm{do}}$; 22:              Compute the causal consistency loss $L_c\left(\theta \right)$; 23:              Compute the mask sparsity regularization $L_m\left(\mathbf{m}\right)$; 24:              Compute the loss function for the policy network: $$L_{\mathrm{actor}}(\theta ,\mathbf{m})=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N{Q}_{{\varphi }_1}({\mathbf{s}}_i,{\pi }_{\theta }\left({\tilde{\mathbf{s}}}_i\right))+{\lambda }_c{L}_c\left(\theta \right)+{\lambda }_m{L}_m\left(\mathbf{m}\right)$$ 25:              Update the mask vector $\mathbf{m}$: $$\begin{array}{cc}\hfill \mathbf{m}& \leftarrow \mathbf{m}-{\eta }_m{\nabla }_{\mathbf{m}}{L}_{\mathrm{actor}}(\theta ,\mathbf{m})\hfill \end{array}$$ 26:              Clip the mask vector to ensure $m_i\in [0,1]$: $$m_i\leftarrow \mathrm{clip}(m_i,0,1),\forall i\in \{1,\cdots ,n\}$$ 27:              Update the policy network and the target network parameters $\theta$, ${\varphi }_j$: $$\begin{array}{cc}\hfill {\varphi }_j\leftarrow {\varphi }_j-{\eta }_{\varphi }{\nabla }_{{\varphi }_j}{L}_{\mathrm{critic}}^j\left({\varphi }_j\right),& j=1,2\\ \hfill \theta \leftarrow \theta -{\eta }_{\theta }{\nabla }_{\theta }{L}_{\mathrm{actor}}(\theta ,\mathbf{m})& \end{array}$$ 28:           end if 29:       end for 30: end for

Finally, the complete framework of the proposed CSM-TD3 algorithm interacts with the vehicle path-planning environment, as illustrated in Figure 4. This framework comprises three core components: the vehicle dynamic path-planning environment, the actor–critic network, and the experience replay buffer. The vehicle dynamic path-planning environment consists of the vehicle dynamics model and the environmental model, designed to interact with the DRL algorithm to generate the data required for training. This dynamic model is capable of simulating the vehicle’s motion states and responses under various environmental conditions, providing diverse training scenarios so that the agent can learn effective path-planning strategies in complex and dynamic environments. The actor–critic network component includes both the online and target actor neural networks, which are responsible for generating appropriate control actions. The online actor network outputs the action policy based on the current state, while the target actor network is utilized to stabilize the training process and mitigate fluctuations during policy updates. Concurrently, the DRL critic component comprises two online critic neural networks and two target critic networks. These are designed to evaluate the value of actions and provide reliable feedback for updating the actor network. The design of dual critic networks effectively reduces the overestimation bias in value estimation, enhancing the stability and performance of the algorithm. The experience-replay buffer stores historical experience data generated by the agent during interactions with the environment. These data include information such as states, actions, rewards, and future states. By leveraging the experience-replay buffer, the algorithm can break the temporal correlations between consecutive samples, improve data efficiency, and stabilize the training process.

5. Comparison of Simulation Results

To validate the effectiveness of the proposed improved DRL algorithm (CSM-TD3) in dynamic path planning, we compared the performance of the CSM-TD3 algorithm with that of the TD3, proximal policy optimization (PPO) and DQN algorithms in terms of convergence and generalization through simulation experiments. All experiments were conducted on a hardware platform equipped with an Intel Core i9-13900K processor, sourced from Intel Corporation, Santa Clara, CA, USA, and an NVIDIA GeForce RTX 4080 graphics card, sourced from NVIDIA Corporation, Santa Clara, CA, USA.

5.1. Convergence Analyses

The simulation parameters for different DRL algorithms are shown in

Table 1. Simulation parameters used in different DRL algorithms.

DRL Algorithm	CSM-TD3	TD3	PPO	DQN
Hidden layer dimension	256	256	256	256
Batch size	256	256	256	256
Discount factor	0.99	0.99	0.99	0.99
Soft update coefficient	0.05	0.05	×	×
Policy noise	0.2	0.2	×	×
Noise clipping range	0.5	0.5	×	×
Policy update frequency	2	2	2	×
Priority exponent	0.6	0.6	0.6	0.6
Learning rate	1 × 10−4	1 × 10−4	1 × 10−4	1 × 10−4

. The discount factor determines the weight of future rewards relative to immediate rewards. The soft update coefficient controls the rate at which the target network converges toward the current network. The exploration noise reflects the degree of variability in exploration behavior, with its variation determined by bounded noise. The refresh frequency of the policy network dictates the update rate of the associated parameters. In the path-planning environment, the starting point is fixed at the center of the geographical coordinate system, while the endpoint is randomly generated within the path-planning boundary. The speed of obstacles varies within a specified range to ensure that the DRL algorithm can be trained in a more diverse environment; the training environment parameters are shown in

Table 2. System parameters of the environment.

Path-Planning Parameter	Parameter
Vehicle mass (kg)	1477
Vehicle yaw inertia ($\mathrm{kg}·{\mathrm{m}}^2$)	1536.7
Boundary length (m)	8
Angular velocity range of obstacles (rad/s)	$[0.2,0.4]$
Radius of obstacles (m)	0.5
Radius of obstacle motion path (m)	2.83

. To ensure comparability among the algorithms, all the hyperparameters and training environment parameters for the comparative algorithms were kept consistent.

In the established training environment, each dynamic path-planning task was performed 10 times to validate the stability and effectiveness of the DRL algorithms. During training, an evaluation report was generated in each assessment. In the comparison graphs for the reward curves, the solid line represents the average results of the reward curves across the 10 trials, while the shaded area indicates the range of variation in the reward curves. The path curves in the dynamic path-planning comparison illustrate the optimal paths generated by the trained agents of the same algorithm across the 10 trials in the same environment.

The average return curves obtained from three DRL algorithms in this convergence comparison experiment—DQN, PPO, TD3, and the proposed CSM-TD3—are compared in Figure 5 and Figure 6, with all the other parameters held constant. The average return of the DQN demonstrates a significantly slower growth rate throughout the training process compared to TD3 and CSM-TD3 and exhibits considerable fluctuations, consistently hovering at a lower level. This is attributed to the DQN’s original design, which is tailored for discrete-action decision making, making it challenging to effectively model and optimize policies for high-dimensional continuous control tasks. Furthermore, the high volatility of the DQN indicates that it is heavily influenced by randomness during policy updates, lacking an effective mechanism to smooth and stabilize its optimization process, presenting certain limitations for continuous control tasks in dynamic path planning.

PPO is a policy gradient-based method that directly optimizes the policy for continuous actions, avoiding the complexity of discretizing actions in continuous spaces as required by DQN. PPO improves training stability by clipping policy updates, thereby restricting the magnitude of each update. This helps to mitigate performance fluctuations caused by excessively large updates, resulting in better convergence performance compared to DQN in continuous action spaces.

In this experiment, TD3 also outperformed the DQN, showing a clear upward trend in average returns and ultimately achieving higher return levels than the DQN. However, the return curve for TD3 exhibited significant fluctuations, indicating a degree of instability, particularly during the initial training phase, where the returns varied greatly. This suggests that TD3’s handling of noise during the policy exploration phase and its ability to improve policies were relatively limited, making it difficult to maintain consistent performance improvements. Thus, although TD3 gradually surpassed the DQN during convergence, its overall stability and final convergence performance were still inferior to those of CSM-TD3.

CSM-TD3, on the other hand, demonstrated a consistently increasing trend in average returns throughout the training process, with relatively small fluctuations. This indicates that the algorithm possesses strong sample efficiency and stable convergence performance, attributable to the improvements made to TD3 through the introduction of causal models and a state-masking mechanism. Specifically, CSM-TD3 employs an SEM to identify and leverage state features that have direct causal effects on control inputs, using a state-masking vector to filter out irrelevant features that do not contribute to decision making, reducing noise and spurious correlations. Moreover, the incorporation of causal consistency constraints and causal consistency loss enables the effective learning of causal relationships, which significantly enhances CSM-TD3’s policy optimization compared to the traditional TD3 algorithm. The comparative results of different DRL algorithms are presented in

Table 3. Mean average returns obtained from different DRL algorithms.

DRL Algorithm	DQN	TD3	CSM-TD3
Mean average return	−781.6090	−508.2607	−318.9694

. Compared to the DQN, the proposed CSM-TD3 improved the convergence performance by 51%, and compared to TD3, it achieved an improvement of 37%.

5.2. Generalization Analyses

In this path-planning simulation, we designed a dynamic path-planning environment to evaluate the generalization capabilities and path-planning performance of three RL algorithms: DQN, TD3, and CSM-TD3. The entire experiment was divided into two phases: the first phase involved a static obstacle environment, while the second phase featured a dynamic obstacle environment. This design allows us to progressively increase the difficulty of path planning from simple to complex scenarios. In the static obstacle environment, the agent only needs to contend with the challenges posed by a static setting, providing a performance baseline for each algorithm under fundamental conditions. In contrast, the dynamic obstacle environment introduces continuously moving obstacles, significantly increasing the complexity of path planning and enabling the assessment of each algorithm’s adaptability and decision-making responsiveness in the face of dynamic changes. Through this phased approach, we can observe how the agent transitions from handling static obstacles to confronting more complex dynamic environments.

The environment is modeled as a two-dimensional plane with dimensions set to $[-5,5]\times [-5,5]$, where the agent’s starting point is at $(0,0)$ and the target points are located at three positions: $(0,4)$, $(-3,4)$, and $(4,4)$. This design maintains consistent initial conditions for each algorithm across different phases, ensuring comparability in their performances. The diversity of target points further tests the agent’s flexibility and adaptability in addressing various path requirements. This design, transitioning from the same starting point to different endpoints, increases the difficulty of path planning incrementally in each phase. As a result, the agent must demonstrate higher generalization and policy optimization capabilities to effectively navigate the challenges presented. In Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12, the black points represent the starting point, gray points represent obstacles, dashed circles represent the moving trajectory of obstacles, and purple points represent the goal point. The arrow indicates the direction of movement of the obstacle

In the static obstacle scenarios, as illustrated in Figure 7, Figure 8 and Figure 9, the path-planning results of the DQN exhibit significant deviations and instability. When faced with static obstacles, the DQN frequently demonstrates unnecessary detours and backtracking, particularly in densely packed obstacle areas, leading to chaotic path selections filled with superfluous turns. This performance issue stems from the DQN’s heavy reliance on discrete Q-value estimations, which results in inaccurate assessments of the optimal path and an inability to generate effective continuous actions. This directly manifests in its paths, showing excessive backtracking and redundant movements, greatly increasing the complexity and overall cost of the planned routes.

TD3 also manages to complete tasks in a static obstacle environment, but its paths are characterized by numerous unnecessary detours and twists. Especially when navigating narrow passages, the paths selected by TD3 lack smoothness, indicating either overly conservative obstacle avoidance or insufficient understanding of the environment. An analysis of the path trajectories reveals that TD3 experiences repeated adjustments in certain areas, suggesting that its strategy fails to optimize control decisions effectively when confronted with complex obstacle distributions. Although TD3 employs delayed policy updates to mitigate overestimation bias, it still faces limitations in extracting and utilizing critical state features in complex environments. Particularly when the agent encounters multiple obstacles, the lack of a global perspective on the optimal path results in redundant avoidance actions.

In contrast, CSM-TD3 demonstrates significant advantages in static obstacle scenarios, producing the smoothest paths that closely approximate the shortest route. The agent is able to navigate around obstacles precisely, reaching the target point with minimal detour costs and optimal decision making. The path diagram indicates that CSM-TD3 avoids unnecessary turns and backtracking, showcasing its superior performance. This advantage is primarily attributed to the causal enhancement strategies introduced in CSM-TD3, which effectively identify key state features that directly impact control decisions through structural equation modeling. The state-masking mechanism filters out redundant noise features so that the agent can focus on the crucial information affecting path planning during decision making, enhancing both the effectiveness and efficiency of path planning. Notably, when the agent traverses narrow corridors, CSM-TD3 exhibits exceptional smoothness and agility in responding to environmental changes, minimizing fluctuations along the entire path. This reflects its accurate judgment of obstacle layouts and efficient planning capabilities.

In the dynamic obstacle scenarios, as shown in Figure 10, Figure 11 and Figure 12, the paths generated by the DQN frequently deviate from the optimal direction. Particularly when obstacles approach, the agent struggles to react in time to avoid them, resulting in repeated backtracking and detours. This behavior highlights the DQN’s problems in handling rapidly changing dynamic obstacles, often causing its path planning to fall into local suboptimal traps and lack a global planning perspective.

TD3’s performance in the dynamic obstacle environment also declines. While TD3 ultimately manages to navigate around obstacles to reach the target point, its paths exhibit noticeable detours. Especially when dynamic obstacles suddenly come close, TD3’s responses appear sluggish, and its path adjustments lag behind, leading to decreased overall smoothness and efficiency. This inflexibility becomes particularly evident in scenarios requiring frequent path adjustments to cope with moving obstacles, resulting in unnecessary circumventions. Such delays directly affect the overall quality of path planning, making the final planned paths more complex and less efficient compared to those generated by CSM-TD3.

CSM-TD3, on the other hand, maintains exceptional performance in the dynamic obstacle environment. Its paths can adapt in real time to the dynamic changes of obstacles, demonstrating the rapid processing of dynamic information and real-time decision-making capabilities. The experimental results show that CSM-TD3 can swiftly adjust paths as obstacles approach, allowing the agent to maintain a smooth trajectory while effectively avoiding all dynamic obstacles, reaching the target point successfully. This performance advantage of CSM-TD3 is attributed to the causal consistency constraints, which ensure that the agent’s strategy focuses on causal features that directly influence control inputs when responding to environmental changes. This reduces the interference of spurious correlations in path planning, enabling the agent to react quickly and make effective adjustments in the presence of dynamic obstacles, achieving optimality and stability of its paths.

Upon combining the above analysis with the path step comparisons in

Table 4. Comparisons of the path steps obtained from the DQN, TD3, and the proposed CSM-TD3 when the obstacle is stationary.

DRL Algorithm	DQN	TD3	CSM-TD3
Location of Goal Point
$(0,4)$	26	24	23
$(-4,3)$	40	35	33
$(4,4)$	67	50	43

and

Table 5. Comparisons of the path steps obtained from the DQN, TD3, and the proposed CSM-TD3 when the obstacle is moving.

DRL Algorithm	DQN	TD3	CSM-TD3
Location of Goal Point
$(0,4)$	49	42	23
$(-4,3)$	44	34	32
$(4,4)$	58	40	36

, we see that CSM-TD3 exhibits exceptional planning performance in both static and dynamic obstacle environments. Leveraging causal enhancement strategies and causal consistency constraints, CSM-TD3 effectively identifies and utilizes state features that have direct causal impacts on control decisions. This capability allows the agent to maintain optimal, shortest, and smooth paths even in complex dynamic environments.

6. Conclusions

This study investigates the challenges faced by traditional DRL algorithms in dynamic path planning, particularly how confounding factors in the environment can impair convergence and generalization, making it difficult to generate high-quality paths. We propose a causal state-masking-based DRL method (CSM-TD3). Initially, we integrate a causal consistency loss term and dynamic state masking to combine causal inference mechanisms with TD3, compelling the policy network to rely solely on state features that reflect true causal relationships for decision making. This effectively filters out irrelevant features that do not contribute to the decision-making process. To enable TD3 to adaptively adjust the causal features across various state dimensions, we introduce an encoding configuration optimized through backpropagation learning. This allows the model to dynamically fine-tune causal features, enhancing the agent’s convergence during training and its generalization ability across different environments. Numerical simulations validate the efficacy of the proposed method. The experimental results indicate a 25% improvement in convergence performance. The CSM-TD3 algorithm effectively addresses dynamic path-planning challenges across various complex environments. Compared to other DRL algorithms, our approach generates shorter paths with better smoothness and demonstrates superior obstacle-avoidance capabilities.

Although the CSM-TD3 method demonstrates advantages in dynamic path planning, its training process relies on a large amount of training data and significant computational resources, which may limit its application in resource-constrained scenarios. Future research could integrate other forms of causal inference and adaptive mechanisms to make the learning strategy more efficient.