Distributed Drive Autonomous Vehicle Trajectory Tracking Control Based on Multi-Agent Deep Reinforcement Learning

Abstract

In order to enhance the trajectory tracking accuracy of distributed-driven intelligent vehicles, this paper formulates the tasks of torque output control for longitudinal dynamics and steering angle output control for lateral dynamics as Markov decision processes. To dissect the requirements of action output continuity for longitudinal and lateral control, this paper adopts the deep deterministic policy gradient algorithm (DDPG) for longitudinal velocity control and the deep Q-network algorithm (DQN) for lateral motion control. Multi-agent reinforcement learning methods are applied to the task of trajectory tracking in distributed-driven vehicle autonomous driving. By contrasting with two classical trajectory tracking control methods, the proposed approach in this paper is validated to exhibit superior trajectory tracking performance, ensuring that both longitudinal velocity deviation and lateral position deviation of the vehicle remain at lower levels. Compared with classical control methods, the maximum lateral position deviation is improved by up to 90.5% and the maximum longitudinal velocity deviation is improved by up to 97%. Furthermore, it demonstrates excellent generalization and high computational efficiency, and the running time can be reduced by up to 93.7%.

1. Introduction

In recent years, distributed drive autonomous vehicles (DDAVs), integrating distributed electric propulsion, have emerged as a pivotal direction in automotive technological advancement because of their notable advantages in energy efficiency, environmental friendliness, rapid response, simple structure, and high transmission efficiency [1,2]. As illustrated in Figure 1, the realization of intelligent vehicle trajectory tracking involves three main stages including perception, decision-making, and execution. Execution (motion control) represents the final frontier in intelligent driving, with trajectory tracking standing at its core. This involves orchestrating the longitudinal and lateral motion control of vehicles through the control of chassis systems such as drive, braking, and steering, enabling precise tracking of desired longitudinal velocities and timely arrival at corresponding reference trajectory points [3]. It is evident from this analysis that the execution phase, encompassing longitudinal and lateral vehicle motion control, constitutes a pivotal determinant of the precision in vehicle trajectory tracking. Simultaneously, the speed of computations plays a crucial role in determining the efficacy of control, particularly in hazardous scenarios. The enhancement of computational speed in the execution phase is therefore vital for improving safety measures. Hence, augmenting both the precision of control and computational speed within the trajectory tracking execution phase emerges as imperative for the advancement of intelligent vehicle driving technology.

In the current academic discourse on trajectory tracking for intelligent vehicles, the mainstream approach involves separately designing lateral motion controllers and longitudinal motion controllers, which are then combined. However, these designed control laws often demand high precision in vehicle dynamic parameters and computational resources [4,5,6,7,8,9]. Common methods for designing longitudinal controllers include proportional–integral–derivative (PID) methods, sliding mode control algorithms, or model predictive control (MPC) methods. These methods achieve autonomous driving longitudinal speed tracking by directly outputting vehicle acceleration or braking torque. Some scholars have proposed improvements to these methods; for instance, reference [10] combines PID methods with radial basis function neural networks to enhance robustness. Reference [11] presents an adaptive sliding mode control method, introducing parameter adaptation laws to compensate for variations in environmental disturbances and model uncertainties. Reference [12] employs MPC algorithms to design hierarchical controllers for distributed-driven vehicles, achieving autonomous driving longitudinal speed tracking and verifying control performance. Reference [13] combines neural networks with sliding mode control algorithms, resulting in an improved adaptive sliding mode control algorithm with better robustness. As for lateral controllers, they aim to stabilize the tracking of desired trajectories by controlling vehicle steering. Currently, they can be categorized into two types as follows: those based on kinematic control and those based on dynamic control. The design of specific control laws mostly involves PID methods, pure tracking algorithms, Linear Quadratic Regulator (LQR) methods, and MPC-based methods. Scholars have also made corresponding improvements in this domain. For example, reference [14] enhances PID methods based on kinematic models, which exhibit improved robustness with the introduction of neural networks. Reference [15] proposes an adaptive pure tracking algorithm based on kinematic models, effectively reducing tracking errors by combining PI control with pure tracking algorithms. Reference [16] proposes an online adaptive module for tire parameters in MPC controllers based on vehicle dynamic models, updating parameters online to reduce model mismatch and enhance the robustness of lateral motion controllers.

However, the dynamic response of DDAV exhibits not only strong nonlinearity but also uncertainty and time-varying characteristics, which can affect the precision of the aforementioned control laws. Compared with mainstream methods, the integration of neural networks can effectively enhance control effectiveness and robustness [17,18,19]. Therefore, some scholars propose utilizing deep reinforcement learning (DRL), which requires less computational power and lower precision of mathematical models in handling nonlinear control problems. Deep reinforcement learning (DRL) is an amalgamation of deep learning and reinforcement learning, possessing both the feature extraction capabilities of deep neural networks and the decision-making advantages of reinforcement learning. It enables end-to-end learning, from perception to decision control, based on deep neural networks. DRL methods engage in trial-and-error learning through interaction with the environment, allowing the model to explore and obtain optimal decision-making for control systems autonomously. Common algorithms such as DQN, DDPG, and DDQN exemplify DRL’s versatility. DRL finds extensive applications in fields like communication control, mechanical control systems, autonomous driving, and intrusion detection systems (IDSs) [20]. Reference [21] applied DRL to traffic control without lanes, designing reward and penalty functions to maintain vehicle ideal speeds while minimizing collisions among vehicles. Reference [22] developed a vision-based DRL autonomous driving agent, which, utilizing patterns detected by perception modules on lane markings and the rear end of preceding vehicles, achieved lane-keeping and car-following tasks. Reference [23] employed DRL to train agents capable of autonomously overtaking both stationary and moving vehicles on roads while avoiding collisions with obstacles ahead and to the right. Reference [24] utilized DRL in conjunction with hybrid A* path planning for parking maneuvers in autonomous vehicles, addressing the issue of traditional model predictive control (MPC) methods’ lack of consideration for the right-of-way of traffic participants in trajectory planning. References [25,26] employ reinforcement learning methods to design autonomous driving longitudinal controllers, achieving vehicle speed cruising by directly controlling vehicle acceleration for autonomous braking and driving. References [27,28] apply reinforcement learning methods to design steering control strategies, enabling trained agents to perform lane changes or turning actions by outputting front wheel angles in autonomous driving environments. It is evident that applying DRL to trajectory tracking control for intelligent vehicles and using single-agent DRL methods in the design of individual longitudinal or lateral controllers demonstrate unique advantages. However, most of these approaches combine mainstream control methods with DRL methods, failing to exploit the advantages of DRL fully.

To address this, our study applies multi-agent systems to the trajectory tracking task of distributed-driven vehicles with distributed drive characteristics. Based on this idea, we propose a distributed drive autonomous vehicle (DDAV) trajectory tracking control method based on multi-agent deep reinforcement learning (MADRL), as illustrated in Figure 2. The main contributions are listed as follows:

(1) Employing MADRL for trajectory tracking controller design, utilizing distributed training among agents, and continuously training and iterating using deep reinforcement learning methods by designing reward and penalty functions and corresponding networks. This approach aims to achieve optimal longitudinal and lateral control while approaching the global optimum to the maximum extent, thereby improving upon mainstream control methods that require high precision in parameters and are prone to “local optima” issues, with higher operational efficiency.

(2) Moreover, to fully exploit the traction capabilities of each driving wheel in distributed-driven vehicles, an optimization approach is employed to design the longitudinal torque distribution controller at the lower level, optimizing the distribution of torque commands executed by each drive motor.

(3) Through simulation, it is verified that the proposed distributed drive autonomous vehicle trajectory tracking control based on the multi-agent deep reinforcement learning method has good trajectory tracking performance and high computational efficiency under different working conditions and different speeds.

The remainder of this paper is structured as follows: Section 2 presents the proposed control method and the corresponding parameter design. Section 3 outlines the training process and experimental results. Finally, Section 4 provides the conclusion.

2. Trajectory Tracking Control for DDAVs Based on MADRL

Deep reinforcement learning can be understood as the integration of deep learning and reinforcement learning. Deep learning is responsible for function approximation using deep neural networks, while reinforcement learning represents the iterative training process. Within this framework, agents are defined within the system. These agents interact with the environment and states, taking actions based on predetermined reward or penalty functions. Through continuous training, the goal is to maximize the total reward, updating the control policies within the agents.

2.1. Analysis of Longitudinal and Lateral Motion Control for DDAVs

In this section, the mathematical formulation of the dynamic control theory for DDAVs is presented. Firstly, the three-degree-of-freedom model of DDAVs is described, as shown in Figure 3. In the realm of vehicle dynamics, commonly employed models include the two-degree-of-freedom vehicle model, the three-degree-of-freedom vehicle model, and the seven-degree-of-freedom vehicle model, which can capture distributed drive characteristics. The two-degree-of-freedom vehicle model simplifies vehicle dynamics by disregarding suspension effects, treating the vehicle as a two-wheeled entity, assuming linear tire lateral characteristics, and neglecting longitudinal driving or resistance forces, thereby considering longitudinal velocity as constant and only representing vehicle yaw and lateral motion. The three-degree-of-freedom vehicle model builds upon the two-degree-of-freedom model by introducing an additional degree of freedom for longitudinal motion while keeping the other two degrees of freedom unchanged for lateral motion and yaw motion. Building upon the foundation of the three-degree-of-freedom model, it is extended to a seven-degree-of-freedom model, which accounts for both lateral, longitudinal, and yaw motions of the vehicle, as well as the rotational states of all four wheels. This extended model comprehensively captures the driving characteristics of the entire vehicle while reflecting the distributed drive nature of DDAVs [29].

$$m\left({\stackrel{·}{V}}_x-V_y\gamma \right)=\left(F_{xfl}+F_{xfr}\right)\cos {\delta }_f-\left(F_{yfl}+F_{yfr}\right)sin{\delta }_f+F_{xrl}+F_{xrr}$$

(1)

$$m\left({\stackrel{·}{V}}_y+V_x\gamma \right)=\left(F_{xfl}+F_{xfr}\right)\sin {\delta }_f+\left(F_{yfl}+F_{yfr}\right)\cos {\delta }_f+F_{yrl}+F_{yrr}$$

(2)

$$I_z\stackrel{·}{\gamma }=l_f\left(F_{xfl}+F_{xfr}\right)\sin {\delta }_f+l_f\left(F_{yfl}+F_{yfr}\right)\cos {\delta }_f-l_r\left(F_{yrl}+F_{yrr}\right)+l_b\left(F_{xfr}-F_{xfl}\right)\cos {\delta }_f+l_b\left(F_{xrr}-F_{xrl}\right)+l_b\left(F_{yrr}-F_{yrl}\right)\sin {\delta }_f$$

(3)

In the above equations, $\mathrm{m}$ represents the total vehicle mass; $F_{xij}$ is the longitudinal force of the tire; $F_{yij}$ is tire lateral force (i represents the front wheel or rear wheel, and j represents the left wheel or right wheel); $V_x$ denotes the longitudinal velocity of the vehicle; $V_y$ represents the lateral velocity; $\gamma$ is the yaw rate of the vehicle; $I_z$ is the yaw inertia coefficient of the vehicle; ${\delta }_f$ is the average front wheel steering angle; $\beta$ is the sideslip angle; $l_f$ and $l_r$ are the distances from the center of gravity to the front and rear axles, respectively; $l_b$ is half of the wheelbase; and $V$ is the forward speed of the vehicle.

In the current scholarly discourse surrounding longitudinal and lateral motion control in autonomous driving vehicles, the predominant methodology largely entails the establishment of a two-degree-of-freedom reference model. This model serves as the basis for deriving control objectives such as the desired yaw rate and lateral deviation angle of the center of mass. Subsequently, a simplified linearization of the tire model is applied to facilitate kinematic control. However, within the context of the seven-degree-of-freedom model of DDAVs, it becomes evident that joint longitudinal and lateral motion control in real vehicle systems is subject to three primary coupling effects including kinematic coupling, load transfer coupling, and tire force coupling [30]. Notably, the dynamic response of both vehicles and tires manifests significant nonlinearity, alongside uncertainties and time-varying characteristics. These factors pose challenges to the precision of mainstream control methods. Moreover, errors in parameter estimation can profoundly impact the adaptability of control laws across various operational conditions. Consequently, the machine learning algorithms introduced in this study offer a comprehensive solution to address the shortcomings of mainstream control methods in terms of precision, efficiency, and robustness. Furthermore, as illustrated in Figure 4, the practical implementation of deep reinforcement learning methods during vehicle operation enables continuous training throughout the vehicle’s lifecycle and periodic policy iteration. This approach significantly reduces the reliance on high-precision mathematical models of vehicles and computational resources when compared with mainstream control methods.

2.2. Design of the MADRL Controller

As stated earlier, trajectory tracking in DDAVs is a responsive outcome under multi-execution system control involving both the propulsion and steering systems. Because of the transformation between the vehicle’s coordinate system and the global coordinate system, the steering system not only induces lateral position changes but also affects the vehicle’s longitudinal velocity [31]. Therefore, when employing deep reinforcement learning methods for joint longitudinal and lateral control of DDAVs, the complexity arises from the multitude of vehicle state indicators and control objectives. Simple control design with a single agent may lead to difficulties in setting reward functions and coordinating priority relationships. Unlike single-agent deep reinforcement learning, MADRL partitions the action space into different subspaces and assigns them to different agents. This effectively reduces the complexity of network design for each agent, allowing multiple control strategies to be trained independently by different agents [32]. Consequently, this approach enables the independent setting of states, actions, and rewards for more targeted training.

The trajectory tracking control strategy proposed in this paper, as illustrated in Figure 5, consists of the DDAV system, lateral control Agent1, and longitudinal control Agent2. Upon receiving the vehicle state, Agent1 outputs steering commands applied to the front wheels while updating its own neural network. Agent2 outputs torque commands and utilizes a lower-level allocator to output motor longitudinal torque to drive the vehicle to track the desired longitudinal velocity, simultaneously updating its own neural network. In DDAVs, the driving motor executes torque actions, and the steering actuator executes steering actions, which, based on kinematic coupling, load transfer coupling, and tire force coupling relationships, induce changes in the overall vehicle state. Corresponding reward values are generated based on reward functions for each agent to train and update according to set parameters.

2.2.1. Design of Reward or Penalty Functions

The Markov Decision Process (MDP) refers to a system where the current state depends only on the preceding state and action. In the trajectory tracking process of DDAVs, each agent’s action leads to an update in the DDAV to a new state. Based on the DDAV’s state, corresponding rewards are generated. Through iterative training, the neural networks of the agents achieve effective training. Therefore, the longitudinal and lateral control processes of DDAVs can be regarded as MDPs.

$${\pi }^{\ast }=\arg \max \mathbb{E}\left[{\displaystyle \sum _{t=0}^{N}r\left(s\left(t\right),a\left(t\right)\right)·T_s\left|s\left(0\right)=s_0\right.}\right]$$

(4)

In the above equation, $s_0$ represents the initial state, $s\left(\cdot \right)$ denotes the state value, $a\left(\cdot \right)$ represents the action value, $r\left(\cdot \right)$ corresponds to the reward, $N$ is the total time duration, and $A$ is the action space.

The future cumulative reward obtained after executing action $a\left(t\right)$ in state $s\left(t\right)$ according to the control strategy $\pi$ can be defined as $Q_{\pi }\left(s,a\right)$. $Q_{\pi }\left(s,a\right)$ is used to evaluate action $a\left(t\right)$.

$$Q_{\pi }\left(s,a\right)=\mathbb{E}\left[{\displaystyle \sum _{t=0}^{N-T}r\left(s\left(t\right),a\left(t\right){\sigma }^k\right)\left|s\left(0\right)=s_T,a\left(0\right)=a_T\right.}\right]$$

(5)

In the above equation, $\sigma$ denotes discount factor, $\sigma \in \left[0,1\right]$.

If the optimal action $a\left(t\right)$ is chosen in a certain state $s\left(t\right)$, then the corresponding optimal $Q^{\ast }\left(s,a\right)$ is obtained at that moment.

$$Q^{\ast }\left(s,a\right)=\underset{\pi }{\max }{Q}_{\pi }\left(s,a\right)$$

(6)

The optimal control policy ${\pi }^{\ast }$ can be represented as follows:

$${\pi }^{\ast }=\arg \max \underset{\pi }{\max }\left[Q^{\ast }s\left(t\right),a\left(t\right)\left|s\left(0\right)=s_0\right.\right]$$

(7)

To achieve the DDAV trajectory tracking objective, the state variables, action variables, and reward–penalty functions for the lateral and longitudinal controllers are defined as follows:

(1)

Agent for lateral control

The control problem of the lateral control agent is described as shown in Equation (8), where $S_{lat}$ is the set of observed values generated by the environment, defined as the lateral position deviation $e_1$ in the vehicle, the yaw angle deviation $e_2$ of the vehicle, and their respective rates of change $\stackrel{·}{e_1}$ and $\stackrel{·}{e_2}$. $A_{lat}$ is the set of action spaces, defined as the front wheel steering command $u$. $R_{lat}$ is the reward and punishment function evaluating the performance of the steering controller. To ensure the accuracy of trajectory tracking while minimizing the output of unnecessary steering actions, the reward and punishment function consists of position deviation and its rate of change, yaw angle deviation and its rate of change, and front wheel steering action; $a_1\dots a_5$ is the weight coefficient. In this paper, for the convenience of training the agent, it is necessary to normalize each component of the reward and punishment function before input.

$$\left\{\begin{array}{c}{S}_{lat}=\left\{e_1,e_2,{\stackrel{·}{e}}_1,{\stackrel{·}{e}}_2\right\}\\ A_{lat}=\left\{u\right\}\\ R_{lat}=\left\{-\left({e_1}^2\cdot a_1+{e_2}^2\cdot a_2+u^2\cdot a_3+{\stackrel{·}{e}}_1\cdot a_4+{\stackrel{·}{e}}_2\cdot a_5\right)\right\}\end{array}\right.$$

(8)

In the above equation, the inclusion of $u^2$ is motivated by the aim to minimize steering interventions while fulfilling trajectory tracking requirements, thereby enhancing the driving experience. This approach prioritizes reducing steering maneuvers without excessively pursuing tracking precision, which could otherwise lead to continual fine adjustments of steering angles. The adjustment process of the weights $a_1\dots a_5$ is a human-driven endeavor, necessitating iterative cycles of experimentation, training, observation, and refinement.

In the above equation, $e_1$ represents the lateral position deviation of the vehicle and $e_2$ represents the yaw angle deviation of the vehicle:

$$\left\{\begin{array}{c}{e}_1=abs\left(y_t-y\right)\\ e_2=abs\left({\omega }_t-\omega \right)\end{array}\right.$$

(9)

In the above equation, $y_t$ represents the expected lateral position of the vehicle at the given moment; $y$ denotes the actual lateral position of the vehicle at that moment; ${\omega }_t$ signifies the expected yaw angle of the vehicle at the given moment; and $\omega$ represents the actual yaw angle of the vehicle at that moment.

The action space constraint is defined as:

$$u_{\min }\le u\le u_{\max }$$

(10)

In the above equation, $u_{\min }$ and $u_{\max }$ correspond, respectively, to the actual range of the front wheel steering angles of the vehicle.

However, because of the strong generality of DRL, there is significant randomness in action selection during the early stages of training, leading to extensive useless exploration and resulting in inefficiencies in exploration, lengthy training cycles, and poor training outcomes. Combining rule-based control strategies is an effective means of efficiently utilizing reinforcement learning tools. When states match predefined rules, priority is given to these rules; otherwise, reinforcement learning is employed for exploration. In this paper, a rule-based terminal state design for the agent is proposed. Terminal states typically correspond to the conclusion of a training episode, often defined by the number of steps to trigger episode termination. Building upon a step-based terminal state framework, this study introduces a rule-based terminal state design. When the current state matches predefined rules for terminating an episode prematurely, the training cycle with unsatisfactory performance can be ended prematurely, allowing for the commencement of the next training cycle to reduce futile exploration in the current cycle.

The terminal conditions $D_{lat}$ for lateral control agent are defined as follows:

$$D_{lat}=\left\{\begin{array}{cc}0& e_1\le 1\\ 1& e_1>1\end{array}\right.$$

(11)

(2)

Agent for longitudinal control

The control problem of the longitudinal control agent is described as shown in Equation (12), where $S_{lon}$ represents the set of observed values generated by the environment, specifically, the desired longitudinal velocity $V_t$ of the vehicle, the actual longitudinal velocity $V_x$, and their deviation $V_e$. $A_{lon}$ denotes the action space set, representing the longitudinal force torque exerted by the drive motor $T_t$. $R_{lon}$ is the reward and punishment function evaluating the performance of the longitudinal velocity controller. To ensure precision in maintaining longitudinal velocity while minimizing abrupt accelerations or decelerations, the reward and punishment function comprises longitudinal velocity deviation and vehicle acceleration, as well as an individual reward component $r$ and an individual punishment component $p$. The introduction of individual reward and punishment components aims to provide substantial rewards for minimal velocity deviations or impose significant penalties for substantial velocity deviations, thus better guiding the training process. $b_1,b_2$ represents the weight coefficient. In this paper, for the convenience of training the agent, it is necessary to normalize the inputs of $V_x$ and $V_e$ within the reward and punishment function before processing.

$$\left\{\begin{array}{c}{S}_{lon}=\left\{V_t,V_x,V_e\right\}\\ A_{lon}=\left\{T_t\right\}\\ R_{lon}=\left\{\left(r-p-ac{c}^2\cdot b_1-{V_{\mathrm{e}}}^2\cdot b_2\right)\right\}\end{array}\right.$$

(12)

In the above equation, longitudinal velocity deviation can be defined as:

$$V_e=V_t-V_x$$

(13)

the individual reward component $r$ can be defined as:

$$r=\left\{\begin{array}{ccc}10& & \left|V_e\right|\le 0.6\%\times V_t\hfill \\ 5& \hfill 0.6\%\times V_t<& \left|V_e\right|\le 2\%\times V_t\hfill \\ 1& \hfill 2\%\times V_t<& \left|V_e\right|\le 6\%\times V_t\hfill \\ 0& & \left|V_e\right|>6\%\times V_t\hfill \end{array}\right.$$

(14)

and the individual punishment component $p$ can be defined as:

$$p=\left\{\begin{array}{cc}0& \left|V_e\right|<20\%\times V_t\\ 100& \left|V_e\right|\ge 20\%\times V_t\end{array}\right.$$

(15)

The terminal conditions $D_{lon}$ for the longitudinal control agent are defined as follows:

$$D_{lon}=\left\{\begin{array}{cc}0& \left|V_e\right|\le 30\%\times V_t\\ 1& \left|V_e\right|>30\%\times V_t\end{array}\right.$$

(16)

It should be noted that the adjustment of the weights $a_i$/$b_i$ in Equations (8) and (12) is an artificial process, which requires continuous trial, training, observation, and adjustment until the desired training effect is achieved.

To fully exploit the adhesion capability of the driving wheels, a longitudinal torque distribution controller is designed at the lower level of the longitudinal control intelligent agent to optimize the allocation of torque commands sent to each driving motor. When the vehicle is traveling straight, the vertical loads on both sides of the driving wheels are equal. However, during steering, the vehicle’s center of gravity shifts outward because of centrifugal force, resulting in an increase in the vertical load on the outer wheel and a decrease in the load on the inner wheel [33]. The concept of “friction ellipse” can be used to express the road adhesion load of a single wheel:

$${\eta }_i=\frac{\sqrt{{F_{xi}}^2+{F_{yi}}^2}}{{\mu }_i{F}_{zi}}$$

(17)

In the above equation, $F_{xi}$ represents the longitudinal force of the driving wheel, $F_{yi}$ represents the lateral force of the driving wheel, $F_{zi}$ represents the vertical load on the driving wheel, and ${\mu }_i$ represents the road adhesion coefficient.

If we assign weight coefficients to each wheel, denoted as $c_i$, the function for the overall vehicle’s road adhesion load can be expressed as follows:

$$J_{\eta }={\displaystyle \sum _{i=1}^4{c}_i{\eta }_i^2=}{\displaystyle \sum _{i=1}^4{c}_i\frac{{F_{xi}}^2+{F_{yi}}^2}{{{\mu }_i}^2{F_{zi}}^2}}$$

(18)

The smaller the function for the overall vehicle’s road adhesion load, the greater the adhesion margin. As it is not feasible to actively control the lateral forces exerted on the wheels, the objective function for optimization allocation is:

$$\min J=\min {\displaystyle \sum _{i=1}^4{c}_i{\eta }_i^2=}\min {\displaystyle \sum _{i=1}^4{c}_i\frac{{T_{xi}}^2}{{\left({{\mu }_i}^2{F_{zi}}^{2}R\right)}^2}}$$

(19)

In the above equation, $T_{xi}$ represents the torque commands sent to each driving motor and $R$ represents the rolling radius of the driving wheel.

Additionally, the constraint boundaries for the optimization allocation controller are:

$$\left\{\begin{array}{c}(T_{fl}+T_{fr})\cos u+T_{rl}+T_{rr}=T_t\\ T_{xi}\le \min (\mu F_{zi}R,T_{max})\end{array}\right.$$

(20)

2.2.2. Design of DQN and DDPG Networks

In the process of autonomous driving, commands for the line-controlled steering motor can be discretely outputted. Discrete command output typically entails simpler control logic, ensuring real-time capability and implementation simplicity of the line-controlled steering motor controller, while also meeting the requirements of the steering system. Moreover, because of the fixed values and ranges of discrete commands, it becomes easier to predict and control the vehicle’s steering behavior, thereby enhancing system stability and controllability. However, the adjustment requirements for the vehicle’s power output section are more refined. Discrete command output may affect the driving comfort of the vehicle. Considering that in operational scenarios of the vehicle, there is a greater emphasis on controlling longitudinal speed, and aiming to fully leverage the advantages of deep reinforcement learning algorithms such as DQN and DDPG in different control dimensions [34], enhancing system robustness and flexibility while reducing the complexity of algorithm design, the DQN algorithm is adopted for lateral control Agent1, while the DDPG algorithm is employed for longitudinal speed control Agent2.

Q-learning aims to maximize by selecting the optimal action for each state and updating it accordingly.

$$Q\left(s,a\right)\leftarrow Q\left(s,a\right)+\alpha \left[r+\lambda \underset{a}{\max }Q\left(s^{\prime },a^{\prime }\right)-Q\left(s,a\right)\right]$$

(21)

In the above equation, $s$ represents the current state, $a$ denotes the current action, $s^{\prime }$ signifies the next moment’s state, $a^{\prime }$ represents the action at the next moment, $\alpha$ is the learning rate, $r$ denotes the reward, and $\lambda$ is the decay coefficient.

In the DQN algorithm employed by the lateral control agent, the training samples for deep learning are derived from reinforcement learning data. The neural network in deep learning performs computations on input data through operations such as convolution, pooling, and fully connected layers, generating action Q-values. Subsequently, the loss function is computed based on the difference between these Q-values and the correct Q-values provided by reinforcement learning. The deep learning network’s weights are then adjusted according to the loss function.

Therefore, the loss function for DQN is

$$L\left(\theta \right)=\mathbb{E}\left[{\left(r+\sigma \underset{a}{\max }-Q\left(s^{\prime },a^{\prime }\left|{\theta }_T\right.\right)-Q\left(s,a\left|\theta \right.\right)\right)}^2\right]$$

(22)

In the above equation, $\theta$ represents the neural network parameters.

Upon obtaining the loss function, DQN utilizes stochastic gradient descent to compute gradients, aiming to minimize the loss and drive action values toward maximization.

$${\nabla }_{\theta }L\left(\theta \right)=\mathbb{E}\left[\left(r+\sigma \underset{a}{\max }-Q\left(s^{\prime },a^{\prime }\left|{\theta }_T\right.\right)-Q\left(s,a\left|\theta \right.\right)\right){\nabla }_{\theta }Q\left(s,a\left|\theta \right.\right)\right]$$

(23)

Additionally, because of the inherent Markovian property of reinforcement learning, which results in high correlation among training samples—an aspect inconsistent with the requirements of deep learning—DQN incorporates an experience replay mechanism, as illustrated in Equation (24):

$$sample=\left(s,a,r,s^{\prime }\right)$$

(24)

Storing samples in a sample pool is employed to calculate the gradient of the loss function and update the neural network parameters, ensuring independence among samples.

DQN also introduces a target network, which calculates temporal difference errors when solving the loss function.

$$L\left(\theta \right)=\mathbb{E}\left[\left(\underset{T\arg et network}{\underbrace{r+\sigma \underset{a}{\max }-Q\left(s^{\prime },a^{\prime }\left|{\theta }_T\right.\right)}}-\underset{Online network}{\underbrace{Q\left(s,a\left|\theta \right.\right)}}\right)\right]$$

(25)

Exclusively utilizing the online network to obtain $\max Q\left(s^{\prime },a^{\prime }\right)$ and $Q\left(s^{\prime },a^{\prime }\right)$ would result in synchronous variations, leading to non-convergence of the loss function. Therefore, calculations are performed separately using both the online network and the target network. The target network directly copies the real-time parameters of the online network to update its parameters after a certain number of time steps.

Simultaneously, to avoid falling into local optima, DQN employs an ε-greedy strategy when selecting actions. Specifically, with a probability of $\left(1-\epsilon \right)$, it selects the action with the maximum Q value, and with a probability of $\epsilon$, it chooses a random action.

The DDPG algorithm employed by the longitudinal control agent consists of two networks as follows: the actor and the critic. Each network includes both a current network and a target network, denoted by ${\theta }^{\mu }$ and ${\theta }^Q$, representing the parameters for the actor and critic networks, respectively. In the actor network, ${\theta }^{\mu }$ is updated based on the value function, and actions are generated according to the current state. The critic network updates ${\theta }^Q$ using the network gradient and evaluates the value of actions. The critic network updates its network parameters ${\theta }^Q$ by minimizing the loss function, which is expressed as follows:

$$L\left({\theta }^Q\right)=\frac{1}{N}{{\displaystyle {\sum }_t\left[y_t-Q\left(s_t,a_t\right)\left|{\theta }^Q\right.\right]}}^2$$

(26)

$$y_t=r_t+\gamma Q^{\prime }\left(s_{t+1},{\mu }^{\prime }\left(s_{t+1}\left|{\theta }^{{\mu }^{\prime }}\right.\right)\left|{\theta }^{Q^{\prime }}\right.\right)$$

(27)

In the above equation, $\gamma$ denotes discount factor.

The actor network updates its network parameters ${\theta }^{\mu }$ using deterministic policy gradients:

$${\nabla }_{{\theta }^{\mu }}J\left(\theta \right)=\frac{1}{N}{\displaystyle {\sum }_t\left[{\nabla }_{a}Q\left(s_t,\mu \left(s_t\left|{\theta }^{\mu }\right.\right)\left|{\theta }^Q\right.\right){\nabla }_{{\theta }^{\mu }}\mu \left(s_t\left|{\theta }^{\mu }\right.\right)\right]}$$

(28)

Simultaneously, the parameters of the two target networks in DDPG are updated using a “soft update” approach, which effectively prevents oscillation and divergence:

$$\left\{\begin{array}{c}{\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^{Q^{\prime }}+\left(1-\tau \right){\theta }^{Q^{\prime }}\\ {\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{{\mu }^{\prime }}+\left(1-\tau \right){\theta }^{{\mu }^{\prime }}\end{array}\right.$$

(29)

In the above equation, $\tau$ denotes the target smooth factor.

DDPG also incorporates the experience replay mechanism from DQN.

$$sample=\left(s,a,r,s^{\prime }\right)$$

(30)

By sampling from the environment and placing the samples into a pool, random mini-batches of data are extracted for learning updates. The execution of actions is obtained as follows:

$$a_t=\mu \left(s_t\left|{\theta }^{\mu }\right.\right)+N_t$$

(31)

In the above equation, $N_t$ represents random noise and $\mu \left(s_t\left|{\theta }^{\mu }\right.\right)$ denotes the optimal behavioral policy.

In this study, the updating processes of respective neural networks and the design of network frameworks are illustrated in Figure 6, Figure 7, Figure 8 and Figure 9. The settings of learning parameters for each agent and the definition of neural network structures are presented in

Table 1. Hyperparameters for DQN.

Hyperparameters	Values
Learn rate	1 × 10−3
Discount factor	0.99
Experience buffer	1 × 106
Minibatch size	128
Target smooth factor	1 × 10−3
Maximum episode	1400
Training time consumed	Almost 300 min

and

Table 2. Hyperparameters for DDPG.

Hyperparameters	Values
Learn rate of actor network	1 × 10−4
Learn rate of critic network	1 × 10−3
Experience buffer	1 × 106
Minibatch size	128
Target smooth factor	1 × 10−3
Discount factor	0.99
Noise variance	0.6
Noise variance decay rate	1 × 10−5
Maximum episode	1400
Training time consumed	Almost 300 min

.

3. Results and Discussion

3.1. Training and Experimental Setup

In the Matlab/Simulink environment, the DDAV model and MADRL trajectory tracking control strategy were constructed. The training platform utilized the 11th Gen Intel(R) Core(TM) i7-1165G7 processor. The parameters of the DDAV are shown in

Table 3. DDAV vehicle parameters.

Symbol	Description	Value
$m$	Vehicle mass/kg	1412
$a$	Distance from the center of mass to the front axle/m	1.015
$b$	Distance from the center of mass to the rear axle/m	1.895
$df$	Front track/m	1.675
$dr$	Rear track/m	1.675
$R$	Tire rolling radius/m	0.325
$hg$	Vehicle center of mass height/m	0.54
$I_z$	Vehicle yaw moment of inertia/kg·m2	1536.7
$J_w$	Wheel rotational inertia/kg·m3	0.9
$uf$	Road friction coefficient	0.018
$\mu$	Adhesion coefficient	0.8
$Cd$	dc drag coefficient	0.4

. Considering the characteristics of the vehicle’s tires, scholars both domestically and internationally have proposed various tire dynamic models based on the nonlinear properties of tire dynamics, such as the “magic formula” semi-empirical tire model and the Dugoff tire model. The “magic formula” semi-empirical tire model demonstrates strong uniformity and high fitting accuracy, making it suitable for fields requiring precise descriptions of tire mechanics [35]. In this study, the vehicle tire model was constructed based on the “magic formula” semi-empirical tire model [36].

The double lane change (DLC) maneuver serves as a robust assessment of lane-keeping capabilities. Thus, a DLC scenario at a velocity of 36 km/h was selected for training purposes, with the reference trajectory depicted in Figure 10 [37]. To augment the adaptability of each intelligent agent, randomization of the initial lateral position deviation and initial longitudinal velocity deviation in the DDAV was performed at the onset of each training session.

3.2. Iterative Training Results

The training objective of deep reinforcement learning is to maximize the reward value; thus, the indication of training completion lies in the stable convergence state of episode reward values after fluctuations. As depicted in Figure 11, longitudinal control shows the agent reaching stable convergence after the 1100th episode, whereas lateral control demonstrates stable convergence after the 200th episode.

3.3. Validation of Control Policy Effectiveness and Generalization

Trained agents were integrated into the control strategy to conduct lane-changing experiments and double lane-change experiments with the DDAV at a speed of 36 km/h. For the lane-changing operation (LC), adhering to the guidelines outlined in the “Urban Expressway Design Code of the People’s Republic of China (CJJ129-2009)”, a lane width of W = 3.75 m was utilized, with a designated completion time of 3 s for the lane-change maneuver. Two validated classical trajectory tracking longitudinal and lateral control methods were selected for comparison as follows: one method combines PID longitudinal velocity control with model predictive control (MPC) for lateral motion control (PM) [5,6], while the other combines PID longitudinal velocity control with pure tracking lateral motion control (PP) [7,8,9].

Figure 12 and Figure 13 respectively illustrate the lateral position and longitudinal velocity scenarios under LC and DLC experiments for both the MADRL method and classical control methods employed in this study.

Figure 12 shows that in the LC experiments, MADRL enables the vehicle to complete the LC maneuver more accurately within the specified time compared with classical control methods, with smaller lateral overshoot and maximum longitudinal velocity deviation compared with classical control methods. Figure 13 reveals that in the DLC experiments, MADRL exhibits a smaller maximum lateral position deviation and better robustness in longitudinal velocity. The deviation between longitudinal velocity and the desired longitudinal velocity remains consistently low, with a maximum longitudinal velocity deviation not exceeding 0.14 km/h, and the longitudinal velocity deviation under stable vehicle operation does not exceed 0.04 km/h. Although MADRL exhibits extremely small deviations in longitudinal velocity under stable vehicle operation, this minimal deviation is due to constraints imposed by the characteristics of the reinforcement learning algorithm itself. Specifically, after achieving all the reward and penalty metrics in the reward function, the algorithm no longer actively pursues further optimization, as doing so would not lead to higher rewards or lower penalties for the intelligent agent.

Additionally, we needed to verify the generalization capability of the trained control strategy at different speeds. Since the agents designed in this study pay more attention to the performance of lateral position deviation and velocity deviation, two moderate-speed states (18 km/h and 54 km/h) where vehicle instability is unlikely to occur were selected for generalization validation through LC and DLC experiments. Figure 14, Figure 15, Figure 16 and Figure 17 depict the lateral position and longitudinal velocity scenarios under LC and DLC experiments at different speeds for both the MADRL method and classical control methods employed in this study.

Figure 14 and Figure 16 reveal that under LC experiments at different speeds, MADRL enables the vehicle to complete the lane-changing maneuver more accurately within the specified time, with a smaller lateral overshoot. Moreover, the maximum longitudinal velocity deviation for the MADRL method is consistently lower than that for classical control methods across all speed conditions. On the other hand, Figure 15 and Figure 17 demonstrate that under DLC experiments at different speeds, MADRL exhibits smaller maximum lateral position deviation and better robustness in longitudinal velocity. The deviation between longitudinal velocity and the desired longitudinal velocity remains consistently low. Specifically, during DLC experiments at 18 km/h, the maximum longitudinal velocity deviation for MADRL does not exceed 0.05 km/h, and under stable vehicle operation, the longitudinal velocity deviation does not exceed 0.02 km/h. Similarly, during DLC experiments at 54 km/h, the maximum longitudinal velocity deviation for MADRL does not exceed 0.4 km/h, and under stable vehicle operation, the longitudinal velocity deviation does not exceed 0.01 km/h (

Table 4. Comparison of the performance of DLC conditions.

Comparison Metrics		PP	PM	MADRL
The maximum lateral position deviation/m	18 km/h	0.21	0.09	0.02
	36 km/h	0.03	0.12	0.01
	54 km/h	0.30	0.09	0.06
The maximum longitudinal velocity deviation/km/h	18 km/h	1.50	0.66	0.05
	36 km/h	0.77	0.76	0.14
	54 km/h	0.94	0.93	0.40

).

The results above indicate that the MADRL method employed in this study, along with its corresponding design of intelligent agent reward and penalty functions, exhibits superior performance. The trained control policies demonstrate a certain degree of generalization capability across different speed scenarios. Under various operating conditions, the maximum lateral position deviation outperforms classical control methods, while the maximum longitudinal velocity deviation remains lower than that of classical control methods. Moreover, under stable vehicle operation, the longitudinal velocity deviation is minimal and within an acceptable range.

On the other hand, computational efficiency stands as a significant indicator for assessing the performance of strategy algorithms. The time consumption for different control strategies during the operation of DDAVs in a dual-lane change scenario is illustrated in Figure 18. When conducting a 60 s DLC experiment at 18 km/h, the time consumed by MADRL is approximately 6% of that by PM and 4% of that by PP. At 36 km/h for a 45 s DLC experiment, MADRL consumes approximately 7% of PM’s time and 5% of PP’s time. Likewise, at 54 km/h for a 30 s DLC experiment, MADRL consumes approximately 13% of PM’s time and 10% of PP’s time. Hence, the trajectory tracking control method based on MADRL in this study exhibits higher computational efficiency and lower computational resource requirements compared with classical control methods.

4. Conclusions

This study focuses on DDAV and proposes a MADRL control method for autonomous driving trajectory tracking. The DDPG algorithm is employed for longitudinal control, while the DQN algorithm is utilized for lateral control. Guided by optimization principles, lower-level torque allocation is integrated, enabling iterative training to achieve DDAV trajectory tracking control. The specific conclusions are as follows: The MADRL method in this study enables more accurate tracking of desired longitudinal velocity and lateral position by vehicles. Additionally, the trained intelligent agent networks demonstrate good generalization capability. In lane-changing and dual-lane change experiments under different speed conditions, the maximum longitudinal velocity deviation, maximum lateral position deviation, and lateral overshoot are lower than those of classical control methods. The maximum lateral position deviation is improved by up to 90.5%, and the maximum longitudinal velocity deviation is improved by up to 97%. Moreover, the MADRL method in this study significantly improves computational efficiency, saving computational resources. Compared with classical control methods, the running time in dual-lane change experiments under different speed conditions can be reduced by up to 93.7%.

In summary, compared with classical control methods, the DDAV trajectory tracking control method based on MADRL in this study exhibits greater adaptability, better robustness, higher computational efficiency, and good generalization. Furthermore, the application of DRL in actual vehicle operation for continuous training and periodic policy iteration throughout the lifecycle significantly reduces reliance on high-precision vehicle mathematical models and high computational resources compared with mainstream control methods. This approach is beneficial for the development and application of advanced control strategies for intelligent driving vehicles. However, DRL endows machines with a certain degree of autonomy in understanding, learning, and decision-making. Nonetheless, advanced intelligent driving cannot disregard the necessity of driver operation under certain conditions, namely, the human–machine co-driving issue in the process of intelligent driving. Especially when external environmental changes occur, introducing human intervention may be necessary, as machines may not be able to react promptly. Confronting such issues in complex environments necessitates the integration of human intelligence with machine intelligence. Promising directions for future research include identifying how to incorporate human judgment and experience in a timely manner based on MADRL, adopting more advanced algorithms such as DDQN and PPO, integrating human and machine intelligence, enhancing the capability of human–machine interactions, and achieving superior human–machine co-driving coordination.