Two-Dimensional Car-Following Control Strategy for Electric Vehicle Based on MPC and DQN

Abstract

For the coupling problem of longitudinal control and lateral control of vehicles, a two-dimensional (2-D) car-following control strategy for an electric vehicle is proposed in this paper. First, a 2-D car-following model for longitudinal following and lateral lane keeping is established. Then, a 2-D car-following control strategy is designed, and the longitudinal following control and lateral lane keeping control are integrated into one model predictive control (MPC) framework. The 2-D car-following strategy can realize the multi-objective coordinated optimization for longitudinal control and lateral control during the 2-D car-following process, and the multiple objectives are: safety, tracking, comfort, lane keeping, lateral stability and economy. In addition, the economy is important for electric vehicles. The weight matrix of the objective function in the MPC framework is symmetric, and the weight coefficients for the weight matrix have a great influence on the control. The contribution of this paper is: in order to adapt to different dynamic processes of lane keeping, the weight coefficients in the MPC framework are optimized in real-time based on the deep Q network (DQN) algorithm. Finally, to verify the 2-D car-following control strategy, a comparison strategy and two experimental scenarios are set, and simulation experiments are carried out. In scenario 1, compared with the comparison strategy, the lane keeping, lateral stability and economy of the proposed strategy are improved by 37.21%, 17.57% and 9.26%, respectively. In scenario 2, compared with the comparison strategy, the lane keeping, lateral stability and economy of the proposed strategy are improved by 36.45%, 16.66% and 18.52%, respectively. Therefore, compared with the comparison strategy, the 2-D car-following control strategy can have better lane keeping, lateral stability and economy on the premise of ensuring other performances during the 2-D car-following process.

1. Introduction

With the development of the automotive industry, the electric vehicle [1] and driving control system (DCS) [2] have become two important vehicle technologies. The battery and motor are two important components of electric vehicles, and electric vehicles are powered by a battery and driven by a motor [3,4]. Therefore, electric vehicles have the characteristics of zero-emission and environmental friendliness. Electric vehicles have high requirements on the mileage of the battery on a single charge, and the mileage of the battery on a single charge is related to economy, so economy is particularly important for electric vehicles [5,6]. Among various DCSs, adaptive cruise control (ACC) has been widely used [7]. ACC can assist with the longitudinal driving of the vehicle, which can effectively reduce the driver’s driving workload and the psychological burden [8]. The working mode of ACC is divided into speed control mode and car-following mode [9]. In the speed control mode, there is no vehicle in front, and the following vehicle travels at a constant longitudinal speed [10]. In the car-following mode, the control process is more complicated and more meaningful because of the involvement of the following vehicle and the preceding vehicle [11]. Therefore, this paper takes the electric vehicle equipped with the ACC system as the research object and studies the car-following process of the electric vehicle. The economy of electric vehicles is fully considered.

The model predictive control (MPC) algorithm is widely used in the design of the car-following control strategies for electric vehicles. The MPC algorithm consists of three parts: predictive model, rolling optimization and feedback correction [12,13,14,15]. The multi-objective optimization problem can be converted to a quadratic programming problem under the MPC framework, and the optimal control variables can be obtained by solving the quadratic programming problem. The existing studies on the car-following control strategy of electric vehicles are mainly designed under MPC frameworks. In [16], an energy-optimized car-following control strategy for electric vehicles was designed under an MPC framework, which planned the longitudinal speed trajectory of the following vehicles in real-time. The traffic information and road conditions were obtained in advance so as to improve the economy. In [17], a car-following control strategy for connected electric vehicles was proposed. Under the MPC framework, the security and economy were optimized. For security, the safe following distance was obtained through the state invariant set theory. For the economy, the speed of the preceding vehicle was predicted through a long short-term memory network. In [18], a car-following control strategy that can improve the cruising range was proposed. Based on the MPC framework, the minimum safe distance was maintained with the preceding vehicle, and the speed reference of the following vehicle was calculated according to the status of the traffic lights, thereby improving the economy of the electric vehicle for the car-following process. In [19], a multi-objective car-following control strategy for electric vehicles was proposed. The strategy adopted a hierarchical control structure. The upper layer optimized the safety, following performance, comfort and economy under the MPC framework, and the lower layer applied regenerative braking to recover energy. In [20], a car-following control strategy applied to in-wheel motor electric vehicles was proposed. The Bayesian network was used to predict the motion state of the vehicle in front, and the control objective in the car-following process was optimized under the MPC framework. Therefore, on the premise of maintaining a safe distance from the vehicle in front, better economy can be obtained.

However, the existing car-following control strategies for electric vehicles mainly focus on longitudinal control and do not consider lateral control. Car-following control that only includes longitudinal control is one-dimensional car-following control. While the electric vehicle is following the preceding vehicle on a single curved lane, lateral control can be achieved with a lane keeping assistance system [21]. The longitudinal following control and lateral lane keeping control are coupled with each other during the driving process, so they also influence each other. When the vehicle is following in a straight lane, the longitudinal following control of the vehicle is mainly considered. When the vehicle is following on a single curved lane, the coupling of longitudinal following control and lateral lane keeping control needs to be considered. If the longitudinal following control and the lateral lane keeping control are optimized with two MPC frameworks, the longitudinal control and lateral control cannot interact with each other well. As a result, the longitudinal control and the lateral control are not accurate enough. Therefore, to follow a preceding vehicle on a single curved lane, it is necessary to optimize the multiple objectives related to longitudinal control and lateral control in one MPC framework. In addition, for the MPC framework, the weight matrix of the objective function is symmetric, and the weight coefficients in the weight matrix have a great influence on the control effect. While the electric vehicle is following the preceding vehicle on a single curved lane, the lateral lane keeping process changes all the time. If constant weight coefficients are adopted, it is difficult to adapt to different dynamic processes of lane keeping, resulting in a poor control effect. In order to adapt to different dynamic processes of lane keeping, the weight coefficients in the MPC framework need to be optimized in real-time.

Therefore, this paper proposes a two-dimensional (2-D) car-following control strategy. The 2-D car-following control strategy adopts a hierarchical control structure and contains upper-layer control and lower-layer control. Multiple control objectives related to the longitudinal and lateral directions are optimized under the MPC framework for the upper-layer control. The driving control, braking control and active steering control are achieved for the lower-layer control. The contribution of this paper is: in order to adapt to different dynamic processes of lane keeping, the weight coefficients in the MPC framework are optimized in real-time based on the deep Q network (DQN) algorithm [22].

The following content of this paper is organized as follows: The model of an electric vehicle is established in Section 2; the 2-D car-following model is established in Section 3; the 2-D car-following control strategy is designed in Section 4; the 2-D car-following control strategy is verified in Section 5; the conclusions are obtained in Section 6.

2. Model of Electric Vehicle

The structure of the target electric vehicle is shown in Figure 1, and the target electric vehicle is a front-drive electric vehicle. The electric vehicle obtains information from the environment through millimeter-wave radar and cameras. Among them, the millimeter wave radar is mainly used to detect the motion state of the preceding vehicle, and the camera is mainly used to detect the lane line. In an electric vehicle model, the motor and battery are two important components.

The motor is a permanent magnet synchronous motor, and the motor model is established with the torque and speed characteristics of the motor [23]. The working state of the motor is divided into two states: driving and braking, and the motor efficiency is considered in the model. During motor modeling, the input and output characteristics of the motor need to be focused on, and the complex dynamic characteristics of the motor can be appropriately simplified. The integrated expression of the motor controller and the motor body is its input and output characteristics, so the motor and the motor controller are usually integrated together for modeling during the modeling process. The torque and speed characteristics of the motor are shown in Figure 2, and motor torque corresponds to driving torque and braking torque. The battery is a lithium battery, and the battery model is established with the equivalent internal resistance model [24]. The working state of the battery is divided into two states of charging and discharging, and the efficiency of the battery is considered in the model. The equivalent internal resistance model of the battery is shown in Figure 3.

During braking, electric vehicles can recover energy through regenerative braking. The regenerative braking control strategy contains the distribution of braking force for motors and hydraulics, as well as the distribution of braking force for the front axle and rear axle. The control strategy for regenerative braking in [19] is adopted in this paper.

Figure 2. Torque–speed characteristics of the motor.

Figure 2. Torque–speed characteristics of the motor.

Figure 3. Equivalent internal resistance model of battery.

Figure 3. Equivalent internal resistance model of battery.

3. Model of 2-D Car-Following Process

The longitudinal control and lateral control of the vehicle are coupled. Figure 4 shows the 2-D car-following process. In the 2-D car-following model, it is necessary to consider the longitudinal following process and lateral lane keeping process. Therefore, the 2-D car-following model can better reflect the longitudinal and lateral dynamic characteristics of the vehicle during the 2-D car-following process.

As shown in Figure 5, it is the longitudinal car-following process and the longitudinal motion relationship between two vehicles can be presented. The mathematical form of the longitudinal car-following process [19] is described as follows:

Figure 5. Longitudinal car-following process.

Figure 5. Longitudinal car-following process.

$$\{\begin{array}{l}\Delta s(k+1)=\Delta s(k)+v_{rel}(k)T_s+\frac{1}{2}{a}_{fx}(k)T_s{}^2-\frac{1}{2}{a}_x(k)T_s{}^2\hfill \\ v_x(k+1)=v_x(k)+a_x(k)T_s\hfill \\ v_{rel}(k+1)=v_{fx}(k+1)-v_x(k+1)\hfill \\ a_x(k+1)=(1-\frac{T_s}{{\tau }_l})a_x(k)+\frac{T_s}{{\tau }_l}{a}_{xdes}(k)\hfill \\ j_x(k+1)=-\frac{1}{{\tau }_l}{a}_x(k)+\frac{1}{{\tau }_l}{a}_{xdes}(k)\hfill \end{array}$$

(1)

The longitudinal car-following process is described in the form of the state equation as follows:

$$x_1(k+1)=A_1{x}_1(k)+B_1{u}_1(k)+G_1{w}_1(k)$$

(2)

where

$$x_1(k)={\left[\begin{array}{lllll}\Delta s(k)\hfill & v_x(k)\hfill & v_{rel}(k)\hfill & a_x(k)\hfill & j_x(k)\hfill \end{array}\right]}^T$$

$$u_1(k)={\left[a_{xdes}(k)\right]}^T$$

$$w_1(k)={\left[a_{fx}(k)\right]}^T$$

$$A_1=\left[\begin{array}{ccccc}1& 0& T_s& -\frac{1}{2}{T}_s{}^2& 0\\ 0& 1& 0& T_s& 0\\ 0& 0& 1& -T_s& 0\\ 0& 0& 0& 1-\frac{T_s}{{\tau }_l}& 0\\ 0& 0& 0& -\frac{1}{{\tau }_l}& 0\end{array}\right]$$

$$B_1={\left[\begin{array}{lllll}0\hfill & 0\hfill & 0\hfill & \frac{T_s}{{\tau }_l}\hfill & \frac{1}{{\tau }_l}\hfill \end{array}\right]}^T$$

$$G_1={\left[\begin{array}{lllll}\frac{1}{2}{T}_s{}^2\hfill & 0\hfill & T_s\hfill & 0\hfill & 0\hfill \end{array}\right]}^T$$

where at time k, state variable x₁(k) is composed of the distance between vehicles Δs(k), longitudinal speed v_x(k), relative speed v_rel(k), longitudinal acceleration a_x(k) and jerk j_x(k); control variable u₁(k) is composed of the desired longitudinal acceleration a_xdes; system disturbance variable w₁(k) is composed of the longitudinal acceleration of the preceding vehicle a_fx. In addition, T_s is the sampling time and ${\tau }_l$is the time lag.

The is the lateral lane keeping process is shown in Figure 6, and the lateral motion relationship and the lateral dynamic characteristics of the following vehicle can be presented. The mathematical form of the lateral lane keeping process [25] is described as follows:

Figure 6. Lateral lane keeping process.

Figure 6. Lateral lane keeping process.

$$\begin{array}{ll}{M}_{veh}{\ddot{e}}_s=& {\dot{e}}_s\left[-\frac{2C_{\alpha f}}{v_x}-\frac{2C_{\alpha r}}{v_x}\right]+e_{\alpha }\left[2C_{\alpha f}+2C_{\alpha r}\right]\\ & +{\dot{e}}_{\alpha }\left[-\frac{2C_{\alpha f}{l}_1}{v_x}+\frac{2C_{\alpha r}{l}_2}{v_x}\right]\\ & +{\dot{\Psi }}_{des}\left[-\frac{2C_{\alpha f}{l}_1}{v_x}+\frac{2C_{\alpha r}{l}_2}{v_x}-v_x\right]+2C_{\alpha f}{\delta }_f\end{array}$$

(3)

$$\begin{array}{ll}{I}_z{\ddot{e}}_{\alpha }=& 2C_{\alpha f}{l}_1{\delta }_f+{\dot{e}}_s\left[-\frac{2C_{\alpha f}{l}_1}{v_x}-\frac{2C_{\alpha r}{l}_2}{v_x}\right]\\ & +e_{\alpha }\left[2C_{\alpha f}{l}_1-2C_{\alpha r}{l}_2\right]+{\dot{e}}_{\alpha }\left[-\frac{2C_{\alpha f}{l}_1{}^2}{v_x}-\frac{2C_{\alpha r}{l}_2{}^2}{v_x}\right]\\ & +{\dot{\Psi }}_{des}\left[-\frac{2C_{\alpha f}{l}_1{}^2}{v_x}-\frac{2C_{\alpha r}{l}_2{}^2}{v_x}\right]\end{array}$$

(4)

The lateral lane keeping process is described in the form of the state equation with approximate discretization as follows:

$$x_2(k+1)=A_2{x}_2(k)+B_2{u}_2(k)+G_2{w}_2(k)$$

(5)

where

$$x_2(k)={\left[\begin{array}{cccc}{e}_s(k)& {\dot{e}}_s(k)& e_{\alpha }(k)& {\dot{e}}_{\alpha }(k)\end{array}\right]}^T$$

$$u_2(k)={\left[{\delta }_f(k)\right]}^T$$

$$w_2(k)={\left[{\dot{\Psi }}_{des}(k)\right]}^T$$

$$A_2=\left[\begin{array}{cccc}1& T_s& 0& 0\\ 0& 1-\frac{2C_{\alpha f}+2C_{\alpha r}}{M_{veh}{v}_x}{T}_s& \frac{2C_{\alpha f}+2C_{\alpha r}}{M_{veh}}{T}_s& -\frac{2C_{\alpha f}{l}_1-2C_{\alpha r}{l}_2}{M_{veh}{v}_x}{T}_s\\ 0& 0& 1& T_s\\ 0& -\frac{2C_{\alpha f}{l}_1-2C_{\alpha r}{l}_2}{I_z{v}_x}{T}_s& \frac{2C_{\alpha f}{l}_1-2C_{\alpha r}{l}_2}{I_z}{T}_s& 1-\frac{2C_{\alpha f}{l}_1{}^2+2C_{\alpha r}{l}_2{}^2}{I_z{v}_x}{T}_s\end{array}\right]$$

$$B_2={\left[\begin{array}{cccc}0,& \frac{2C_{af}}{M_{veh}}{T}_s,& 0,& \frac{2C_{af}{l}_1}{I_z}{T}_s\end{array}\right]}^T$$

$$G_2={\left[\begin{array}{cccc}0,& -\frac{2C_{\alpha f}{l}_1-2C_{\alpha r}{l}_2}{M_{veh}{v}_x}{T}_s-v_x{T}_s,& 0,& -\frac{2C_{\alpha f}{l}_1{}^2+2C_{\alpha r}{l}_2{}^2}{I_z{v}_x}{T}_s\end{array}\right]}^T$$

where at time k, state variable x₂(k) is composed of the lateral distance deviation e_s(k), derivative of lateral distance deviation $\dot{e}$_s(k), directional deviation e_α(k) and derivative of directional deviation $\dot{e}$_α(k); control variable u₂(k) is composed of the targeted turning angle δ_f; system disturbance variable w₂(k) is composed of the desired yaw rate $\dot{\Psi }$_des. M_veh is the mass of the electric vehicle; I_z is the moment of inertia of the electric vehicle; C_αf and C_αr are the cornering stiffness of the front and rear wheels, respectively; l₁ and l₂ are the distances from the center of mass to the front and rear axles, respectively.

With the models in [19,25], the 2-D car-following model is established. The equation for the 2-D car-following model is described as follows:

$$x(k+1)=Ax(k)+Bu(k)+Gw(k)$$

(6)

where

$$x(k)={\left[\begin{array}{lllllllll}\Delta s(k)\hfill & v_x(k)\hfill & v_{rel}(k)\hfill & a_x(k)\hfill & j_x(k)\hfill & e_s(k)\hfill & {\dot{e}}_s(k)\hfill & e_{\alpha }(k)\hfill & {\dot{e}}_{\alpha }(k)\hfill \end{array}\right]}^T$$

$$u(k)={\left[\begin{array}{cc}{a}_{xdes}(k)& {\delta }_f(k)\end{array}\right]}^T$$

$$w(k)={\left[\begin{array}{cc}{a}_{fx}(k)& {\dot{\Psi }}_{des}(k)\end{array}\right]}^T$$

$$A=\left[\begin{array}{c}{A}_1\\ 0\end{array}\begin{array}{c}0\\ A_2\end{array}\right]$$

$$B=\left[\begin{array}{c}{B}_1\\ 0\end{array}\begin{array}{c}0\\ B_2\end{array}\right]$$

$$G=\left[\begin{array}{c}{G}_1\\ 0\end{array}\begin{array}{c}0\\ G_2\end{array}\right]$$

4. Two-Dimensional Car-Following Control Strategy

4.1. Hierarchical Control Structure

As shown in Figure 7, the hierarchical control structure is adopted in the 2-D car-following control strategy, including the upper-layer controller and the lower-layer controller. The longitudinal and lateral control are coupled with each other in the upper-layer controller. The longitudinal speed is input into the lateral control in real-time to affect the lateral performance, and the lateral acceleration is input into the longitudinal control in real-time to constrain the magnitude of the longitudinal acceleration. The lower-layer controller contains the driving control algorithm, braking control algorithm and active steering control algorithm. The focus of this paper is on the upper-layer controller.

The vehicle state and lane centerline information are obtained through sensors, thereby establishing a 2-D car-following model. In the upper-layer controller, the longitudinal following control and lateral lane keeping control are integrated into an MPC framework, and the control objectives include safety, following performance, comfort, lane keeping, lateral stability and economy. In order to smooth the system response characteristics, a reference trajectory is set with an exponential decay function. To solve the problem for selecting the weight coefficients in the MPC framework, an optimization algorithm for weight coefficients based on the DQN algorithm is proposed. The optimal weight coefficients are obtained through the trained deep neural network in real-time so as to better adapt to different dynamic processes of lane keeping. The upper-layer controller obtains the optimal control variables through rolling optimization and the optimal control variables correspond to the desired longitudinal acceleration and the targeted turning angle.

In the lower-layer controller, the driving state is determined through the state transformation logic. The driving and braking control algorithms are designed based on the PID algorithm with feedforward and feedback functions. The motor and hydraulic braking system are controlled through driving and braking control algorithms. The active steering actuator for front wheels is controlled through the active steering control algorithm.

Figure 7. Hierarchical control structure for the 2-D car-following process.

Figure 7. Hierarchical control structure for the 2-D car-following process.

4.2. Multi-Objective Optimization MPC Algorithm

In the 2-D car-following process, to facilitate the optimization of multiple objectives for longitudinal following and lateral lane keeping, the corresponding performance variables are set as follows:

$$y(k)={\left[\begin{array}{cccc}{\delta }_s(k)& v_{rel}(k)& a_x(k)& \begin{array}{cccc}{j}_x(k)& e_s(k)& {\dot{e}}_s(k)& \begin{array}{cc}{e}_{\alpha }(k)& {\dot{e}}_{\alpha }(k)\end{array}\end{array}\end{array}\right]}^T$$

(7)

where δ_s is the error of distance between vehicles, and δ_s is calculated as follows:

$${\delta }_s(k)=\Delta s(k)-(v_x{t}_h+d_0)$$

(8)

where t_h is the time headway, and d₀ is the safe vehicle distance, which is the sum of the minimum distance between the vehicles and a body length.

The performance variables can be calculated based on the state variables in Equation (4):

$$y(k)=Cx(k)-Z$$

(9)

where

$$C=\left[\begin{array}{c}{C}_1\\ 0\end{array}\begin{array}{c}0\\ C_2\end{array}\right]$$

$$C_1=\left[\begin{array}{ccccc}1& -t_h& 0& 0& 0\\ 0& 0& 1& 0& 0\\ 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 1\end{array}\right]$$

$$C_2=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$$

$$Z=\left[\begin{array}{c}{Z}_1\\ 0\end{array}\right]$$

$$Z_1={\left[\begin{array}{llll}{d}_0\hfill & 0\hfill & 0\hfill & 0\hfill \end{array}\right]}^T$$

To ensure tracking, it is necessary to minimize the error in distance between vehicles and relative speed; to ensure comfort, it is necessary to minimize longitudinal acceleration and jerk; to minimize energy consumption, it is necessary to minimize desired longitudinal acceleration [26]; to ensure lane keeping, it is necessary to minimize lateral distance deviation and directional deviation [19]; in order to ensure lateral stability, the derivatives of lateral distance deviation and directional deviation need to be minimized, and the turning angle needs to be minimized [25]. Therefore, to optimize the multiple objectives in the 2-D car-following process, the performance variables and control variables need to be minimized:

$$\mathrm{Objectives}:\{\begin{array}{c}\min |y(k)|\\ \min |u(k)|\end{array}$$

(10)

In order to smooth the response characteristics of the system, a reference trajectory for the performance variables is set with the exponential decay function. The reference trajectory is described as:

$$y_{ref}(k+i)={\phi }^{i}y(k)$$

(11)

where

$$\phi =\left[\begin{array}{cc}{\phi }_1& 0\\ 0& {\phi }_2\end{array}\right]$$

$${\phi }_1=\left[\begin{array}{cccc}{\rho }_{\delta }& 0& 0& 0\\ 0& {\rho }_v& 0& 0\\ 0& 0& {\rho }_a& 0\\ 0& 0& 0& {\rho }_j\end{array}\right]$$

$${\phi }_2=\left[\begin{array}{cccc}{\rho }_{e_s}& 0& 0& 0\\ 0& {\rho }_{{\dot{e}}_s}& 0& 0\\ 0& 0& {\rho }_{e_{\alpha }}& 0\\ 0& 0& 0& {\rho }_{{\dot{e}}_{\alpha }}\end{array}\right]$$

where ρ_δ, ρ_v, ρ_a, ρ_j, ${\rho }_{e_s}$, ${\rho }_{{\dot{e}}_s}$, ${\rho }_{e_{\alpha }}$ and ${\rho }_{{\dot{e}}_{\alpha }}$ are the coefficients for the δ_s, v_rel, a_x, j_x, e_s, $\dot{e}$_s, e_α and $\dot{e}$_α in reference trajectories, respectively.

In the 2-D car-following process, for the longitudinal following control, it is necessary to ensure safety, and for the lateral lane keeping, it is necessary to set corresponding constraints on the targeted turning angle; the vehicle is limited by its own capability. The constraints are as follows:

$$s.t.\{\begin{array}{l}\begin{array}{l}\Delta s(k)\ge d_c\hfill \\ v_{x\min }\le v_x(k)\le v_{x\max }\hfill \end{array}\hfill \\ a_{x\min }\le a_x(k)\le a_{x\max }\hfill \\ j_{x\min }\le j_x(k)\le j_{x\max }\hfill \\ \begin{array}{l}{u}_{1\min }\le u_1(k)\le u_{1\max }\\ u_{2\min }\le u_2(k)\le u_{2\max }\end{array}\hfill \end{array}$$

(12)

where u₁ and u₂ are used to represent the desired longitudinal acceleration and the targeted turning angle, respectively.

For the stability of the vehicle, the lateral acceleration and longitudinal acceleration need to meet the following conditions:

$$a_{veh}=\sqrt{a_x{}^2+a_y{}^2}\le {\mu }_{\max }g$$

(13)

where a_y is the lateral acceleration, a_veh is the resultant acceleration of an electric vehicle and μ_max is the maximum adhesion coefficient of vehicle tires and ground.

In order to ensure a certain adhesion margin, the above equation is modified into the following form:

$$a_{veh}=\sqrt{a_x{}^2+a_y{}^2}\le {\mu }_{\max }g-\epsilon$$

(14)

where ε is the coefficient for attachment margin, which is set to 1.

Converting the above equation results in the following equation:

$$-\sqrt{{({\mu }_{\max }g-\epsilon )}^2-a_y{}^2}\le a_x\le \sqrt{{({\mu }_{\max }g-\epsilon )}^2-a_y{}^2}$$

(15)

In the 2-D car-following control, the maximum and minimum of u₁ are set as follows:

$$u_{1\min }(k)=\max (u_{1low},-\sqrt{{({\mu }_{\max }g-\epsilon )}^2-a_y{(k)}^2})$$

(16)

$$u_{1\max }(k)=\min (u_{1up},\sqrt{{({\mu }_{\max }g-\epsilon )}^2-a_y{(k)}^2})$$

(17)

where u_1up and u_1low are the maximum and minimum of the control variable for desired longitudinal acceleration without lateral lane keeping.

Longitudinal control and lateral control are integrated into an MPC framework, and the objective function is calculated as follows:

$$\begin{array}{ll}J=& {\displaystyle \sum _{j=1}^p{{\displaystyle [{\widehat{y}}_p(k+j/k)-y_{ref}(k+j)]}}^T}Q[{\widehat{y}}_p(k+j/k)-y_{ref}(k+j)]\\ & +{\displaystyle \sum _{j=0}^{m-1}u(k+j)}Ru(k+j)\end{array}$$

(18)

where p and m are the time domains for prediction and control, respectively.

Under the MPC framework, combining Equation (13) and Constraint 7, the optimization problem for multiple objectives can be converted into a constrained quadratic programming problem. By solving the quadratic programming problem, the optimal control sequence can be acquired. However, merely the first value in the control sequence is applied to the ACC system. The above optimization is repeated at the next sampling time.

The weight matrices Q and R in the objective function can be described as:

$$Q=\left[\begin{array}{llllllll}{w}_{\delta }(k)\hfill & \hfill & \hfill & \hfill & \hfill & \hfill & \hfill & \hfill \\ \hfill & w_v(k)\hfill & \hfill & \hfill & \hfill & \hfill & \hfill & \hfill \\ \hfill & \hfill & w_a(k)\hfill & \hfill & \hfill & \hfill & \hfill & \hfill \\ \hfill & \hfill & \hfill & w_j(k)\hfill & \hfill & \hfill & \hfill & \hfill \\ \hfill & \hfill & \hfill & \hfill & w_{e_s}(k)\hfill & \hfill & \hfill & \hfill \\ \hfill & \hfill & \hfill & \hfill & \hfill & w_{{\dot{e}}_s}(k)\hfill & \hfill & \hfill \\ \hfill & \hfill & \hfill & \hfill & \hfill & \hfill & w_{e_{\alpha }}(k)\hfill & \hfill \\ \hfill & \hfill & \hfill & \hfill & \hfill & \hfill & \hfill & w_{{\dot{e}}_{\alpha }}(k)\hfill \end{array}\right]$$

(19)

$$R=\left[\begin{array}{cc}{w}_{u_1}(k)& \\ & w_{u_2}(k)\end{array}\right]$$

(20)

where, at sampling time k, w_δ(k), w_v(k), w_a(k), w_j(k), $w_{e_s}\left(k\right)$, $w_{{\dot{e}}_s}\left(k\right)$,$w_{e_{\alpha }}\left(k\right)$ and $w_{{\dot{e}}_{\alpha }}\left(k\right)$ are the weights of error of the distance between vehicles, relative speed, longitudinal acceleration, jerk, lateral distance deviation, derivatives of lateral distance deviation, directional deviation and derivatives of directional deviation, respectively. In addition, $w_{u_1}\left(k\right)$ and $w_{u_2}\left(k\right)$ are the weight coefficients for the desired longitudinal acceleration and targeted turning angle.

As shown in

Table 1. The parameters of the MPC framework.

Symbol	Value	Symbol	Value
Ts	0.05 s	ρj	0.94
th	1.5 s	Mveh	1550 kg
τ	0.15 s	l1	1.1 m
d0	7 m	l2	1.58 m
dc	5 m	Iz	2873 kg m3
vxmin	0 m/s	Cαf	80 KN/rad
vxmax	36 m/s	Cαr	80 KN/rad
axmin	−5.5 m/s2	u2min	−5 deg
axmax	2.5 m/s2	u2max	5 deg
u1min	−5.5 m/s2	${\rho }_{e_s}$	0.6
u1max	2.5 m/s2	${\rho }_{{\dot{e}}_s}$	0.6
jxmin	−3 m/s3	${\rho }_{e_{\alpha }}$	0.6
jxmax	3 m/s3	${\rho }_{{\dot{e}}_{\alpha }}$	0.6
R	diag (1,1)	p	10
ρδ	0.94	m	5
ρv	0.94	T	50 s
ρa	0.94	-	-

, the parameters in the MPC framework are set.

4.3. Weight Optimization Algorithm Based on DQN

In the 2-D car-following control, it is necessary to consider the longitudinal following and the lateral lane keeping at the same time. The MPC framework with constant weight is difficult to adapt to different dynamic processes of lane keeping. The weights of the algorithm need to be optimized in real-time.

Only the weight coefficients in the Q₃ matrix are optimized in real-time, and the weight coefficients w_u1(k) and w_u2(k) in the R₃ matrix are set to a constant value of 1. The purpose of not adjusting R₃ is to ensure that the relative values among the weights are all based on w_u1(k) and w_u2(k). Since weight matrix Q₃ contains eight weight coefficients, and some of the weight coefficients interact with each other, it is difficult to optimize the weight coefficients by traditional modeling methods.

For this reason, the weight coefficients are optimized through the method based on reinforcement learning optimized in real-time. The framework of reinforcement learning and the framework of the DQN algorithm is introduced in this paper first, and then the weight optimization algorithm is designed based on the above two frameworks.

Reinforcement learning, as a branch of machine learning, derives its ideas from behavioral psychology [27]. In reinforcement learning, the agent realizes the learning process by interacting with the environment [28]. The goal of the learning process is to obtain the optimal policy by maximizing the expected value of the accumulated reward. Figure 8 shows the framework of reinforcement learning.

Reinforcement learning can be modeled by a Markov decision process (MDP) [28]; MDP is described as a tuple with five components:

$$M=(S,A,P,R,\gamma )$$

(21)

where S is the state set, A is the action set, P is the state transition function, R is the reward function, r(s,a) is the reward value obtained by the agent that takes action a in state s, γ is the discount factor of the reward and γ ∈ (0,1).

The action-value function is the expected reward with s and a. If the agent operates with the policy π, the action-value function is defined as follows:

$$Q^{\pi }(s,a)={\rm E}[{\displaystyle \sum _{t=0}^{\infty }{\gamma }^t{r}_t}]$$

(22)

where

$$r_t=r(s_t,a_t),a_t=\pi (s_t)$$

$$s_{t+1}~P(s_t,a_t)$$

$$s_0=s,a_0=a$$

where t represents the time step.

The optimal policy π* is calculated as follows:

$${\pi }^{*}=\underset{\prod }{\mathrm{argmax}}{Q}^{\pi }(s,a)$$

(23)

where Π is the policy space.

The optimal action-value function is described as follows:

$$Q^{*}(s,a)=Q^{\pi *}\left(s,a\right)={{\rm E}}_{s^{\prime }~P(s,a)}[r+\gamma \underset{a^{\prime }}{\max }{Q}^{*}(s^{\prime },a^{\prime })]$$

(24)

During the solution process of MDP, the state transition function and action-value function are unknown, but the observed sequence (s_t, a_t, s_t+1, r_t) can be obtained from the MDP. The observed sequence is also called the trajectory, and the trajectory can be applied to estimate the action-value function. Most solvers of MDP are implemented based on the iterative estimation of the optimal action-value function. Among them, the calculation process of the Q-learning algorithm [29] is as follows:

$$Q^{l+1}(s_t,a_t)\leftarrow Q^l(s_t,a_t)+{\alpha }_t[r_{t+1}+\gamma \underset{a^{\prime }}{\max }{Q}^l(s_t,a_t)-Q^l(s_t,a_t)]$$

(25)

where l is the index, α_t is the learning rate at step t and the action a_t is selected in a semi-random manner for the ε-greedy strategy.

When the scale of the spaces for state and action are large, the accuracy of the Q-learning algorithm cannot be guaranteed, so the method of parameter fitting is applied. Deep neural networks can be applied to fit action-value functions, and the DQN algorithm is an example.

The framework of the DQN algorithm is shown in Figure 9. In addition, the DQN algorithm contains two Q networks, namely the main network and the target network. The main network is used to predict the optimal action for each step, and the parameter is θ; the target network is applied to acquire the target of the gradient descent step, and the parameter is $\widehat{\theta }$. The difference between the estimation and the target for the action-value function can be expressed in the form of a loss function so that the solution of the MDP is transformed to calculate parameter θ to minimize the loss function. The loss function of the DQN algorithm is set as:

$$L(\theta )={{\rm E}}_{(s,a,r,s^{\prime })~U(D)}[{(r+\gamma \underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime };\widehat{\theta })-Q(s,a;\theta ))}^2]$$

(26)

where (s, a, r, s′)~U(D) means a random and uniform sample drawn from the empirical pool.

The DQN algorithm uses the experience replay and target network to solve instability and divergence problems [30]. In the experience replay, the observed sequence (s_t, a_t, r_t, s_t+1) is stored in the replay buffer, the correlation between the data is eliminated by random sampling, and the data distribution is smooth. Observed sequences are also called experiences. In the target network, the network only keeps individual parameters for the network and is regularly updated to reduce the correlation between the estimated value and the target value of the action-value function.

The selection of weight coefficients in the weight matrix is converted into an MDP first, and then the weight coefficients are optimized based on the DQN algorithm. The state s_t and action a_t of the MDP are defined as follows:

$$s_t=(\begin{array}{llllllll}{\delta }_{s,t}\hfill & v_{rel,t}\hfill & a_{x,t}\hfill & j_{x,t}\hfill & e_{s,t}\hfill & {\dot{e}}_{s,t}\hfill & e_{\alpha ,t}\hfill & {\dot{e}}_{\alpha ,t}\hfill \end{array})$$

(27)

$$a_t=\left(\begin{array}{llllllll}{w}_{{\delta }_s,t}\hfill & w_{v_{rel},t}\hfill & w_{a_x,t}\hfill & w_{j_x,t}\hfill & w_{e_s,t}\hfill & w_{{\dot{e}}_s,t}\hfill & w_{e_{\alpha },t}\hfill & w_{{\dot{e}}_{\alpha },t}\hfill \end{array}\right)$$

(28)

When the vehicle is driving on a straight road, the longitudinal following control should be given priority, and greater weight should be applied to the weight coefficients related to longitudinal following under the MPC framework. When the vehicle is driving on a curved road, the lateral lane keeping control should be given priority, and greater weight is applied to the weight coefficients related to lane keeping under the MPC framework. Based on the above analysis and combined with trial-and-error experiments, the reward settings for the t time step are as follows:

$$\begin{array}{ll}{r}_t=& -(5{({\delta }_{s,t})}^2+5{(v_{rel,t})}^2+50{(a_{x,t})}^2+50{(j_{x,t})}^2)*0.001\\ & -(50{(e_{s,t})}^2+250{(e_{\alpha ,t})}^2+50{({\dot{e}}_{s,t})}^2+250{({\dot{e}}_{\alpha ,t})}^2)*0.001\\ & -10{\zeta }_1+2{\zeta }_2+{\zeta }_3\end{array}$$

(29)

where ζ₁, ζ₂ and ζ₃ are the fine-tuning coefficients. If the simulation ends, ζ₁ = 1, otherwise ζ₁ = 0; if v_rel² < 1, ζ₂ = 1, otherwise ζ₂ = 0; if e_s² < 0.01, ζ₃ = 1, otherwise ζ₃ = 0.

For the weight optimization algorithm, the network in DQN consists of the input layer, the hidden layer and the output layer. Among them, the hidden layer consists of three layers. In each hidden layer, the number of neurons is 48, 96 and 48, respectively. In the output layer, the activation function ReLU is adopted. Since the weight coefficients are all greater than zero, the output of the activation function for the output layer needs to be modified. The minimum value of each weight coefficient mapped by the activation function is greater than or equal to 10⁻⁴. The DQN algorithm adopts an offline training process; a single agent interacts with the environment and trains the neural network according to the state, reward function and action. When the training is completed, no more learning is required. The neural network directly outputs the weight coefficients of the Q₃ matrix according to the state.

The maximum iteration round of the DQN algorithm is set to 12,000, the learning rate is set to 0.01, the capacity of the replay buffer is 2000, the size of the batch is 32, the discount factor in the reward function is set to 0.95 and the parameters for the target network and main network are synchronized every 10 iteration rounds, the initial ε of the ε-greedy policy is set to 0.99 and then decays linearly, in turn, with each iteration round.

5. Simulation Experiment and Analysis

5.1. Experimental Method

In the 2-D car-following process, the control objectives contain safety, following performance, comfort, lane keeping, lateral stability and economy. The strategy proposed in this paper is the target strategy, which is abbreviated as Integ_MPC for the convenience of the following description.

In order to verify the target strategy, a comparison strategy is set up. The comparison strategy adopts two MPC frameworks to achieve longitudinal and lateral control. The longitudinal control adopts the longitudinal following strategy in [19], and the lateral control adopts the lateral lane keeping strategy in [25]. For the convenience of the following description, the comparison strategy is abbreviated as: MPC1_MPC2.

In order to evaluate the target strategy, the following evaluation criteria are set: (1) whether the distance between the vehicles is greater than the minimum safe distance (5 m) is regarded as the judgment standard for vehicle safety; (2) following performance is judged by the root mean square estimation for the error of distance between vehicles and relative speed [31]; (3) the comfort is judged by whether the maximum value of the absolute value of the jerk is less than 3 m/s³; (4) lane keeping is judged by root mean square estimation of the horizontal and vertical coordinate deviation; (5) the lateral stability is judged by the root mean square estimation of the sideslip angle of the centroid, lateral acceleration, turning angle and yaw rate; (6) the economy is judged by the ratio of the change in SOC and the distance.

In order to facilitate the analysis of the following performance and lane keeping, the horizontal and vertical coordinate deviation in the geodetic coordinate system Δ_XY and root mean square estimation (RMSE) [31] are introduced as follows:

$${\Delta }_{XY}(i)=\sqrt{{(X(i)-X_{ref}(i))}^2+{(Y(i)-Y_{ref}(i))}^2}$$

(30)

$$RMS{E}_{var}=\sqrt{\frac{1}{n_{tot}}{\displaystyle \sum _{i=1}^{n_{tot}}{(var(i))}^2}}$$

(31)

$$n_{tot}=\frac{T}{T_s}$$

(32)

where X and Y denote the actual horizontal and vertical coordinates for the centroid of the vehicle, respectively; X_ref and Y_ref denote the referenced horizontal and vertical coordinates for the centroid of the vehicle; var(i) denotes the various variables at moment i, the error for the distance between vehicles is δ_s, the relative speed is v_rel and the lateral and longitudinal distance deviation is Δ_XY, the sideslip angle of the centroid is β, lateral acceleration is a_y, turning angle is δ_f, yaw rate is $\dot{\psi }$ and n_tot is the number of calculations.

In order to verify the target strategy, the following experimental scenarios are set up: (1) tracking a preceding vehicle with a time-varying speed; (2) lane change insertion of the preceding vehicle. The specific settings of the scenarios are shown in

Table 2. Experiment scenario setting.

Scenarios	ini_Δs (m)	ini_vx (m/s)	ini_vf (m/s)	amp_ax (m/s2)	μ
scenario1	50	20	25	2	0.45
scenario2	70	25	20	2	0.45

ini_Δs is the initial distance between vehicles, ini_v_x is the initial longitudinal speed for the following vehicle, ini_v_f is the initial longitudinal speed for the vehicle in front, amp_a is the amplitude of longitudinal acceleration for the preceding vehicle and μ is the ground adhesion coefficient.

.

The complete lane centerline is shown in Figure 10, which contains a straight road with a length of 200 m and a curve with 18 arcs. The first arc is tangent to the X-axis, and the radii of the arcs are different. The radius setting of the arc comprehensively considers the constraints of the lateral acceleration and longitudinal speed of the vehicle.

The control strategy in the simulation experiment is constructed in Matlab/Simulink, and the vehicle dynamics and driving scenarios are constructed in CarSim. In addition, the DQN algorithm is constructed by Python and PyTorch, and real-time data transmission is realized through the communication module. The experimental software and hardware environment are shown in

Table 3. Software and hardware for the experiment.

Name	Property
Matlab	2018a
CarSim	2016.1
CPU	Intel Core i7-4790 (3.60 GHz)
GPU	NVIDIA TITAN V
Memory	32.00 GB (3200 MHz)
Operating system	Windows 10 (64-bit)
Python	3.8.8
PyTorch	1.7.1
CUDA	10.1

.

5.2. Analysis of Results

5.2.1. Scenario 1

In scenario 1, the vehicle follows a preceding vehicle whose speed varies over time. In this scenario, the motion state of the vehicle in front changes at all times. The vehicle is approaching from a long distance. In order to track the desired distance between vehicles and the longitudinal speed of the preceding vehicle, the following vehicle changes with the vehicle in front. The experimental results of this scenario are shown in Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15.

The resultant acceleration of scenario 1 is shown in Figure 11. The resultant acceleration is the vector sum of the longitudinal acceleration and the lateral acceleration. This paper mainly analyzes the numerical change in the resultant acceleration. In the 2-D car-following control, the lateral motion and the longitudinal motion influence each other. The lateral acceleration is input into the longitudinal control to limit the magnitude of the longitudinal acceleration in real-time so that the magnitude of the resultant acceleration is maintained within a reasonable range. There are two upper limits for the resultant acceleration, which are μg and μg-ε. Among them, μg is the maximum upper limit of the resultant acceleration. In order to ensure a certain adhesion margin, the upper limit is corrected to μg-ε. In the 2-D car-following process, compared with MPC1_MPC2, Integ_MPC has a smaller fluctuation range of the resultant acceleration. In addition, the resultant accelerations of the two strategies are always smaller than the revised upper limit value, thus ensuring that the vehicle can run stably during the car-following process, which is the premise of the performance optimization for the 2-D car-following process.

Figure 11. Resultant acceleration in scenario 1.

Figure 11. Resultant acceleration in scenario 1.

Figure 12 shows the responses of longitudinal motion in scenario 1, including the distance between vehicles, longitudinal speed, longitudinal acceleration and jerk.

Figure 12a shows the change in distance between vehicles, where Integ_MPC_∆s and MPC1_MPC2_∆s are the distances between vehicles for Integ_MPC and MPC1_MPC2, respectively. Integ_MPC_∆s_des and MPC1_MPC2_∆s_des are the desired distances between vehicles for Integ_MPC and MPC1_MPC2, respectively. In the whole experiment process, the minimum distances between vehicles of both strategies are larger than the minimum safe distance (5 m), so the safety in the two strategies is guaranteed during the 2-D car-following process.

Following performance contains the tracking of the actual distance to the desired distance between vehicles and the tracking of the longitudinal speed of the following vehicle to the longitudinal speed of the vehicle in front. In terms of tracking the desired distance between vehicles, the root mean square estimation for Integ_MPC and MPC1_MPC2 are: 6.63 and 8.27 m, respectively; in terms of tracking the longitudinal speed of the preceding vehicle, the root mean square estimation for Integ_MPC and MPC1_MPC2 are 1.72 and 1.61 m/s, respectively. As shown in Figure 12b, after entering the curve, the fluctuation of longitudinal speed for Integ_MPC is smaller than that of MPC1_MPC2. Integ_MPC has better following performance for the desired distance between vehicles, and MPC1_MPC2 has better following performance for the longitudinal speed of the vehicle in front.

Figure 12c shows the change in longitudinal acceleration. In the early stage of the experiment, the following vehicle accelerates to reduce the error of distance between vehicles in two strategies. Therefore, for the longitudinal acceleration, both strategies have a large acceleration in the early stage of the experiment. When the actual distance approaches the desired distance between vehicles, both strategies adjust the longitudinal speed to approach the longitudinal speed of the preceding vehicle. Figure 12d shows the change in jerk. Both strategies ensure that the maximum absolute value of the jerk during the 2-D car-following process is less than 3 m/s³. Therefore, the comfort of the two strategies in the 2-D car-following process is guaranteed.

Figure 13 shows the vehicle trajectory in scenario 1, and XY_ REF is the lane centerline. For Integ_ MPC and MPC1_MPC2, the root mean square estimation of the horizontal and vertical coordinate deviation are 0.0437 and 0.0696 m, respectively, so the lane keeping is better in Integ_MPC.

Figure 12. Responses of longitudinal motion in scenario 1.

Figure 12. Responses of longitudinal motion in scenario 1.

Figure 13. Vehicle trajectory in scenario 1.

Figure 13. Vehicle trajectory in scenario 1.

Shown in Figure 14 are the responses related to the lateral stability for scenario 1, including the sideslip angle of the centroid, lateral acceleration, turning angle and yaw rate. Before entering the curve, since the road is straight, the four responses are close to 0. When entering the curve, the four responses change greatly. After entering the curve, since the radius of the curve is not constant, the four responses also fluctuate slightly with the change in the radius of the arc.

The change in sideslip angle for the centroid is shown in Figure 14a. Since the vehicle is driving on a straight road at the beginning of the experiment, the sideslip angle of the centroid is 0 deg. When the driving distance exceeds 200 m, it begins to enter the curve. The longitudinal speed is low, so the sideslip angle of the centroid is positive. With the increase in the longitudinal speed, the vehicle tends to move centrifugally. The sideslip angle of the centroid for the two strategies decreases to a negative value. When the longitudinal speed decreases, the sideslip angle of the centroid gradually increases; then the longitudinal speed repeats the process of increasing and decreasing, but the change range decreases and the sideslip angle of the centroid decreases first and then increases. Finally, as the longitudinal speed approaches a stable value, the sideslip angle of the centroid also changes to be a stable value in the two strategies. During the change in the sideslip angle of the centroid, compared with MPC1_MPC2, the variation range for the sideslip angle of the centroid for Integ_MPC strategy is smaller, and the change tends to be stable rapidly. This is because, in Integ_MPC, the DQN-based weight optimization algorithm can obtain optimal weight coefficients, so as to optimize longitudinal following and lateral lane keeping coordinately. Therefore, the sideslip angle of the centroid can be effectively controlled when entering the curve and after entering the curve.

Figure 14b–d show the changes in lateral acceleration, turning angle and yaw rate, and the three responses for the two strategies change similarly. When entering the curve, the three responses in the two strategies fluctuate to a certain extent and then change with the longitudinal speed in real-time. When the longitudinal speed increases, the three responses of lateral motion also increase, and when the longitudinal speed decreases, and the three responses of lateral motion are also decreased.

For the Integ_MPC, the root mean square estimation of the sideslip angle of the centroid, lateral acceleration, turning angle and yaw rate are 0.0423 deg, 0.4933 m/s², 0.3108 deg and 1.4099 deg/s. For the MPC1_MPC2, the root mean square estimation of the sideslip angle of the centroid, lateral acceleration, turning angle and yaw rate are 0.0501 deg, 0.5923 m/s², 0.3901 deg and 1.7096 deg/s. Therefore, compared with MPC1_MPC2, Integ_MPC has better lateral stability.

Figure 14. Responses related to lateral motion in scenario 1.

Figure 14. Responses related to lateral motion in scenario 1.

Figure 15 shows the change in battery power and SOC in scenario 1. From Figure 15a, in the initial stage of the experiment, to reduce the distance from the vehicle in front and approach the vehicle in front, the following vehicles in the two strategies accelerate rapidly, so the battery power is larger. When the distance between the vehicles is close to the desired distance, and the longitudinal speed of the following vehicle approaches the longitudinal speed of the preceding vehicle, the battery power gradually decreases. In the subsequent 2-D car-following process, the preceding vehicle decelerates first, then accelerates, and then decelerates, and finally, the speed tends to be stable, and the following vehicle also maintains the same trend of change, so the battery power changes between positive and negative values. The battery is in the energy consumption state when the battery power is positive. The battery is in the energy recovery state when the battery power is negative. As shown in Figure 15b, since the vehicle switches between acceleration and deceleration repeatedly, the SOC of the two strategies switches between falling and rising states repeatedly. The reduction of SOC corresponds to energy consumption, and the increase in SOC corresponds to energy recovery. Due to the conservation of energy, the energy recovered is less than the energy consumed, so the SOC showed a downward trend throughout the experiment. The indicators of economy for Integ_MPC and MPC1_MPC2 are 0.0049 and 0.0054 km⁻¹, respectively, so the economy of Integ_MPC is better.

Figure 15. Battery power and SOC in scenario 1.

Figure 15. Battery power and SOC in scenario 1.

To sum up, in scenario 1, the following vehicle tracks a preceding vehicle whose speed varies with time. In this scenario, Integ_MPC integrates longitudinal following control and lateral lane keeping control in one MPC framework. The weight coefficients in the objective function are optimized in real-time through the DQN-based weight optimization algorithm, thereby obtaining the optimal control variable. When the vehicle is driving on a straight road, longitudinal following is a priority; when the vehicle is driving on a curve, lane keeping is a priority. While the following vehicle is passing through the curve, since the longitudinal speed can have an impact on the lateral stability, Integ_MPC improves the lateral stability by reducing the amplitude of the longitudinal speed. Although the following performance for the longitudinal speed is reduced, the impact is small. In the 2-D car-following process, the fluctuation range of longitudinal speed and longitudinal acceleration becomes smaller, thereby reducing the energy consumption on the actuator. From

Table 4. Comparison of three performance indicators of the two strategies in scenario 1.

Objectives	Lane Keeping	Lateral Stability				Economy
Indicators	$RMS{E}_{{\Delta }_{XY}}$	$RMS{E}_{\beta }$	$RMS{E}_{a_y}$	$RMS{E}_{{\delta }_f}$	$RMS{E}_{\dot{\psi }}$	ΔSOC/s
Integ_MPC	0.0437 m	0.0423 deg	0.4933 m/s2	0.3108 deg	1.4099 deg/s	0.0049 km−1
MPC1_MPC2	0.0696 m	0.0501 deg	0.5933 m/s2	0.3901 deg	1.7096 deg/s	0.0054 km−1
Improvements	37.21%	15.57%	16.85%	20.33%	17.53%	9.26%
		Average: 17.57%

, the improvement in lane keeping is 37.21%, the average improvement in lateral stability is 17.57% and the improvement in economy is 9.26%. Compared with the comparison strategy, Integ_MPC achieves better lane keeping, lateral stability and economy on the premise of ensuring other performances during the car-following process. Therefore, Integ_MPC can realize the coordinated optimization of longitudinal control and lateral control.

5.2.2. Scenario 2

In Scenario 2, the vehicle follows a vehicle in front, and then the vehicle in front of the adjacent lane suddenly inserts into the lane to become the new preceding vehicle. The vehicle re-determines the following object and takes corresponding measures. The experimental results of this scenario are shown in Figure 16, Figure 17, Figure 18, Figure 19 and Figure 20.

Figure 16 shows the resultant acceleration in scenario 2. The resultant acceleration is the vector sum of the longitudinal acceleration and the lateral acceleration; this paper mainly analyzes the numerical change in the resultant acceleration. In the 2-D car-following control, the longitudinal motion and the lateral motion influence each other, and the lateral acceleration is input into the longitudinal control in real-time to limit the magnitude of the longitudinal acceleration so that the magnitude of the resultant acceleration is maintained within a reasonable range. There are two upper limits for the resultant acceleration, which are μg and μg-ε. Among them, μg is the maximum upper limit of the resultant acceleration. In order to ensure a certain adhesion margin, the upper limit is corrected to μg-ε. During the whole experiment, compared with MPC1_MPC2, Integ_MPC has a smaller fluctuation range of the resultant acceleration. The resultant accelerations of the two strategies are always smaller than the revised upper limit value, thus ensuring that the vehicle can run stably during the 2-D car-following process, which is the premise of the performance optimization for the 2-D car-following process.

Figure 16. Resultant acceleration in scenario 2.

Figure 16. Resultant acceleration in scenario 2.

As shown in Figure 17, the responses of longitudinal motion in scenario 2 contain the distance between vehicles, longitudinal speed, longitudinal acceleration and jerk.

Figure 17a shows the change in distance between vehicles, where Integ_MPC_∆s and MPC1_MPC2_∆s are the distances between vehicles for Integ_MPC and MPC1_MPC2, respectively. Integ_MPC_∆s_des and MPC1_MPC2_∆s_des are the desired distances between vehicles for Integ_MPC and MPC1_MPC2, respectively. In the whole experiment process, the minimum distances between vehicles of both strategies are larger than the minimum safe distance (5 m), so the safety in the two strategies is guaranteed during the 2-D car-following process.

Following performance contains the tracking of the actual distance to the desired distance between vehicles and the tracking of the longitudinal speed of the following vehicle to the longitudinal speed of the vehicle in front. In terms of tracking the desired distance between vehicles, the root mean square estimation of Integ_MPC and MPC1_MPC2 are 5.86 and 8.19 m, respectively. In terms of tracking the longitudinal speed of the preceding vehicle, the root mean square estimation of Integ_MPC and MPC1_MPC2 are 1.45 and 1.35 m/s, respectively. As shown in Figure 17b, before entering the curve, in order to reduce the error between the actual distance and the desired distance for vehicles, the longitudinal speed of Integ_MPC is greater than that of MPC1_MPC2. When the following vehicle is passing through the curve, compared with MPC1_MPC2, the variation range of longitudinal speed in Integ_MPC is reduced. Integ_MPC performs better in tracking the desired distance between vehicles; MPC1_MPC2 is better in tracking the longitudinal speed of the preceding vehicle.

Figure 17c shows the change in longitudinal acceleration. In the early stage of the experiment, since the longitudinal speed of the following vehicle is greater than that of the vehicle in front, the following vehicle in both strategies decelerates. The acceleration of MPC1_MPC2 fluctuates greatly, while the acceleration of Integ_MPC is smoother. Figure 17d shows the jerk. The maximum values of absolute value for the jerk in the two strategies are always less than 3 m/s³. Compared with MPC1_MPC2, the fluctuation range of Integ_MPC is smaller. The comfort of the two strategies is guaranteed as the jerk fluctuates within a reasonable range.

Figure 17. Responses of longitudinal motion in scenario 2.

Figure 17. Responses of longitudinal motion in scenario 2.

Figure 18 shows the vehicle trajectory in scenario 2, and XY _REF is the lane centerline. For Integ_MPC and MPC1_MPC2, the root mean square estimation of the horizontal and vertical coordinate deviation are 0.0469 and 0.0738 m, respectively, so the lane keeping is better in Integ_MPC.

Figure 18. Vehicle trajectory in scenario 2.

Figure 18. Vehicle trajectory in scenario 2.

Figure 19 shows the responses of lateral stability in scenario 2, including sideslip angle of the centroid, lateral acceleration, turning angle and yaw rate. Before entering the curve, since the road is straight, the four responses are all close to 0. When entering the curve, the four responses change greatly. After entering the curve, since the curve radius is not constant, the response also fluctuates slightly with the change in the radius for the curve. In Figure 19a, the change in the sideslip angle of the centroid is shown. Before entering the curve, since the road is straight, the sideslip angle of the centroid is 0 deg. When the following vehicle is driving on the curve, the sideslip angle of the centroid increases slightly. When the following vehicle completely enters the curve, due to the large longitudinal speed, the vehicle has a centrifugal tendency, and the sideslip angle of the centroid decreases to a negative value rapidly. When the longitudinal speed decreases, the sideslip angle of the centroid gradually increases to a positive value. In the subsequent 2-D car-following process, the longitudinal speed for the following vehicle changes with the vehicle in front, and the sideslip angle of the centroid also changes. The sideslip angle of the centroid decreases to be negative first and then increases to be positive. As the longitudinal speed approaches a stable value, the sideslip angle of the centroid also approaches a stable value. After getting into the curve, the fluctuation in the sideslip angle of the centroid in Integ_MPC is smaller than that of MPC1_MPC2.

Figure 19b–d are the changes in lateral acceleration, turning angle and yaw rate, and the change in the three responses for the two strategies are similar. When the following vehicle is getting into the curve, the three responses in both strategies fluctuate to a certain extent and then change in real-time with the change in the longitudinal speed. When the longitudinal speed increases, the three responses also increase; and when the longitudinal speed decreases, the three responses also decrease. After entering the curve, compared with MPC1_MPC2, because the weight coefficients related to lane keeping in Integ_MPC are bigger, it focuses on improving the performance related to lateral lane keeping. The fluctuation range for responses related to lateral stability in Integ_MPC is smaller, and the lateral stability of Integ_MPC is better. This is because the DQN-based weight optimization algorithm can obtain optimal weight coefficients, the three responses can be effectively controlled after entering the curve.

For the Integ_MPC, the root mean square estimation of the sideslip angle of the centroid, lateral acceleration, turning angle and yaw rate are 0.0496 deg, 0.5432 m/s², 0.3257 deg and 1.4806 deg/s. For the MPC1_MPC2, the root mean square estimation of the sideslip angle of the centroid, lateral acceleration, turning angle and yaw rate are 0.0577 deg, 0.5695 m/s², 0.4053 deg and 1.7748 deg/s. Therefore, compared with MPC1_MPC2, Integ_MPC has better lateral stability.

Figure 19. Responses related to lateral motion in scenario 2.

Figure 19. Responses related to lateral motion in scenario 2.

Figure 20 shows the changes in the battery power and SOC in scenario 2. From Figure 20a, it can be seen that in the initial stage of the experiment, the longitudinal speed of the following vehicle is greater than that of the vehicle in front. In Integ_MPC, the following vehicle decelerates, so the battery power at this stage is negative, and the battery is in a state of energy recovery. While in MPC1_MPC2, the following vehicle accelerates and decelerates alternately, the battery power switches between positive and negative values rapidly, and the battery switches between energy consumption and energy recovery rapidly. In the subsequent 2-D car-following process, the preceding vehicle accelerates first and then decelerates, and the following vehicle also accelerates first and then decelerates. The battery power is positive during the acceleration process, and the battery power is negative during the deceleration process. From Figure 20b, it can be obtained that the SOC of the two strategies is accompanied by several rising and falling processes. The rising of SOC corresponds to energy recovery, and the falling of SOC corresponds to energy consumption. At the end of the experiment, the indicators of economy in Integ_MPC and MPC1_MPC2 are 0.0044 and 0.0054 km⁻¹, respectively. Through the above analysis, the economy of Integ_MPC is better.

Figure 20. Battery power and SOC in scenario 2.

Figure 20. Battery power and SOC in scenario 2.

To sum up, in scenario 2, the preceding vehicle in the adjacent lane changes lanes and becomes the new preceding vehicle. Integ_MPC optimizes the longitudinal following control and lateral lane keeping control in an MPC framework. The weight coefficients in the objective function are optimized in real-time through the DQN-based weight optimization algorithm, thereby obtaining the optimal control variables. While the vehicle is following on a straight road, longitudinal following is the priority, and while the vehicle is following on a curve, lane keeping is the priority. Since the longitudinal vehicle speed can affect the lateral stability, after entering the curve, Integ_MPC focuses on improving the lateral stability, so the lateral stability is improved by reducing the fluctuation range of the longitudinal vehicle speed. Although the tracking performance for longitudinal speed is weakened, the effect is small. Since the fluctuations of longitudinal speed and longitudinal acceleration in Integ_MPC are reduced when driving on the curved road, it reduces the energy consumption on the actuator. From

Table 5. Comparison of three performance indicators of the two strategies in scenario 2.

Objectives	Lane Keeping	Lateral Stability				Economy
Indicators	$RMS{E}_{{\Delta }_{XY}}$	$RMS{E}_{\beta }$	$RMS{E}_{a_y}$	$RMS{E}_{{\delta }_f}$	$RMS{E}_{\dot{\psi }}$	ΔSOC/s
Integ_MPC	0.0469 m	0.0496 deg	0.5432 m/s2	0.3257 deg	1.4806 deg/s	0.0044 km−1
MPC1_MPC2	0.0738 m	0.0577 deg	0.6495 m/s2	0.4053 deg	1.7748 deg/s	0.0054 km−1
Improvements	36.45%	14.04%	16.37%	19.64%	16.58%	18.52%
		Average: 16.66%

, the improvement in lane keeping is 36.45%, the average improvement in lateral stability is 16.66%, and the improvement in economy is 18.52%. Compared with the comparison strategy, Integ_MPC achieves better lane keeping, lateral stability and economy on the premise of ensuring other performances in the 2-D car-following process. Thus, the proposed strategy can realize the coordinated optimization of longitudinal control and lateral control.

6. Conclusions

For the coupling problem of longitudinal control and lateral control of the vehicle, the 2-D car-following control strategy is proposed in this paper. First, a 2-D car-following model is established. Then, the 2-D car-following model is designed. In addition, the longitudinal following control and lateral lane keeping control are optimized under an MPC framework. In order to adapt to different dynamic processes of lane keeping, the weight coefficients in the MPC framework are optimized in real-time based on the DQN algorithm, and the optimal weight coefficients are obtained through the trained deep neural network. Finally, to verify the effectiveness of the 2-D car-following control strategy, a comparison strategy and two experimental scenarios are set up. During the experiment, the longitudinal speed is input into the lateral lane keeping control in real-time to influence the lateral lane keeping control; and the lateral acceleration is input into the longitudinal following control in real-time, which constrains the longitudinal following control in real-time. Compared with the comparison strategy, under the premise of ensuring other performances of the 2-D car-following process, the proposed strategy achieves improvements for lane keeping, lateral stability and economy in scenario 1 and scenario 2. For scenario 1, the lane keeping, lateral stability and economy of the proposed strategy are improved by 37.21%, 17.57% and 9.26%, respectively. For scenario 2, the lane keeping, lateral stability and economy of the proposed strategy are improved by 36.45%, 16.66% and 18.52%, respectively.