Integrating Vehicle Positioning and Path Tracking Practices for an Autonomous Vehicle Prototype in Campus Environment

Abstract

This paper presents the implementation of an autonomous electric vehicle (EV) project in the National Taiwan University of Science and Technology (NTUST) campus in Taiwan. The aim of this work was to integrate two important practices of realizing an autonomous vehicle in a campus environment, including vehicle positioning and path tracking. Such a project is helpful to the students to learn and practice key technologies of autonomous vehicles conveniently. Therefore, a laboratory-made EV was equipped with real-time kinematic GPS (RTK-GPS) to provide centimeter position accuracy. Furthermore, the model predictive control (MPC) was proposed to perform the path tracking capability. Nevertheless, the RTK-GPS exhibited some robust positioning concerns in practical application, such as a low update rate, signal obstruction, signal drift, and network instability. To solve this problem, a multisensory fusion approach using an unscented Kalman filter (UKF) was utilized to improve the vehicle positioning performance by further considering an inertial measurement unit (IMU) and wheel odometry. On the other hand, the model predictive control (MPC) is usually used to control autonomous EVs. However, the determination of MPC parameters is a challenging task. Hence, reinforcement learning (RL) was utilized to generalize the pre-trained datum value for the determination of MPC parameters in practice. To evaluate the performance of the RL-based MPC, software simulations using MATLAB and a laboratory-made, full-scale electric vehicle were arranged for experiments and validation. In a 199.27 m campus loop path, the estimated travel distance error was 0.82% in terms of UKF. The MPC parameters generated by RL also achieved a better tracking performance with 0.227 m RMSE in path tracking experiments, and they also achieved a better tracking performance when compared to that of human-tuned MPC parameters.

1. Introduction

Vehicle positioning and path tracking are essential to the development of autonomous vehicles. Vehicle positioning is responsible for providing stable, real-time, and accurate location information for navigations. The path tracking is developed to navigate the vehicle by following the desired trajectory to reduce the tracking errors. In general, the position accuracy with a conventional, stand-alone global positioning system (GPS) is approximately 0.5 m to 3.5 m on average. By applying real-time kinematic (RTK) technology, the GPS positioning accuracy can be improved to the centimeter level. However, the RTK-GPS performance is still restricted when jumping and drifting occur in an obstructed area, such as a blockage and multipath effects in an urban region with buildings and landscapes, under a bridge, or in a tunnel.

Due to the aforementioned RTK-GPS limitations, wheel odometry and inertial measurement units (IMUs) were complementarily combined with the global navigation satellite system (GNSS) to realize a stable and accurate multisensory, fusion-based vehicle positioning system. The aforementioned multisensory fusion problem is nonlinear; hence, an extended Kalman filter (EKF) or unscented Kalman filter (UKF) could be considered. However, the UKF outperforms the conventional EKF in terms of accuracy performance at a comparable level of system complexity [1]. Hence, this work used the UKF as the vehicle position estimation.

On the other hand, a commonly used model predictive control (MPC) method in a dynamic vehicle control system was further utilized in this work. The MPC controller calculates the system output according to the linear time-varying (LTV) model. Nevertheless, due to vehicle dynamics, hardware limitations, and environmental disturbances, system stability and trajectory tracking accuracy were a challenge. The MPC parameter settings are highly related to the controller performance. Practically, trial-and-error blind tuning of MPC parameters takes time and is inefficient. Therefore, applying reinforcement learning (RL) is a helpful way to produce proper MPC parameters to improve the trajectory tracking performance in terms of defining the rewards, states, and actions. Such an RL model works based on the tuning experience of the human MPC model parameters. The pre-trained MPC parameters are capable of providing the datum value rather than trial-and-error. As a consequence, the MPC parameters generated by the RL methods efficiently and effectively supported the MPC to perform an accurate path tracking performance. Such MPC performance measures were evaluated in terms of a simulation environment and a laboratory-made, full-scale electric vehicle.

The rest of the paper is organized as follows. Section 2 surveys the related works. The methods regarding the system architecture, vehicle model, implementation of the UKF-based position estimation, and the RL-based MPC algorithm are discussed in Section 3. In Section 4, the simulation of the proposed system and experiments on the evaluations of the position estimator and RL-based MPC trajectory tracking with a full-scale EV are elaborated. Finally, the conclusion of the proposed study and future works are presented in Section 5.

2. Related Works

This paper first surveys the related works within vehicle positioning. In general, a stand-alone GPS could suffer from a signal mismatch or failure. In addition, inaccurate GPS positioning cannot be directly applied to autonomous vehicle driving purposes unless extra efforts are made, such as image-based lane detection methods [2]. RTK-GPS provides a center centimeter level, and it has been widely used in low-speed (1 Hz) surveying and mapping systems. With the RTK (fixed mode), the position error may be less than 10 cm by following the radiotechnical commission for maritime (RTCM) service standards. Furthermore, the strength of the signal must be larger than 40 dB, and it is expected to receive 16 satellites normally to meet the lowest requirements [3].

Practically, the RTK-GPS is basically composed of a fixed base station and a rover to reduce the rover’s positioning error. Hence, communication between the base station and the rover must be established. An RF module is convenient; however, the disadvantage of utilizing RF modules is that the transmission distance might be limited by the rated power or environment interference. Hence, the stability of signal transmission using RF modules is a challenge [4].

When applying RTK-GPS as a solution to autonomous driving, low-evaluation satellites may suffer from larger atmospheric errors. Practically, implementation with a Kalman filter (KF) estimation could obtain integer ambiguities that allow individuals to be corrected by all ambiguity parameters in practical applications [5]. Moreover, the performance of RTK-GPS may be limited when driving in an interferential environment. The performance evaluation method could be performed in terms of statistically comparing the traveled path by using the mean absolute error and the root mean square (RMS) error, as addressed in [6].

Network instability and component failure are concerns of applying RTK-GPS in practical usages. Hence, Um et al. [7] proposed a redundancy solution, and their system could switch the master system to one of the available slave systems to deal with such network or component failures. In addition, the GPS accuracy may drop due to the multipath error that could occur during signal reflections among large amounts of objects. Therefore, the GSA message was used to provide position dilution of precision (PDOP) measurements according to the geometric constellation of GPS satellites [8,9].

Based on the NMEA (National Electrical Manufacturers Association) GST instructions, the standard deviations of latitude and longitude values are known. Such information could be applied for EKF-based visual odometry and GPS sensor fusion solutions. Once the measurement data are convincing enough, the visual odometry position can be corrected by GPS [10]. Furthermore, the transformation matrix of GPS can be determined to transform the local tangent plane (LTP) coordinates to world coordinates [11]. Within the considerations of the RTK carrier phase ambiguities and the pseudorange multipath errors, the state prediction could be more robust and precise [12]. Odometry methods are categorized into five main types, i.e., wheel, inertial, laser, radar, and visual. Most of the research applied multiple sensors to fuse or optimize the estimation of vehicle position [13].

In general, accumulative error is a critical factor of odometry. Hence, Mikov et al. [14] integrated inertial MEMS (Micro-Electromechanical Systems) and odometry solutions for land vehicle localizations. When the GNSS (Global Navigation Satellite System) signal is available, it can become the odometry correction factor to fix the position. The characteristics of GNSS and IMU are complementary, and they are fused in terms of KFs. Their results achieved a higher sampling rate and a high bandwidth by utilizing an online error state covariance estimation [15]. In addition to the EKF, the UKF is suitable for solving nonlinear problems. The advantage of using the UKF is that it does not need to calculate the Jacobian matrix and apply Taylor expansion. Hence, the UKF eventually results in a higher precision of estimation.

In addition to vehicle positioning, the related works within path tracking problems are also discussed. The single-track kinematics model is simple, and it is often applied to the MPC controller design. The optimal steering wheel angle and longitudinal force can be obtained by using the standard quadratic cost function based on the design constraints [16,17,18]. Under different speed ranges, the prediction model needs to be robust enough to describe the overall vehicle motion behaviors; otherwise, the prediction error will significantly increase [19].

Tang et al. [20] presented the lane changing control method in highway environments. Their study applied MPC as a controller for lane changing path planning. A multi-constraint model predictive control algorithm and optimization for lane changing were proposed. By determining the Q and R cost coefficients in the MPC model, the objective function can be minimized after the iterations. Assuming that the path tracking controller and driver behavior can be synthesized, the robustness and performance of driver-oriented maneuvers, such as cutting in and overtaking, can be further improved [21,22].

Optimization-based autonomous racing RC (Remote Control) cars were also carried out in [23]. That study combined the multitask problem as a nonlinear optimization problem (NLP) for contouring control. The proposed control system was created to build local approximations of NLPs in convex quadratic programs (QPs) at each time step. The algorithm solved the constrained infinite horizon optimal control problem at each time step and iteration. Moreover, an approximated safe set was added to satisfy the real-time computation and to prevent the occurrence of insoluble solutions [24]. The recursive least square algorithm and first-order Lagrange interpolation can be used to further estimate the cornering stiffness and road friction [25].

On the other hand, MPC is highly dependent on the prediction model. The properties of MPC and reinforcement learning (RL) are quite different but complementary, such as model requirements, robustness, stability, feasibility, etc. [26]. The RL applied the concept of the Markov decision process. Based on this assumption, the whole model can be simplified and the next state can be estimated [27].

The combination of RL and MPC was successfully applied to connected and automated vehicles at intersections [28]. The trained coordinate policy in a simulation setting and proximal policy optimization were applied. Their results demonstrated that the computing time and traffic efficiency outperformed other works. Another work [29] was proposed by Tange et al. Their study proposed model predictive control based on the deep RL approach. By evaluating the control performance and learning convergence, their discrete-valued input deep RL method achieved a better performance than direct P-based and PI-based RL methods.

Finally, parameter tuning could be applied by RL methods in various fields. A segmented and adaptive RL approach was proposed in [30] to provide successful noise reduction. By searching possible solution spaces, all the coefficient values can be well tuned in a short learning time. Such an RL-based parameter tuning concept is also applied in this paper to find the MPC parameters.

3. Methods

3.1. System Architecture

The overall system is primarily composed of a UKF-based position estimator and an RL-based MPC (RLMPC) controller design. The Robot Operating System (ROS) system was used in this work to develop all the programs. The first part addressed the UKF-based vehicle positioning system, and stable and accurate vehicle position information was further used for path tracking. The overall operation flowchart of this study is shown in Figure 1.

When the operator intends to start the autonomous driving function, the initialization of the GPS device needs to be ensured. Once the RTK correction signal is received and set, the electric vehicle (EV) is ready for operation. It is noted that the 4G communication modules are convenient to provide a stable signal transmission channel between the base station and the rover.

Autonomous driving also must ensure that the desired navigation path is created. If the path file is not found, a vehicle mission path file can be created either by recording the manual drive path by receiving the UKF vehicle positions or by planning. After the mission path file is loaded, the EV can start the navigation mission at any arbitrary location near the path start location of the trip. The mission task is divided into two parts: The first part is to autonomously drive from the vehicle initial position to the desired path start location; the second part is the path mission part, and the RLMPC controller is capable of tracking the desired trajectory from the path start location to the path destination location by maintaining minimum tracking error. The details of UKF position estimation and RLMPC path tracking will be explained in detail in Section 3.3 and Section 3.4.

3.2. Vehicle Modeling

To evaluate the performance of vehicle positioning and path tracking, a full-scale, laboratory-made electric vehicle (EV) was used for experiments on the campus of National Taiwan University of Science and Technology (NTUST). A kinematic bicycle model was applied since the steering geometry and the control maneuvers can only fit at low speeds and low lateral acceleration in the campus experimental setting. The EV for experiments is shown on the left-hand side of Figure 2. The dimensions of the EV were 3.57 m (L) × 1.55 m (W) × 1.87 m (H), and the vehicle was 1300 kg in weight. The maximum speed was 30 km/h to meet the campus driving setting. The RTK-GPS receiver was installed on the top center of the vehicle. The rotary encoders were mounted inside two rear motors, and the IMU was fixed in front of the EV. It is noted that two 2-D outdoor usage lidars placed inside the right front and left rear of the front and rear bumpers were used to form a 360° detection of obstacles for collision avoidance. Because reactive obstacle avoidance was not the purpose of this paper, the vehicle temporarily stopped autonomous navigation when any obstacles were detected from two LiDars within a 2 m range for safety.

Based on the EV model, the inertial reference frames OXY, $\left(x_f, y_f\right),$ and $\left(x_r, y_r\right)$ were the coordinates at the front axle and rear axle, respectively, as shown on the right-hand side of Figure 2, where $\phi$ is the vehicle heading angle, $v=v_r$, and it is the magnitude of the velocity vector of the center mass, $v_f$ and $v_r$ are the center velocities of the front axle and rear axle, respectively, $l_f$ and $l_r$ are the distances to the front axle and rear axle from the center of mass, respectively, $l$ is the distance from the front axle to the rear axle, and ${\delta }_f$ is the steering angle. It was assumed that there was no slip between the road and the tire for low-speed campus driving. The velocity model can be obtained in Equations (1) and (2).

$$\{\begin{array}{c}{\dot{x}}_r=v_r\cos \phi \\ {\dot{y}}_r=v_r\sin \phi \end{array}$$

(1)

$$\{\begin{array}{c}R=v_r/\omega \\ {\delta }_f={\tan }^{-1}\left(l/R\right)\end{array}$$

(2)

From Equations (1) and (2), the vehicle kinematic model can be derived in Equation (3)

$$\left[\begin{array}{c}{\dot{x}}_r\\ {\dot{y}}_r\\ \dot{\phi }\end{array}\right]=\left[\begin{array}{c}\cos \phi \\ \sin \phi \\ \tan {\delta }_f/l\end{array}\right]v_r$$

(3)

The kinematic bicycle model can be denoted in a generalized form ${\dot{\xi }}_{\mathrm{kin}}=f_{\mathrm{kin}}\left({\xi }_{\mathrm{kin}},u_{\mathrm{kin}}\right)$. The state is ${\xi }_{\mathrm{kin}}={\left[x_r, y_r,\phi \right]}^T$ and the control vector is $u_{\mathrm{kin}}={\left[v_r, {\delta }_f\right]}^T$.

3.3. UKF-Based Position Estimation

In this subsection, the position estimator consisted of two steps: (1) data prefiltering and (2) multisensory fusion. To accomplish the prefiltering of the poor data point, the covariance matrices were defined as the inverse matrix of the covariance matrix of RTK (C_rtk) and the covariance matrix of odometry (C_odm). The vectors collected from stand-alone RTK-GPS and odometry were $X_{\mathrm{rtk}}={\left(x_{\mathrm{rtk}},y_{\mathrm{rtk}}\right)}^T$ and $X_{\mathrm{odm}}={\left(x_{\mathrm{odm}},y_{\mathrm{odm}}\right)}^T$, respectively. The sampling rates of the RTK and odometry were 1 Hz and 10 Hz, respectively. Due to the different sampling rates of the sensors, orientation jumping occurred severely. Hence, in this paper, the Mahalanobis distances $M_p$ and $M_{\phi }$ are defined independently as Equations (4) and (5). In addition, the covariance matrices are defined in Equations (6) and (7).

$$M_p\left(t\right)=\sqrt{{\left(X_{\mathrm{rtk}}\left(t\right)-X_{\mathrm{odm}}\left(t\right)\right)}^T{\left(\mathsf{\Sigma }{C}_{\mathrm{rtk}}+\mathsf{\Sigma }{C}_{\mathrm{odm}}\right)}^{-1}\left(X_{\mathrm{rtk}}\left(t\right)-X_{\mathrm{odm}}\left(t\right)\right)}$$

(4)

$$M_{\phi }\left(t\right)=\sqrt{{\left({\phi }_{\mathrm{rtk}}\left(t\right)-{\phi }_{\mathrm{odm}}\left(t\right)\right)}^T{\left({\sigma }_{\phi \mathrm{rtk}}{}^2+{\sigma }_{\phi \mathrm{odm}}{}^2\left(t\right)\right)}^{-1}\left({\phi }_{\mathrm{rtk}}\left(t\right)-{\phi }_{\mathrm{odm}}\left(t\right)\right)}$$

(5)

$$C_{\mathrm{rtk}}=\left[\begin{array}{cc}{\sigma }_{\mathrm{xrtk}}{}^2& 0\\ 0& {\sigma }_{\mathrm{yrtk}}{}^2\end{array}\right]=\left[\begin{array}{cc}0.36& 0\\ 0& 0.36\end{array}\right]; C_{\mathrm{odm}}=\left[\begin{array}{cc}{\sigma }_{\mathrm{xodm}}{}^2& {\sigma }_{\mathrm{xyodm}}\\ {\sigma }_{\mathrm{xyodm}}& {\sigma }_{\mathrm{yodm}}{}^2\end{array}\right]=\left[\begin{array}{cc}3.1& 1.2\\ 1.2& 3.1\end{array}\right]$$

(6)

$${\sigma }_{\phi \mathrm{rtk}}=1.5; {\sigma }_{\phi \mathrm{odm}}=0.5$$

(7)

Within the RTK data, the covariance matrix $C_{\mathrm{rtk}}$ and the standard deviation ${\sigma }_{\phi \mathrm{rtk}}$ were obtained by the fixed-point experiments, and they were compared with the GGA as well as the VTG message from the NMEA standard. Practically, it was impossible to obtain an accurate position when the vehicle was traveling. This is the reason why the statistical data were measured by fixed points. Meanwhile, the covariance matrix $C_{\mathrm{odm}}$ and standard deviation ${\sigma }_{\phi \mathrm{odm}}$ were based on the experiments that drove through a fixed distance, and they were calculated from the recorded data.

Moreover, $M_p$ and $M_{\phi }$ are the measurement errors, which need to be lower than the threshold values of $M_{p,\mathrm{thres}}$ and $M_{\phi ,\mathrm{thres}}$. The validation criterion depends on the threshold values, and a lower $M_{\mathrm{thres}}$ results in stricter judgment. During the iteration, once the measured error is lower than $M_{\mathrm{thres}}$, the sample point $X_{\mathrm{odm}}$ will be inputted to the UKF framework. Otherwise, the sample point will be ignored. In this paper, the threshold values for the proposed EV are $M_{p,\mathrm{thres}}=4.75$ and $M_{\phi ,\mathrm{thres}}=0.6$. Both settings were determined by empirical knowledge in this model. By applying the above rules, the pre-filtered data points can be divided into four types, which are described as follows.

• Case 1: If $M_p<M_{p,\mathrm{thres}}$ and $M_{\phi }<M_{\phi ,\mathrm{thres}}$, the correction set of $X_{\mathrm{pf}}$ will take into account both position and orientation data from odometry, where $X_{\mathrm{pf}}$ and the covariance matrix of measurement noise $R_{\mathrm{ukf}}$ are defined as in Equations (8) and (9):

  $$X_{\mathrm{pf}}={\left(x_{\mathrm{odm}}\left(t\right),y_{\mathrm{odm}}\left(t\right),{\phi }_{\mathrm{odm}}\left(t\right)\right)}^T$$

  (8)

  $$R_{\mathrm{ukf}}=\mathrm{diag}\left({\sigma }_{\mathrm{xodm}}{}^2, {\sigma }_{\mathrm{yodm}}{}^2,{\sigma }_{\phi \mathrm{odm}}{}^2\right)$$

  (9)
• Case 2: If $M_p<M_{p,\mathrm{thres}}$ and $M_{\phi }\ge M_{\phi ,\mathrm{thres}}$, the correction set of $X_{\mathrm{pf}}$ will only take the position data from odometry, where $X_{\mathrm{pf}}$ and the covariance matrix of measurement noise $R_{\mathrm{ukf}}$ are defined as in Equations (10) and (11):

  $$X_{\mathrm{pf}}={\left(x_{\mathrm{odm}}\left(t\right),y_{\mathrm{odm}}\left(t\right),{\phi }_{\mathrm{lukf}}\left(t\right)\right)}^T$$

  (10)

  $$R_{\mathrm{ukf}}=\mathrm{diag}\left({\sigma }_{\mathrm{xodm}}{}^2, {\sigma }_{\mathrm{yodm}}{}^2,0\right)$$

  (11)
  It is noted that ${\phi }_{\mathrm{lukf}}\left(t\right)$ is the estimated orientation of the UKF from the last iteration.
• Case 3: If $M_p\ge M_{p,\mathrm{thres}}$ and $M_{\phi }<M_{\phi ,\mathrm{thres}}$, the correction set of $X_{\mathrm{pf}}$ will only take the orientation data from odometry, where $X_{\mathrm{pf}}$ and the covariance matrix of measurement noise $R_{\mathrm{ukf}}$ are defined as in Equations (12) and (13):

  $$X_{\mathrm{pf}}={\left(x_{\mathrm{lukf}}\left(t\right),y_{\mathrm{lukf}}\left(t\right),{\phi }_{\mathrm{odm}}\left(t\right)\right)}^T$$

  (12)

  $$R_{\mathrm{ukf}}=\mathrm{diag}\left(0, 0,{\sigma }_{\phi \mathrm{odm}}{}^2\right)$$

  (13)
  It is noted that $x_{\mathrm{lukf}}\left(t\right)$ and $y_{\mathrm{lukf}}\left(t\right)$ are the components of the estimated position of the UKF from the last iteration.
• Case 4: If $M_p\ge M_{p,\mathrm{thres}}$ and $M_{\phi } \ge M_{\phi ,\mathrm{thres}}$, the vehicle will not take the odometry measurement data as the correction input of the UKF.

After the above processes, the pre-filtered data point $X_{\mathrm{pf}}$ can be obtained, and it will be used as the observation of UKF frameworks.

It is noted that the UKF worked at 10 Hz in this work. Based on the unscented transformation (UT), the use of sigma points to describe the nonlinear system would not sacrifice the high-order terms that make the system more precise. Compared with the EKF, the UKF can achieve higher efficiency and marvelous performance without calculating the Jacobian matrix and applying the Tylor expansion. The UKF iteration includes prediction and update processes. The UKF is briefly introduced [1]. Assume that there is a discrete-time, nonlinear system, as in Equation (14).

$$X_{k+1}=\Gamma \left(X_k,{\mho }_k,Q_k\right); Y_k=G\left(X_k,R_k\right)$$

(14)

where $\Gamma$ and $G$ represent the dynamic model of the nonlinear systems, and they are assumed to be known, $X_k$ is the unobserved state of the system, ${\mho }_k$ is the system input, $Y_k$ is the only observed state, $Q_k$ is the process noise, and $R_k$ is the measurement noise of the system observation. When the variable $X$ with dimension L goes through a nonlinear system $Y=G\left(X\right)$, $X$ has the mean $\overline{X}$ and the covariance $P_X$. Then, a matrix $\chi$ called a sigma matrix with size 2L + 1 is formed. The sigma point ${\chi }_i$ and the corresponding weights $W_i$ can be defined as follows:

$$\lambda ={\alpha }^2\left(L+\kappa \right)-L; {\chi }_k^{\left(0\right)}=\overline{X}$$

(15)

$${\chi }_k^{\left(i\right)}=\overline{X}+{\left(\sqrt{\left(L+\lambda \right)P_X}\right)}_i,i=1,2,\dots ,L; {\chi }_k^{\left(i\right)}=\overline{X}-{\left(\sqrt{\left(L+\lambda \right)P_X}\right)}_{i-L},i=L+1,\dots ,2L$$

(16)

$$W_0^{\left(m\right)}=\frac{\lambda }{\left(L+\lambda \right)},W_0^{\left(c\right)}=\frac{\lambda }{\left(L+\lambda \right)}+\left(1-{\alpha }^2+\beta \right), W_i^{\left(m\right)}=W_i^{\left(c\right)}=\frac{1}{2\left(L+\lambda \right)},i=1,2,\dots ,2L$$

(17)

where $\lambda$ is the scaling parameter, $\alpha$ can be used to determine the spread of the sigma point around $\overline{X}$ and is usually set to a small positive value such as 0.01, $\beta$ is used to combine prior knowledge of the distribution of $X$, $\kappa$ is a secondary scaling parameter that is set to 0, and ${\left(\sqrt{\left(L+\lambda \right)P_X}\right)}_i$ is the i-th row of the matrix square root that predicts the sigma point with the transformation matrix $\Gamma$. Based on the weights of each sigma point, the predicted mean ${\overline{X}}_{k|k}^{-}$ and the predicted covariance matrix $P_{k|X}^{-}$ with the process noise $R_k$ can be obtained.

$${\overline{X}}_{k|k}^{-}={\displaystyle \sum }_{i=0}^{2L}{W}_i^{\left(m\right)}{\chi }_{k|k-1}^{\left(i\right)}$$

(18)

$$P_{k|X}^{-}={\displaystyle \sum }_{i=0}^{2L}{W}_i^{\left(m\right)}\left({\chi }_{k|k-1}^{\left(i\right)}-{\overline{X}}_{k|k}^{-}\right){\left({\chi }_{k|k-1}^{\left(i\right)}-{\overline{X}}_{k|k}^{-}\right)}^T+R_k$$

(19)

Moreover, the calculated sigma point will propagate through the nonlinear function $G$. The approximation of the measurement means ${\overline{Y}}_{k|k}^{-}$ based on the predicted state is indicated in Equation (20):

$${\overline{Y}}_{k|k}^{-}={\displaystyle \sum }_{i=0}^{2L}{W}_i^{\left(m\right)}{Y}_{k|k-1}^{\left(i\right)}$$

(20)

The measurement covariance matrix $P_{{\overline{Y}}_k{\overline{Y}}_k}$ with measurement noise $Q_k$ and the covariance matrix $P_{{\mathrm{X}}_k{\mathrm{Y}}_k}$ of the cross-correlation measurement for $Y$ are estimated by using the weighted mean and the covariance of the posterior sigma point, as indicated in Equations (21) and (22):

$$P_{{\overline{Y}}_k{\overline{Y}}_k}={\displaystyle \sum }_{i=0}^{2L}{W}_i^{\left(c\right)}\left(Y_{k|k-1}^{\left(i\right)}-{\overline{Y}}_{k|k}^{-}\right){\left(Y_{k|k-1}^{\left(i\right)}-{\overline{Y}}_{k|k}^{-}\right)}^T+Q_k$$

(21)

$$P_{X_k{Y}_k}={\displaystyle \sum }_{i=0}^{2L}{W}_i^{\left(c\right)}\left({\chi }_{k|k-1}^{\left(i\right)}-{\overline{X}}_{k|k}^{-}\right){\left(Y_{k|k-1}^{\left(i\right)}-{\overline{Y}}_{k|k}^{-}\right)}^T$$

(22)

Finally, the system updates the mean of the system state and its covariance matrix and then calculates the Kalman gain $K_k$

$$K_k=P_{X_k{Y}_k}{P}_{{\overline{Y}}_k{\overline{Y}}_k}{}^{-1}$$

(23)

$${\overline{X}}_{k|k}={\overline{X}}_{k|k}^{-}+K_k\left(Y_k-{\overline{Y}}_{k|k}^{-}\right)$$

(24)

$$P_{k|X}=P_{k|X}^{-}-K_k{P}_{{\overline{Y}}_k{\overline{Y}}_k}{K}_k{}^T$$

(25)

Assume that the driving surface is a plane; hence, the vehicle motion state and input can be expressed as:

$$Y_k=\left[\begin{array}{c}{x}_{\mathrm{rtk},k}\\ y_{\mathrm{rtk},k}\\ {\phi }_{\mathrm{rtk},k}\end{array}\right], X_k=X_{pf}, {\mho }_{k }=\left[\begin{array}{c}{D}_k\\ \mathsf{\Delta }{\phi }_k\end{array}\right]$$

(26)

where $D_k$, $\mathsf{\Delta }{\phi }_k$, and the state equation are defined as follows:

$$D_k=\sqrt{{(d_x)}^2+{\left(d_y\right)}^2}$$

(27)

$$\mathsf{\Delta }{\phi }_k={\phi }_k-{\phi }_{k-1}$$

(28)

$$\left[\begin{array}{c}{x}_k\\ y_k\\ {\phi }_k\end{array}\right]=\left[\begin{array}{c}{x}_{k-1}+D_k\cos \left({\phi }_k+\mathsf{\Delta }{\phi }_k\right)\\ x_{k-1}+D_k\sin \left({\phi }_k+\mathsf{\Delta }{\phi }_k\right)\\ {\phi }_{k-1}+\mathsf{\Delta }{\phi }_k\end{array}\right]$$

(29)

The number of the input state is 3. As a consequence, $L=3$, $\kappa =0$, and $\alpha =0.01$. According to the Gaussian distribution, $\beta =2$ is optimal; hence, $\lambda =-2.9997$. The definitions of the process noise matrix $Q_k$ and measurement noise $R_k$ are shown as follows:

$$Q_k=\left[\begin{array}{ccc}{\sigma }_{\mathrm{lat}}{}^2& {\sigma }_{\mathrm{lat}}{\sigma }_{\mathrm{lon}}& {\sigma }_{\mathrm{lat}}{\sigma }_{\phi \mathrm{rtk}}\\ {\sigma }_{\mathrm{lat}}{\sigma }_{\mathrm{lon}}& {\sigma }_{\mathrm{lon}}{}^2& {\sigma }_{\phi \mathrm{rtk}}{\sigma }_{\mathrm{lon}}\\ {\sigma }_{\phi \mathrm{rtk}}{\sigma }_{\mathrm{lat}}& {\sigma }_{\phi \mathrm{rtk}}{\sigma }_{\mathrm{lon}}& {\sigma }_{\phi \mathrm{rtk}}{}^2\end{array}\right]$$

(30)

$$R_k=R_{\mathrm{ukf}}$$

(31)

The standard deviations of the latitude, longitude, and orientation are obtained from the GST message, and they are compared with the position dilution of precision (PDOP) from the GSA message. If the PDOP is greater than the PDOP_avg (i.e., =1.5), the standard deviation will stay with the values equaling 0.6 for latitude and longitude and 1.5 for orientation. The values are obtained by experiments; otherwise, the standard deviation will be dynamic with the GST message. After the completion of the UKF framework definition, the position estimator can provide robust positioning ability by fusing the RTK-GPS signal and IMU/odometry.

3.4. Reinforcement Learning-Based Model Predictive Control

When designing the EV trajectory tracking controller, the prediction model needs to be robust enough to describe the overall dynamics of the system. In addition, the system model also needs to be simple enough, allowing the optimization problem to be computed in real time. In this paper, the prediction model and the quadratic cost function focus on a linear time-varying (LTV) model as the validation criterion. The vehicle state equation $\dot{X}$ and its reference ${\dot{X}}_r$ used in the MPC controller are shown as follows:

$${\dot{X}}_r=f\left(X_r,u_r\right), \dot{X}=f\left(X,u\right)$$

(32)

In Equation (32), f is a state function. By applying the Tylor expansion, the state-space model for the MPC can be expressed as:

$$\dot{\tilde{X}}=A\left(t\right)\tilde{X}+B\left(t\right)\tilde{u}$$

(33)

where

$$A\left(t\right)=\left[\begin{array}{cccccc}0& 0& -v_r\sin \phi & \cos \phi & 0& 0\\ 0& 0& v_r\cos \phi & \sin \phi & 0& 0\\ 0& 0& 0& \tan {\delta }_f/l& 0& 0\\ 0& 0& 0& 1& 0& 0\\ f^{\prime }\left(x\right)& -1& v_r\mathrm{ecos}\left(e\phi \right)dt& \sin \left(e\phi \right)dt& 0& v_r\cos \left(e\phi \right)dt\\ 0& 0& 1& 0& 0& 0\end{array}\right], B\left(t\right)=\left[\begin{array}{cc}\cos \phi & 0\\ \sin \phi & 0\\ \tan {\delta }_f/l& v_r/l{\cos }^2{\delta }_f\\ 1& 0\\ \sin \left(e\phi \right)dt& 0\\ 0& 0\end{array}\right]$$

(34)

$\tilde{X}={\left[x y \phi v_r cte e\phi \right]}^T$; $\tilde{u}={\left[v_r {\delta }_f\right]}^T$; and $v_r$, $cte$, and $e\phi$ are the reference velocity, tracking error, and orientation error, respectively. In addition, the variables in Equation (34) refer to the kinematic bicycle model indicated in Figure 2. It is noted that this equation is derived in terms of the partial differentiation from the state vectors and control vectors, which applied a basic single-trail bicycle model and the parameters indicated in the paragraph. Moreover, Equation (33) needs to be discretized since the MPC algorithm is only used for the discrete control system, as shown in Equation (35). It is noted that the forward Euler discretization was applied in our proposed algorithm.

$$\tilde{X}\left(k+1\right)=A_{k,t}\tilde{X}\left(k\right)+B_{k,t}\tilde{u}\left(k\right)$$

(35)

With the sampling time T, $A_{k,t}=I+TA\left(t\right)$ and $B_{k,t}=TB\left(t\right)$. On this basis, the new state space Equation (37) can be defined with the transformation shown in Equation (36):

$$\xi \left(k|t\right)=\left[\begin{array}{c}\tilde{X}(k|t)\\ \tilde{u}(k-1|t)\end{array}\right]$$

(36)

$$\{\begin{array}{c}\xi \left(k+1|t\right)={\tilde{A}}_{k,t}\xi \left(k|t\right)+{\tilde{B}}_{k,t}\mathsf{\Delta }u\left(k|t\right)\\ \eta \left(k|t\right)={\tilde{C}}_{k,t}\xi \left(k|t\right)\end{array}, \mathrm{and}$$

(37)

$${\tilde{A}}_{k,t}=\left[\begin{array}{cc}{A}_{k,t}& B_{k,t}\\ 0_{m\times n}& I_m\end{array}\right],{\tilde{B}}_{k,t}=\left[\begin{array}{c}{B}_{k,t}\\ I_m\end{array}\right],{\tilde{C}}_{k,t}=\left[\begin{array}{cc}{C}_{k,t}& 0\end{array}\right]=\left[\begin{array}{cc}1& 0\end{array}\right]$$

(38)

Assume that $N_p$ (i.e., =8) and $N_c$ (i.e., =7) are the prediction horizon and control horizon of the system. To simplify the calculation, we assume that $A_{k,t}=A_t$, $B_{k,t}=B_t$, and $k=1, 2,\dots , t+N-1$. Hence, the prediction formula can be expressed as:

$${\rm Y}\left(t\right)=\Psi \xi \left(t|t\right)+\Theta \mathsf{\Delta }U\left(t\right)$$

(39)

where

$$\begin{array}{c}{\rm Y}\left(t\right)={\left[\eta \left(t+1|t\right) \eta \left(t+2|t\right)\cdots \eta \left(t+N_p|t\right)\right]}^T, \Psi ={\left[{\tilde{C}}_t{\tilde{A}}_t {\tilde{C}}_t{\tilde{A}}_t{}^2\cdots {\tilde{C}}_t{\tilde{A}}_t{}^{N_p}\right]}^T,\\ \Theta =\left[\begin{array}{ccc}{\tilde{C}}_t{\tilde{B}}_t& \cdots & 0\\ ⋮& \ddots & ⋮\\ {\tilde{C}}_t{\tilde{A}}_t{}^{N_p-1}{\tilde{B}}_t& \cdots & {\tilde{C}}_t{\tilde{A}}_t{}^{N_p-N_c-1}{\tilde{B}}_t\end{array}\right],\\ \mathsf{\Delta }U\left(t\right)={\left[\mathsf{\Delta }u(t|t) \mathsf{\Delta }u(t+1|t)\cdots \mathsf{\Delta }u(t+N_c|t)\right]}^T\end{array}$$

(40)

To guarantee that the autonomous vehicle can track the reference trajectory fast and smoothly, the controller needs to optimize the system state and control increments. As a result, in the process of controller design, the objective function is considered as follows:

$$J\left(k\right)={\displaystyle \sum }_{i=1}^{N_p}||\eta (k+i|t)-{\eta }_{ref}(k+i|t){||}_{Q_{mpc}}^2+{\displaystyle \sum }_{i=1}^{N_c-1}||\mathsf{\Delta }U(k+i|t){||}_{R_{mpc}}^2+\rho {\epsilon }^2$$

(41)

The first term in Equation (41) is the tracking error between the current pose and the reference path, which represents the tracking ability of the reference path. Additionally, our goal is to minimize the term in order to get the optimized result in each iteration, where

$$Q_{mpc}=\mathrm{diag}\left(Q_{n+1},Q_{n+2},\cdots ,Q_{n+N_p}\right), R_{mpc}=\mathrm{diag}\left(R_n,R_{n+1},\cdots , R_{n+N_p-1}\right)$$

(42)

$$\left(k\right)=\left[\begin{array}{c}\begin{array}{c}x\\ y\\ \phi \end{array}\\ \begin{array}{c}v\left(k\right)\\ cte\left(k\right)\\ e\phi \left(k\right)\end{array}\end{array}\right], {\eta }_{ref}\left(k\right)=\left[\begin{array}{c}\begin{array}{c}{x}_{ref}\\ y_{ref}\\ {\phi }_{ref}\end{array}\\ \begin{array}{c}{v}_{max}\\ 0\\ 0\end{array}\end{array}\right], \mathsf{\Delta }U\left(k\right)=\left[\begin{array}{c}\mathsf{\Delta }a\\ \mathsf{\Delta }\delta \end{array}\right]$$

(43)

The first term in Equation (41) represents the tracking ability of the reference path. The second term represents the constraints to the change of the control vector. Q_mpc and R_mpc are the weighting matrices where Q_mpc $\succcurlyeq$ 0 and R_mpc $\succcurlyeq$ 0. The mutation of the control vector will be inevitable, and it affects the continuity of the control vector since the objective function is not able to constrain the increments of the control vector in every period. Hence, soft constraints are applied to the implementation objective function in this study. The $\rho$ is the weighting coefficient, and $\epsilon$ is the relaxation factor. It can constrain the increments of the control vector directly, and it can also prevent the algorithm from obtaining an insoluble solution. In Equation (43), the state model $\eta \left(k\right)$, reference state model ${\eta }_{ref}\left(k\right)$, and the actual increments of the control vector $\mathsf{\Delta }U\left(k\right)$ are addressed. Moreover, $J\left(k\right)$ will be transformed into a QP form as in Equation (44).

$${\rm Y}\frac{1}{2}{x}^{{\rm T}}Hx+f^{{\rm T}}x, x={\left[\mathsf{\Delta }U \epsilon \right]}^T$$

(44)

The purpose of the MPC design is to minimize the tracking error, and the determination of weighting matrices $Q_n$ and $R_n$ is crucial to the MPC performance. However, parameter tuning is difficult and scenario oriented. It usually relies on empirical knowledge and trial-and-error methods. Hence, such a parameter tuning procedure is time consuming and inefficient. To solve this problem, this paper proposes a reinforcement learning-based MPC (RLMPC) controller to generate certain MPC parameters.

The concept of applying RL is simple, and a typical RL system is formed with the interaction of an agent and an environment. The RL training framework is shown in Figure 3, and $\pi (a_t|S_t)$ is the policy that can determine which action $a_t$ could be applied according to the observed state $S_t$. The reward function $R_t$ evaluates the rewards for the action applied to the RL. The agent is desired to react with the environment to obtain a higher reward by updating the policy.

$Q\left(S_t,a_t\right)$ is a component of the agent, and it updates in each iteration. By applying the Markov decision process (MDP) Equation (45), a Q-function can estimate the future state and reward of the system with the current state and action. As a consequence, the updated $Q\left(S_t,a_t\right)$ is indicated in Equation (46). The iteration process is further applied for the generation of optimal weighting of the RLMPC. The operation of the proposed RLMPC is to generate a datum value of $cte\left(k\right)$, $e\phi \left(k\right)$, and $v\left(k\right)$, and the rest of the parameters remain manually tuned to reduce the RL complexity. The definitions of state, action, and reward for the proposed RLMPC are shown in Equation (47). The relative RL setting is shown in

Table 1. RL Parameter Setting.

α-Parameter	ε-Parameter	γ-Parameter	Training Episodes
0.5	0.2	0.95	2000

, and $\alpha$, $\epsilon$, and $\gamma$ are denoted as the learning rate, greedy index, and discount rate, respectively.

$$P\left(S_{t+1}|S_t\right)=P\left(S_{t+1}|S_1,S_2,\dots ,S_t\right)$$

(45)

$$Q^{*}\left(S_t,a_t\right)=Q\left(S_t,a_t\right)+\alpha \left(R_t+\left(\gamma \underset{a\in A}{max} Q\left(S_{t+1},a_{t+1}\right) \right)-Q\left(S_t,a_t\right)\right)$$

(46)

$$\begin{array}{c}\mathrm{State}=\{\begin{array}{cc}0.895 \mathrm{m}\le e\le 0.91 \mathrm{m}& R_t=10\\ else& R_t=-30\end{array},\\ \mathrm{Action}=\{\begin{array}{cc}{q}_V=700~1000& \pm 100\\ q_{cte}=600~1100& \pm 100\\ q_{e\phi }=600~1100& \pm 100\end{array} \Rightarrow 144 \mathrm{Actions}\end{array}$$

(47)

Furthermore, the constraints of the control vector and control increments are designed by the Kronecker product in the form indicated in Equations (48) and (49).

$$u_{min}\left(t+k\right)\preccurlyeq u\left(t+k\right)\preccurlyeq u_{max}\left(t+k\right), k=0,1,\dots ,N_c-1$$

(48)

$$\mathsf{\Delta }{u}_{min}\left(t+k\right) \preccurlyeq u\left(t+k\right) \preccurlyeq \mathsf{\Delta }{u}_{max}\left(t+k\right), k=0,1,\dots ,N_c-1$$

(49)

According to the performance limitation of the proposed EV, the constraints applied in this paper are:

$$0 \mathrm{m}/\mathrm{s}\le v_r\le 1.25 \mathrm{m}/\mathrm{s}, -1.0 \mathrm{m}/{\mathrm{s}}^2\le a\le 1.0 \mathrm{m}/{\mathrm{s}}^2, -17°\le {\delta }_f\le 17°$$

(50)

By combining Equations (41), (48) and (49), the optimized objective expression is defined in terms of applying the barrier interior point method (BIPM) with RL pretrained weighting matrices. After solving the QP problem in every time step, a series of control increments in the control horizon can be obtained, as shown in Equation (51).

$$\mathsf{\Delta }{U}_t^{*}={[\mathsf{\Delta }{u}_t^{*}, \mathsf{\Delta }{u}_{+1t}^{*},\cdots , \mathsf{\Delta }{u}_{t+N_c-1}^{*}]}^T$$

(51)

The first element of the control series in Equation (51) is the actual input increment of the system, as indicated in Equation (52).

$$u\left(t\right)=u\left(t-1\right)+\mathsf{\Delta }{u}_t^{*}$$

(52)

In the next time step, the system predicts a new output according to the state and undergoes the optimization process for new control increments. This procedure iterates until the whole path tracking mission is finished. As a consequence, the abovementioned RLMPC iteration process is summarized in Figure 4.

4. Simulations and Experiments

To evaluate the performance of the proposed vehicle positioning and path tracking methods, simulations and experiments on a full-scale, laboratory-made EV were arranged, and they are organized in three subsections.

4.1. Simulation of RLMPC-Based Path Tracking

The simulations were arranged to evaluate the path tracking performance with respect to the manually tuned MPC and the RL-trained MPC methods. The trajectories of the abovementioned simulation results are shown in Figure 5 and Figure 6. It is noted that the green line represents the calculated path in every MPC iteration. The weighting matrices of the manually tuned MPC and the RL-trained MPC are indicated in Equations (53) and (54), respectively. The vehicle started from (0,0) with a heading of 0 rad, and it attempted to track the trajectory following a line path equation of y = 2. The manual parameter tuning took time, and the trajectory exhibited an overshoot at the desired line path. On the other hand, with the RL, a proper MPC parameter, indicated in Equation (54), was obtained without a time-consuming trial-and-error process. It was apparent that the RLMPC tracked the line path well without overshoots.

$$Q_n=\mathrm{diag}\left(10 10 5 1000 5000 1500\right), R_n={\left[\begin{array}{cc}40& 100\end{array}\right]}^T$$

(53)

$$Q_{pt}=\mathrm{diag}\left(10 10 5 800 1100 700\right), R_{pt}={\left[\begin{array}{cc}40& 100\end{array}\right]}^T$$

(54)

Finally, the RL performance is also indicated in Figure 7. As the learning iteration accumulated, the successful counts increased. Because there were only two states in this training procedure, this study raised the learning rate for fast convergence. The training result with the parameters shown in Table 1 is indicated in Equation (54).

4.2. Validation of Estimated Distance with Position Estimation

The second experiment was arranged to evaluate the UKF-based vehicle positioning system, and a rectangular path on the NTUST campus was arranged for validation, as shown in Figure 8. It is noted that, due to the campus driving speed and route limitation of the small campus of NTUST (90,000 m²), the following experiments were arranged simply. However, such a simple test environment was sufficient to examine the feasibility of integrating the aforementioned two technical aspects.

The RTK-GPS position obtained from the GGA instruction needed to be projected from the WGS84 system to the Cartesian coordinate system. Because the validation region in this paper was small, the equirectangular projection method was applied. The total distance of the rectangular path was approximately 199.37 m. The comparison of the traveling distance and the trajectory of odometry, RTK-GPS, and UKF are shown in

Table 2. Comparison of Travel Distances Using Odometry, RTKGPS, and UKF.

Methods	Ground Truth (m)	Estimated Distance (m)	Error (%)	Mean Error (m)	Standard Deviation (m)
Odometry	199.27	192.882	3.205	6.387	4.856
RTKGPS	199.27	197.462	1.684	3.356	2.243
UKF	199.27	198.201	0.826	1.646	1.198

and Figure 9. Moreover, the comparison of odometry and the RTK and UKF methods are shown in

Table 3. Performance Comparison with literature [10,14].

	Proposed Method	Wei [10]	Mikov [14]
Ground Truth (m)	199.27	674.5	10,352 ± 3749
Estimated Distance (m)	198.201	683.18	10,404 ± 3777
Error (%)	0.826	1.29	0.502 ± 0.746
Mean Error (m)	1.646	1.23	-
Standard Deviation (m)	1.198	1.05	28

.

The experimental results showed that the error of the UKF position estimator was 0.826%, as evaluated by the ground truth measured by the authors. Table 2 also indicates that the UKF represented the minimum error, mean error, and standard deviation when compared to odometry and RTK. Moreover, Table 3 shows a performance comparison with the state-of-the-art methods [10,14]. Our purpose was to know the performance of our UKF localization. We evaluated if the error percentage produced by the UKF was superior to other approaches. Although the experimental range and path were not the same, the proposed UKF demonstrated acceptable mean error and standard deviation.

Finally, Figure 9 shows the trajectory comparison of odometry and the RTK-GPS and UKF methods. It is also worth noting that on the top left of Figure 9 the RTK-GPS exhibits an outward jump. However, the UKF position estimator remained quite stable. The UKF position estimator also reduced the accumulative error that occurred when using odometry only.

4.3. Integrated Experiment with EV by Applying RL-Based MPC

The path can be any combined planner equations. For example, the test scenario 2 was composed of four segments. The reference path was formed by recording the trajectory of manual driving. The recorded trajectory was manually processed in terms of driving as four segments, and each segment was further represented as an equation in terms of the curve fitting approach. Such combined equations are available to be tracked in terms of RL MPC.

In this experiment, an EV was used for trajectory tracking based on human-tuned and RLMPC control schemes. Two experimental scenarios were arranged on the NTUST campus: (1) a straight-line path and (2) a combinational path. It is noted that the straight-line path of scenario 1 is indicated as in Equation (55) and the combinational path of scenario 2 is indicated in Equation (56). For the combinational path in Equation (56), a smoothing spline was utilized to obtain a piecewise linear function with four intervals (i = 1 to 4). The weight was set as $w_i=1$, and the corresponding smoothing parameter for each interval is indicated in Equation (57).

$$y\left(x\right)=-1.095x-260.7$$

(55)

$$p{\displaystyle \sum }_i{w}_i{\left(y_i-s\left(x_i\right)\right)}^2+\left(1-p\right){\displaystyle \int }{\left(\frac{d^{2}s}{d{x}^2}\right)}^{2}dx$$

(56)

$$p=\{\begin{array}{cc}0.999& , \mathrm{for} \mathrm{interval} 1\\ 0.991& , \mathrm{for} \mathrm{interval} 2\\ 0.769& , \mathrm{for} \mathrm{interval} 3\\ 0.763& , \mathrm{for} \mathrm{interval} 4\end{array}$$

(57)

$$Q_n=\mathrm{diag}\left(10 10 5 900 2100 1000\right), R_n={\left[\begin{array}{cc}40& 100\end{array}\right]}^T$$

(58)

Mechanism tolerance, hardware limitations, and other factors might influence practical implementation. This work applied a pre-trained weighting matrix, shown in Equation (54), as the datum value of a full-scale EV experiment. Based on empirical knowledge and the pre-trained datum value of the weighting matrix, it can significantly reduce the parameter tuning time. The practical weighting matrices $Q_n$ and $R_n$ were further revised for RLMPC, as indicated in Equation (58). The weighting matrices applied in non-RLMPC that were tuned by the operator were the same as the simulation case indicated in Equation (53).

For scenario 1 experiments, the path tracking results of MPC and RLMPC are shown in Figure 10, and the tracking errors of MPC and RLMPC are indicated in Figure 11. The line path tracking results were quite similar to the aforementioned simulation results shown in Figure 5 and Figure 6. The human-tuned MPC represented some oscillation when the EV reached the line path. Nevertheless, the RLMPC exhibited a smaller error after the 70th sample.

For the scenario 2 experiments, the path tracking results of MPC and RLMPC are shown in Figure 12, and the tracking errors of MPC and RLMPC are indicated in Figure 13. It was apparent that the RLMPC outperformed the tracking error compared to the human-tuned MPC. To provide a confident and quantitative error evaluation, all the experiments were performed three times for the performance comparison, as indicated in

Table 4. Comparison of Path Tracking Performance of Scenario 2.

Method	Maximum Error (m)	Average Error (m)	Standard Deviation (m)	RMSE (m)
MPC	0.671	0.291	0.138	0.257
RLMPC	0.615	0.196	0.112	0.227

. Table 4 shows the relative statistical data of averaging the values of the three trials. Both of the average RMSEs were less than 0.3 m, and the maximum errors were less than 0.7 m. The overall results showed that the RLMPC and human-tuned MPC followed the same trajectory well. However, with well-converged parameters, RLMPC had better performance than MPC tuned by humans in terms of maximum error, average error, standard deviation, and RMSE.

5. Conclusions and Future Works

In this paper, a reinforcement learning-based MPC framework is presented. The proposed RLMPC significantly reduced the efforts of tuning MPC parameters. The RLMPC executed with the UKF-based vehicle positioning system that considered the RTK, odometry, and IMU sensor data. The proposed UKF vehicle positioning and RLMPC path tracking methods were validated with a full-scale, laboratory-made EV on the NTUST campus. On a 199.27 m loop path, the UKF estimated travel distance error was 0.82%. The MPC parameters generated by RL achieved an RMSE of 0.227 m in the path tracking experiments, and it also exhibited better tracking performance than the human-tuned MPC parameters.

Moreover, the aim of this work was to integrate two important practices of realizing an autonomous vehicle in a campus environment, including vehicle positioning and path tracking. Such a project is helpful to students in university to easily reach, learn, and practice key technologies of autonomous vehicles. As a consequence, this work was not aiming at providing significant improvement on the localization accuracy or RL MPC performance. Hence, the future works on the localization accuracy and RL MPC performance in terms of two independent projects will be studied based on the laboratory-made electric vehicle and the preliminary localization and path tracking results. For example, the use of robust MPC based on safe reinforcement learning [31] would be an important direction to improve the path tracking performance. Moreover, due to the campus driving speed and route limitation of the small campus of NTUST (90,000 m²), this paper just showed the preliminary results on the successful integration of vehicle localization and path tracking. Moving the vehicle to a more complicated test environment would be another important future work.

On the other hand, obstacle avoidance was not a purpose of this work. The 360° scan formed with two lidar devices temporarily stopped the driving of EVs during path tracking when any obstacles were detected to ensure safety. In the future, an effective reactive obstacle avoidance approach will be further considered to improve autonomous driving efficiency in a dynamic obstacle environment. Moreover, the proposed UKF can further combine 3D-LiDar and visual SLAM to improve the vehicle positioning robustness. On the other hand, only three components in $Q_n$ were considered for RL in this paper. In the future, all parameters in this matrix will be considered and evaluated.