Improved Attention Mechanism for Human-like Intelligent Vehicle Trajectory Prediction

Abstract

In order to overcome the low long-term predictive accuracy associated with mainstream prediction models and the limited consideration of driver characteristics, this study presents an enhanced attention mechanism for human-like trajectory prediction, which is based on Long Short-Term Memory (LSTM). A novel database structure is proposed that incorporates data about driving style and driving intent, pertaining to human factors. By utilizing the convolution computation of Convolutional Social-Long Short-Term Memory (CS-LSTM) for surrounding vehicles, spatial feature extraction around the target vehicle is achieved. Simultaneously, we introduce a dynamic driving style recognition model and a human-like driving intent recognition model to fulfill the output of the human-like module. From a temporal perspective, we employ a decoder attention mechanism to reinforce the emphasis on key historical information, while refining the attention mechanism based on driving style for human-like weight assignment. Comparative analysis with other models indicates that the proposed Driving Style-based Attention-enhanced Convolutional Social-Long Short-Term Memory (DACS-LSTM) model exhibits notable advantages in predicting human-like trajectories for long-term tasks. Visualizing the predicted trajectories of both the Attention-enhanced Convolutional Social-Long Short-Term Memory (ACS-LSTM) and our proposed model, and analyzing the impact of the human-like module on the predicted trajectory, shows that our model’s predicted trajectory aligns more closely with the actual one. By comparing the weight distribution of the conventional attention mechanism and the enhanced attention mechanism proposed here, and analyzing the trajectory changes in conjunction with the driving styles, it becomes evident that our proposed model offers a marked improvement.

1. Introduction

As the automotive industry advances, the number of motor vehicles domestically has seen a rapid surge, reaching 412 million by September 2022, with cars comprising 76.5% of this total [1,2]. This represents a year-on-year increase of 4.3% compared to the 395 million recorded in 2021. The burgeoning number of motor vehicles has subsequently intensified traffic congestion and accidents [3,4,5]. Statistics reveal that human factors, primarily stemming from inappropriate driver actions, account for 94% of total traffic incidents [6]. As such, accurately interpreting the driving intentions and predicting the future trajectories of drivers with varying driving styles in diverse environments have become pivotal tasks. Comprehensive analysis of driving styles to predict and understand driving behaviors, as well as incorporating driving styles into trajectory prediction models, is crucial for enhancing vehicle safety, access efficiency, and occupant comfort.

Trajectory prediction models can be broadly categorized based on the modeling approach into: physical models [7], classical machine learning-based methods [8], and deep learning-based methods [9].

Traditional prediction methods primarily rely on physical models. For example, Barth et al. utilized Kalman filtering, inputting image data to achieve high short-term prediction accuracy for vehicle trajectories [10]. Carvalho et al. employed interactive multi-model Kalman filtering (IMM-KF) to estimate and predict target vehicles’ locations [11]. Xie et al. [12] proposed an Interactive Multiple Model Trajectory Prediction (IMMTP) method, incorporating uncertainty in prediction models based on physics using an Unscented Kalman Filter. They also introduced random elements to address uncertainty in each maneuver’s trajectory through dynamic Bayesian network inference. However, Kalman filtering tends to be less accurate for long-term prediction as the target physical model grows complex due to iterative computation based solely on the current state.

Classical machine learning-based approaches perform trajectory prediction through a data-driven methodology. Liu et al. [13] utilized an integrated approach that combined low-level and high-level vehicle information with the driver characteristic and intention estimation (DCIE) model and the Gaussian Process (GP) model. This approach was tested and analyzed in a highway lane-changing scenario. Feng et al. [14] analyzed vehicle lateral motion trajectory based on the Next-Generation Simulation (NGSIM) dataset, optimized through combining a lane shift recognition model with 97% accuracy using the support vector machine’s nonlinear learning and pattern recognition capabilities. Gao et al. [15] utilized a hidden Markov model to construct a maneuver recognition model for ensuring long-term prediction accuracy, validating the prediction model with real vehicle tests. Jiang et al. [16] proposed a probabilistic vehicle trajectory prediction method based on a Dynamic Bayesian Network (DBN) model that integrates the driver’s intent, maneuver behavior, and vehicle dynamics, achieving accurate long-term trajectory prediction in both lane-keeping and lane-changing scenarios. Yet, most classical machine learning methods require strategies and methods to provide or identify predicted trajectories in advance, and struggle to manage complex environments effectively.

In recent years, with the widespread application of neural networks, deep learning-based prediction methods have gained considerable attention. Dai et al. [17] proposed a spatio-temporal LSTM prediction model to address the low prediction accuracy of dense traffic trajectories, validating the model based on the NGSIM dataset, demonstrating higher prediction accuracy. Xing et al. [18], leveraging the LSTM encoder–decoder structure, assigned a decoder to each driving style to achieve personalized trajectory prediction. Sheng et al. [19] proposed a graph-based spatial-temporal convolutional network (GSTCN) that uses past trajectories to predict future trajectory distributions for all neighboring vehicles, achieving excellent results when evaluated on the NGSIM dataset. Guo et al. [20] developed a trajectory prediction method that combines temporal attention and spatial attention mechanisms, integrating LSTM encoding with the ego vehicle motion trends using a dual attention mechanism. This proposed approach showed good prediction accuracy and model generalization when evaluated on the NGSIM dataset.

Recent research has indeed enhanced the accuracy of vehicle trajectory prediction as model complexity has increased, particularly in the deep learning prediction models that have been the focus in recent years. These models can observe trajectory characteristics in time and space. However, there is limited research that combines driver characteristics with human factors. Previous studies on personalized trajectory prediction models have assigned encoders and decoders according to driving styles, and the complexity of such models increases exponentially with further refinement of driving styles. To address the shortfall in personalized prediction, this paper proposes an enhancement to the neural network prediction model structure by introducing an attention mechanism and a human-like module. The significance and novelty of this study are as follows:

The primary significance of this study lies in the substantial improvement it brings to the accuracy of long-term trajectory predictions. By introducing an enhanced attention mechanism and a human-like module, the DACS-LSTM model outperforms conventional approaches, demonstrating a remarkable reduction in the root mean square error (RMSE) of predictions, particularly within the critical 3 to 4 s window. This enhancement is crucial for the safety and efficiency of intelligent vehicles in real-world scenarios, providing reliable predictions for longer-term planning and decision-making.

Another key contribution of this study is the integration of driver behavior attributes into the trajectory prediction model. Through the incorporation of driving style recognition and intention recognition models, the proposed approach bridges a significant gap in the existing literature. This novel inclusion enables the model to adapt predictions according to individual driver characteristics, leading to more realistic and contextually relevant trajectory estimations.

This study’s innovative database structure, rooted in CS-LSTM, enables the extraction of spatial features surrounding the target vehicle. This facilitates a more holistic understanding of vehicle–road–human interactions, a crucial aspect in achieving human-like trajectory prediction. By considering not only the vehicle dynamics but also the driving style and intention, the model achieves a higher level of fidelity in predicting real-world trajectories.

2. Prediction Model and Database Design

2.1. Human-like Trajectory Prediction Model Design

In this study, we improve the CS-LSTM prediction model with an attention mechanism, as originally proposed by Zhang et al. [21]. This improvement takes into account the driving style (abbreviated as DS) recognition model, considering both environmental factors and the dynamic human-like driving intention (abbreviated as DI) recognition model proposed in this study. These modifications cater to the unique characteristics of vehicle trajectory prediction. The design of our human-like vehicle prediction model is illustrated in Figure 1.

(1)

Spatially based feature extraction

Trajectory prediction necessitates consideration of the interaction between the target vehicle and surrounding vehicles. Using a method rooted in the Social-LSTM model for surrounding pedestrian interaction based on grid processing, this model employs a convolutional layer and pooling layer to extract the influence of surrounding vehicles’ features on the target vehicle.

In this study, we assign each vehicle within the divided vehicle segment as the target vehicle. We generate a grid with lengths of ‘grids_height’ and widths of ‘grids_width’ and store the IDs, as well as the horizontal and vertical positions of the surrounding vehicles, in a table. This table is used to filter the implied state (ht-1) of surrounding vehicles at the previous moment. The implied state of the prior moment corresponding to the ID is then embedded into the 3D tensor Social Tensor. For grids without vehicles, we use disposition-1. Finally, the feature information of surrounding vehicles is extracted via the convolutional and maxpooling layers.

(2)

Human-like module

When considering human factors, this study integrates dynamic driving style recognition with human-like driver intention recognition. A driving style is not an instantaneous feature. Hence, this paper calculates feature values such as the mean and variance of vehicle motion over a segment based on 35 frames of input data. We tag the ID, number, and spatial relationships of surrounding vehicles and input the processed data into the driving style recognition model to dynamically output the driver’s driving style. The driving style recognition model employs a three-layer fully connected neural network, with the hidden layer consisting of 128 neurons.

Regarding the recognition of the driver’s driving intention, we take into account the driver’s driving style. Furthermore, the time-to-collision (TTC) between the target vehicle and surrounding vehicles needs to be calculated. These data are then compiled and input into the driver’s driving intention recognition model. The design of the intention recognition model is depicted in Figure 2.

(3)

Driving-Style Based Improved Attention Mechanism

In terms of time, the historical trajectory of the target vehicle forms a temporal input, with each frame of data contributing differently to the final result. To reasonably leverage key information within the historical trajectory data, this model applies an attention-weighted feature extraction mechanism to the historical trajectory of the target vehicle.

Considering that drivers with different styles attribute varying degrees of importance to historical information—for instance, conservative drivers, as compared to aggressive ones, tend to consider earlier historical information and act only after collecting sufficient information—this paper introduces a driving-style based attention mechanism. This mechanism allows for human-like attention weight assignment, implemented as follows:

The improved attention mechanism necessitates a transfer learning driving style recognition model. This model replicates the parameters of the trained driving style recognition model, retains the parameters of the first three network layers, and replaces the model’s output layer with an output dimension corresponding to the temporal dimension of the input data, thereby forming a new driving style attention model. The model input consists of two parts: one, the input data for the driving style recognition model; and two, the set of implicit states of the decoder LSTM for each frame of data. The neural network output comprises the driving style weights Wd.

The new attention score is the product of the original attention score and the driving style weights:

$$e_t^{\prime }=W_d\ast e_t$$

(1)

The improved attention scores are passed through softmax to calculate the attention weights:

$$s_t=\mathrm{softmax}(e_t^{\prime })=\frac{e^{(e_t^{\prime })}}{{\displaystyle {\sum }_{t=1}^{obs}{e}^{(e_t^{\prime })}}}$$

(2)

The weighted summation of timing data’s implied states is used to obtain new encoder outputs:

$$c_t={\displaystyle \sum _{t=1}^{obs}{h}_t{s}_t}$$

(3)

2.2. Improved Database

Traditional studies typically create datasets centered on vehicle attributes and motion information, though these datasets often necessitate enrichment through multi-source data integration. In contrast, this study utilizes the HighD dataset, a rich repository of real-world driving data. The HighD (Highway Drone) dataset, one of the largest datasets of its kind, contains high-resolution, drone-captured observations from German highways. These observations offer valuable insights into real-world driving behaviors and interactions, captured from a bird’s-eye perspective. This unique vantage point enables the study of vehicular movement in great detail, providing superior understanding of complex driving scenarios, and thus facilitating the development of more advanced and accurate trajectory prediction models.

Improving upon traditional research, our input dataset includes three types of data: vehicle-related, road-related, and human factor-related data. The vehicle-related data incorporate vehicle motion information (v_s) and inherent vehicle attributes (v_g). The road-related data include the number of current road lanes and the ID of the lane in which the target vehicle is located. The human factor data introduce a human-like module that recognizes driving style and driving intention. The input data for the trajectory prediction model are represented as follows:

$$v_{input}=\{v_s,v_g,v_l,v_d\}$$

(4)

Regarding vehicle motion information, this paper selects the lateral and longitudinal positions and velocities, which significantly influence the future trajectory of the vehicle. The vehicle motion information is expressed as:

$$v_s=\{X,Y,v_x,v_y\}$$

(5)

As for inherent vehicle attributes, considering the significant size differences between cars and trucks on the road, which will influence future trajectories, we select the length and width dimensions of the vehicle as inherent attribute information. The vehicle inherent attribute information is represented as:

$$v_g=\{Width,Height\}$$

(6)

The lane information of a driving vehicle also impacts the planning of its future trajectory. The number of road lanes can be obtained from the road information file within the HighD dataset, as calculated using the following formula:

$$lane\_num=len1+len2-2$$

(7)

The lane information is expressed as:

$$v_l=\{lane\_ID,lane\_num\}$$

(8)

where len1 corresponds to upperLaneMarkings in the HighD dataset, which represents the lateral position of the lane lines in the upper lane, while len2 corresponds to lowerLaneMarkings in the dataset, representing the lateral position of the lane lines in the lower lane.

Regarding human factor information, this paper introduces a driving style recognition model that takes into account the environment, and a dynamic human-like driving intention recognition model to identify real-time changes in the driver’s driving style and intentions. The human factor information is represented as:

$$v_d=\{DS,DI\}$$

(9)

where DS is driver driving style and DI is driver driving intention.

3. Predictive Model Results and Analysis

3.1. Model Training

In the experiments, we input data spanning 7 s, including 2 s of historical trajectory and a predicted trajectory of 5 s. We extract feature values according to the model’s network design. The training, validation, and test sets are divided into a 7:1:2 ratio, respectively. The experiments are trained using a GTX3080Ti graphics card. We selected Adam as the training network optimizer, and the learning rate is self-adjusted, with the initial value set at 0.01. The learning rate decreases every 10 epochs, and the calculation formula is:

$$lr=\frac{1}{1+0.5\ast epoch}$$

(10)

Before executing long-term predictions, a pre-training model for short-term predictions needs to be trained first. The loss function used in this case is the root mean squared error (RMSE). For long-term predictions, the loss function used is a combination of RMSE and negative log-likelihood loss (NLL). During training, the model parameters are updated at each epoch and saved in the ‘net.pkl’ file, with an average calculated every 5 batches. The variation of the training loss value with training steps is illustrated in Figure 3. As can be observed from the figure, the loss value gradually decreases with the increase in the number of training iterations and basically stabilizes after approximately 3000 training steps.

3.2. Human-like Module Recognition Results

The training results for both the driving style recognition model and the vehicle intention recognition model are depicted in Figure 4 and Figure 5, respectively.

The results demonstrate that the driving style recognition model achieves considerable accuracy fairly quickly, reaching 99% recognition accuracy within just 200 training steps. Similarly, the intention recognition model performs impressively, attaining a 97.5% test accuracy within 100 training steps. These high accuracy rates signify that both models are effectively trained and can provide robust predictions related to driving styles and intentions.

3.3. Comparison of Different Forecasting Models

In order to compare the prediction performance of the proposed models cross-sectionally, the following mainstream prediction models are selected, respectively:

(1)

CV: a Kalman filter prediction model with uniform motion, which is a prediction model without training parameters.

(2)

LSTM: a traditional LSTM prediction model with an encoder–decoder structure.

(3)

CS-LSTM: one of the current mainstream prediction architectures, which uses spatial convolution pooling to extract surrounding vehicle features based on the traditional encoder and decoder architectures.

(4)

ACS-LSTM: based on CS-LSTM, the temporal attention mechanism is added and subtracted from the decoder.

(5)

DSCS-LSTM with an improved attention mechanism based on driving style proposed in this paper.

Each model uses RMSE as the evaluation index, and the model inputs 2 s of historical trajectory data and outputs the prediction error results for the next 5 s, as shown in the

Table 1. Comparison of prediction results of various models.

Algorithms	1 s	2 s	3 s	4 s	5 s
CV	2.032	4.091	6.179	8.324	11.524
LSTM	0.862	1.868	3.392	5.475	8.816
CS-LSTM	0.735	1.707	2.975	4.713	7.603
ACS-LSTM	0.540	1.479	2.665	4.602	7.363
DACS-LSTM	0.544	1.461	2.375	4.348	7.208

:

As illustrated in Figure 6, the prediction error for each model incrementally increases as the length of the predicted future increases. At the same time, the prediction accuracy generally decreases with the increasing complexity of the model. Notably, the model proposed in this study adjusts the weighted attention mechanism based on individual driving styles, incorporating human-like modules to feed the decoder with driving styles and intentions that characterize driver traits. While the short-term prediction accuracy is comparable to that of CS-LSTM, our model displays superior accuracy for long-term prediction tasks.

3.4. Human-like Module Analysis

Drivers with different driving styles can have varying responses to identical situations, leading to distinct future trajectories. To achieve human-like trajectory prediction, this paper trains a specific trajectory prediction model using driving styles to output future trajectories that align with driver characteristics.

To further analyze the impact of driving styles on trajectory prediction, we visualized the historical trajectories of similar vehicles within a certain segment, their future actual trajectories, and the predicted trajectories from two different models.

As shown in Figure 7, the solid line marked with an asterisk (*) represents the input historical trajectory, while the solid line denoted by a dash (-) corresponds to the actual future trajectory. The dashed line illustrates the predicted trajectory of the ACS-LSTM model, and the dotted line represents the predicted trajectory from our proposed human-like trajectory prediction model, which considers driving styles.

The driving style of the driver in the target vehicle No.1 is categorized as aggressive.

As depicted in Figure 7, this is a typical situation where two vehicles gradually approach each other in position, creating an interaction. Both models generate relatively satisfactory results for predicting the future trajectory of Vehicle No.1, being able to anticipate future trends based on the input data.

Since both models employ convolutional neural networks, they are capable of extracting the feature information of the surrounding vehicles. As Vehicle No.1 continues along its historical path, the models notice that Vehicle No.4 is closer to the left-hand side of the target vehicle and predict a future tendency to steer clear to the right.

The ACS-LSTM model is more sensitive to the surrounding vehicles, demonstrating a more pronounced avoidance trajectory within the next 2 s. In contrast, our proposed DACS-LSTM model takes into account the aggressive driving style of this vehicle, initiating the avoidance maneuver only when Vehicle No.4 is significantly closer. This aligns well with the driving characteristics of aggressive drivers, who typically have a relatively low threshold for the distance between vehicles.

As a result, while the two models are quite similar regarding short-term prediction, the prediction model that incorporates driving style achieves greater accuracy for long-term prediction.

3.5. Improved Attention Mechanism Analysis

The traditional LSTM recurrent neural network prediction model’s output is the hidden state from the final frame of data, suggesting that the ultimate prediction result is directly connected to the last frame of data. However, the original data’s previous data inputs cannot be explicitly modeled.

To further extract the crucial information in the temporal inputs, this model integrates the attention mechanism and improves the attention mechanism’s weight assignment based on the driving style. Figure 8 depicts the attention weight assignment for the target vehicle.

Figure 8a displays the weight distribution of the conventional attention mechanism of the ACS-LSTM model at a prediction time of 2 s. In contrast, Figure 8b shows the weight distribution of the attention mechanism combined with the driver’s driving style at the same time, where the driver of this target vehicle demonstrates an aggressive driving style.

From the trajectory map, it can be observed that an inflection point occurs for the surrounding vehicle No.4 between the 10th and 15th trajectory points. This inflection point represents crucial information within the historical data. The traditional attention mechanism correctly assigns a relatively high weight to this inflection point. However, the improved attention mechanism proposed in this study, which takes into account the aggressive driving style, assigns a lower, evenly distributed weight to the inflection point and places more emphasis on the hidden state of the more recent moments. This effectively realizes an attention weight allocation that is adapted to the driver’s style.

4. Discussion

In this section, we provide a comprehensive evaluation of the proposed approach, highlighting both its strengths and potential limitations.

4.1. Advantages of the Proposed Approach

• Enhanced Long-term Prediction Accuracy: The DACS-LSTM model demonstrates notable advancements in long-term trajectory prediction accuracy compared to conventional models. This improvement is attributed to the incorporation of the enhanced attention mechanism, which effectively leverages historical information.
• Consideration of Human Factors: By integrating driving style recognition and driving intention recognition modules, the proposed model accounts for critical human factors. This enables the prediction of trajectories that align with distinct driver characteristics and behaviors.
• Spatial and Temporal Feature Extraction: The model leverages a novel database structure based on CS-LSTM, allowing for convolutional computations for surrounding vehicles. This facilitates the extraction of spatial features around the target vehicle, enhancing prediction accuracy.

4.2. Shortcomings and Areas for Improvement

• Computational Complexity: One potential area for improvement lies in the computational complexity of the DACS-LSTM model, particularly when dealing with large-scale datasets. Future research efforts could focus on optimizing computational efficiency, ensuring scalability.
• Data Quality and Granularity: The model’s performance may be influenced by the quality and granularity of input data. Further studies could explore techniques to handle diverse and noisy real-world data sources, enhancing the model’s robustness.
• Consideration of Higher-Order Kinematics: Previous studies have indicated that incorporating higher-order kinematics such as acceleration and jerk may further increase the accuracy of trajectories [22,23], especially for human-like movements. Future research could investigate the integration of these parameters.
• Alternative Performance Metrics: The calculation of RMSE is employed for evaluating prediction accuracy. However, it is worth considering that while RMSE provides valuable insights, it may not always comprehensively represent prediction performance. In some cases, especially when dealing with dynamic and complex driving scenarios, incorporating metrics like maximum error could offer a more nuanced understanding of the model’s predictive capabilities. Further investigations into diverse evaluation metrics could provide a more comprehensive assessment of the model’s performance under varying conditions.
• Post-Prediction Compensation for Trajectory Deviation: Despite the significant advancements demonstrated in long-term prediction accuracy compared to conventional models, it is imperative to acknowledge a noteworthy observation. Our model excels in accuracy for the initial four seconds of prediction; however, beyond this timeframe, a discernible deviation from actual trajectories becomes evident. This observation necessitates further investigation and refinement. Efforts will be focused on refining the model architecture and training methodology to achieve a more consistent performance throughout the entire prediction horizon.

4.3. Future Research Directions

• Integration of Environmental Context: Future research could explore the integration of environmental context data, such as weather conditions and road layouts, to enhance the model’s adaptability to diverse driving scenarios.
• Real-Time Implementation and Hardware Optimization: Investigating real-time implementation and hardware optimization strategies would be valuable for deploying the DACS-LSTM model in practical, real-world applications.
• Validation in Complex Urban Environments: Extending the validation of the model to complex urban environments with high traffic density and intricate road networks would provide valuable insights into its performance under challenging conditions.
• Driver-Specific Adaptation: Further research could focus on developing adaptive mechanisms that allow the model to dynamically adjust to individual driver characteristics and behaviors in real time, enhancing its personalized prediction capabilities.

By critically analyzing the strengths and weaknesses of DACS-LSTM and outlining future research directions, we aim to contribute to the ongoing development of intelligent vehicle systems and advance the field of trajectory prediction.

5. Conclusions

In this study, we present a novel, human-like trajectory prediction model, underpinned by an enhanced attention mechanism. This development addresses two primary challenges in the field: the limited accuracy of long-term predictions offered by conventional models, and the insufficient integration of predicted trajectory models with driver behavior attributes. The forthcoming trajectory of a target vehicle is shaped by the triad of vehicle–road–human interactions. Consequently, we designed an innovative database structure rooted in CS-LSTM, which enables convolutional computations for surrounding vehicles and thus facilitates the extraction of spatial features around the target vehicle. Concurrently, we introduce a dynamic driving style recognition model and a human-like driving intention recognition model to furnish the output of the human-like module. From a temporal perspective, we employ a decoder attention mechanism to underscore crucial historical information, and further refine the attention mechanism based on driving style to realize human-like weight allocation. Cross-sectional comparisons with other models reveal that the DACS-LSTM model that we propose offers superior accuracy in long-term prediction tasks, reducing the root mean square error of prediction by 0 to 0.29 m. The most significant enhancement in prediction accuracy is observable within the 3 to 4 s window, substantiating the advantages of the human-like trajectory prediction model that leverages the enhanced attention mechanism. We visualize the predicted trajectories from various models, analyze the influence of the human-like module on predicted trajectories, and synthesize the performance of the two models. The predicted trajectories of this model exhibit a closer alignment with the actual trajectories. Lastly, we dissect and analyze the weight distribution of both the conventional attention mechanism and the improved attention mechanism, in conjunction with driving styles. In the future, we will work towards addressing the limitations of the existing research.