M²Tames: Interaction and Semantic Context Enhanced Pedestrian Trajectory Prediction

Abstract

Autonomous driving pays considerable attention to pedestrian trajectory prediction as a crucial task. Constructing effective pedestrian trajectory prediction models depends heavily on utilizing the motion characteristics of pedestrians, along with their interactions among themselves and between themselves and their environment. However, traditional trajectory prediction models often fall short of capturing complex real-world scenarios. To address these challenges, this paper proposes an enhanced pedestrian trajectory prediction model, M²Tames, which incorporates comprehensive motion, interaction, and semantic context factors. M²Tames provides an interaction module (IM), which consists of an improved multi-head mask temporal attention mechanism (M²Tea) and an Interaction Inference Module (I²). M²Tea thoroughly characterizes the historical trajectories and potential interactions, while I² determines the precise interaction types. Then, IM adaptively aggregates useful neighbor features to generate a more accurate interactive feature map and feeds it into the final layer of the U-Net encoder to fuse with the encoder’s output. Furthermore, by adopting the U-Net architecture, M²Tames can learn and interpret scene semantic information, enhancing its understanding of the spatial relationships between pedestrians and their surroundings. These innovations improve the accuracy and adaptability of the model for predicting pedestrian trajectories. Finally, M²Tames is evaluated on the ETH/UCY and SDD datasets for short- and long-term settings, respectively. The results demonstrate that M²Tames outperforms the state-of-the-art model MSRL by 2.49% (ADE) and 8.77% (FDE) in the short-term setting and surpasses the optimum Y-Net by 6.89% (ADE) and 1.12% (FDE) in the long-term prediction. Excellent performance is also shown on the ETH/UCY datasets.

1. Introduction

Pedestrian trajectory prediction (PTP) is a critical task in the fields of computer vision [1,2,3] and autonomous driving [4], marked by significant challenges. The core of this research extends beyond understanding the individual motion characteristics of pedestrians, also known as agents: it necessitates a thorough investigation into the complex interactions among agents as well as between agents and their environment. As illustrated in Figure 1, using the blue agent as an example, the interactions among agents and between agents and the environment are depicted. These interactions are not only multimodal but also highly dynamic, influenced by human self-awareness and social behavioral norms. Thus, predicting the motion trajectories of multiple agents requires modeling or considering two key factors: temporal aspects and social dynamics.

Temporally, the historical state of each agent impacts its future state. It is essential to model the temporal dependencies of each agent’s past state to predict their future trajectories. Socially, the states of surrounding agents and environmental information continuously influence the decision-making of a central agent. Therefore, modeling the interactions among agents and the complexities of the surrounding environment is necessary to capture intricate social dynamics.

Early models for crowd trajectory prediction primarily relied on manually designed energy functions to simulate interactions among individuals. This approach necessitates extensive feature engineering and is limited in capturing pedestrian interactions in busy scenes. However, the advancement of deep neural network technologies, particularly the application of Recurrent Neural Networks (RNNs) in pedestrian trajectory prediction, offers new perspectives to address this challenge. RNN-based models [1,2], by analyzing the hidden states of pedestrians, capture their motion trajectories and simulate interactions among individuals by integrating the hidden states of surrounding pedestrians. This approach has somewhat improved the understanding of social behavioral dynamics.

As a complement to RNN models, social-pooling [1,2] treats the states of neighboring pedestrians equitably and uses pooling techniques to integrate these states, thereby considering the spatial relationships between agents. In contrast, attention mechanisms [5,6], as advanced models, differentiate the importance of nearby pedestrians through learned weighting functions, providing a more flexible approach to trajectory prediction. Despite these advances, existing attention mechanisms still exhibit limitations in fully simulating the complex interactions among agents and between agents and their environment [7].

The traditional attention mechanism faces challenges in handling complex interactions among multiple pedestrians. In scenarios such as intersections, agents may exhibit various interaction modes, such as avoiding and merging. Traditional attention mechanisms struggle to adapt flexibly to these variations, resulting in a suboptimal performance in capturing relationships between multiple agents. Furthermore, the conventional single-head attention mechanism is inadequate in processing multi-scale inputs. In pedestrian trajectory prediction, agent movements may involve different spatio-temporal scales. The single-head structure of traditional mechanisms cannot simultaneously focus on and adapt to these varying scale features, thereby limiting the model’s capacity to handle multi-scale information.

On the other hand, in many traditional models, the environment is treated as a static backdrop or mere obstacles, without an in-depth consideration of its complex semantic information, such as road signs and traffic signals. This information is crucial for understanding and predicting pedestrian behavior patterns in specific environments. For instance, in urban street scenes, pedestrians’ choices of crossing locations are often influenced by the presence or absence of crosswalks. They tend to safely cross at locations with traffic signals, rather than riskily traversing streets with dense vehicle flow. Effectively integrating spatial information with pedestrian interaction data remains an open question.

In response to these challenges, this study introduces an innovative deep learning model M²Tames (multi-head mask temporal attention motion, interaction, semantic context) to improve the accuracy and adaptability of trajectory prediction. The new model encompasses not only the motion characteristics of agents but also the interactive features among agents as well as between agents and their environment. By employing an innovative multi-head mask temporal attention mechanism M²Tea, M²Tames is capable of simultaneously focusing on the agent’s motion characteristics and the interactive features between agents. Furthermore, I² allows the model to capture complex social behavioral patterns among agents, such as avoidance, following, and congregating. A deep understanding of these interaction modes significantly supports prediction accuracy. Lastly, through the integration of the U-Net architecture, M²Tames learns and interprets scene semantic context, thereby deepening the understanding of spatial relationships between agents and their environment.

In this context, our research aims to further enhance the accuracy of pedestrian trajectory prediction. Compared to Agentformer [5], our proposed model, M²Tames, not only considers the agents’ historical trajectories but also incorporates semantic information about the environment. The introduced M²Tea module employs advanced multi-scale feature processing strategies to address the shortcomings of previous works that focused solely on interactions among agents or solely between agents and the environment. Moreover, an improved interaction module, I², has been developed. Unlike previous models [1,2,6,7], this module emphasizes the importance of considering the movement patterns of nearby agents for accurately predicting pedestrian trajectories. This model is more effective in learning and adapting to complex interactions between agents and their environments, especially in densely populated scenarios.

The contributions of this paper are as follows:

• This paper introduces a deep learning model M²Tames for pedestrian trajectory prediction. The model comprehensively considers the motion characteristics of agents and the interactions among themselves, or with their environment, thereby improving the accuracy and adaptability of trajectory prediction;
• M²Tea designs a multi-head mask temporal attention mechanism that deeply characterizes temporal and interactive features, extracting the dynamic interaction features of pedestrians to enhance the model’s ability to learn and simulate real-world scenarios. Additionally, I² is designed to infer specific interactive relationships between agents. The goal of I² is to deduce the interactive relationships between agents based on the input motion data;
• Extensive experiments are conducted on the proposed M²Tames using the publicly available real-world datasets from SDD and ETH/UCY, and the results demonstrate that M²Tames achieves high accuracy in predicting pedestrian trajectories in both long-term and short-term settings.

2. Related Work

2.1. Interaction Modeling

Agent-to-agent interactions: In the field of trajectory prediction, the application of artificial neural networks leads to significant advancements. Social-LSTM [1], as a pioneering work, employs LSTM networks to model pedestrian dynamics and captures interactions among individuals through social pooling. SR-LSTM [3] further quantifies these interactions using a state refinement module’s weighted mechanism. Social-GAN [2] introduces generative adversarial networks, offering diverse possibilities for trajectory prediction tasks for the first time. Sophie [8] integrates attention mechanisms to merge scene and social information for trajectory prediction. Trajectron++ [9] proposes a modular, graph-structured recurrent model to predict the future trajectory distribution of multiple pedestrians. Conditional Generative Neural System (CGNS) [10] combines the advantages of conditional latent space learning and variable dispersion minimization to learn the future trajectory distribution space. PECNet [11], employing an endpoint-conditioned encoder, simulates human trajectories and generates socially compliant trajectories through social spatio-temporal graphs. Despite these advances, these models still face challenges in handling complex social interactions.

I², proposed in this study and inspired by the work of [12], comprehensively considers agents’ historical trajectories and the motion patterns of nearby agents. It learns interaction types through multi-scale feature processing and effectively reduces the complexity and prediction error in pedestrian trajectory forecasting.

Agent-to-environment interactions: Early trajectory prediction research relies on physical models and statistical techniques, such as the social force model [13,14]. However, these models often neglect environmental semantic information, especially in complex urban settings. With the advancement of deep learning, particularly the application of Convolutional Neural Networks (CNNs), in image recognition and scene parsing, new trajectory prediction models begin to incorporate environmental semantics. For instance, Sophie [8] integrates attention mechanisms to merge scene and social information for trajectory prediction. Social Ways [15] uses generative adversarial networks to learn the distribution of pedestrian trajectories, emphasizing the generation of realistic trajectories while considering interactions among agents and between agents and the environment. SHENet [16] introduces a cross-modal interaction module for modeling the interactions between individual past trajectories and their surroundings. This study utilizes refined semantic segmentation through the modified U-Net architecture to learn spatio-temporal context information, aiding the model in better understanding interactions between agents and the environment.

2.2. Long-Term Trajectory Prediction

Recently, human motion prediction, particularly in the field of pedestrian movement, garners significant attention. Researchers categorize prediction problems into short-term (1–2 s), long-term (up to 20 s), and very long-term (over 20 s) forecasts. Various models developed for long-term prediction, such as Bera et al. [17], use Bayesian inference to learn global and local motion patterns for long-term trajectory prediction, including prediction based on vehicle routing and GPS data [18]. Andrey et al. [19] considered the semantic attributes of static environments for predicting long-term human movements, like utilizing the prior knowledge of environmental targets for planning-based predictions. Rudenko et al. [20] proposed long-term global motion prediction models based on Markov Decision Processes (MDPs). Tran [21] introduced a goal-driven long-term trajectory prediction model, simulating the process of determining pedestrian goals and their impact on future trajectories.

The motion prediction model in M²Tames utilizes the U-Net [22] architecture, originally from medical image segmentation. The U-Net encoder, through convolution and pooling operations, progressively extracts the scene semantic and complex features of the trajectory sequence. This process enables the model to develop a higher-level understanding of trajectories and scenes, encompassing a broad range of spatio-temporal context and global scene semantics. The design of U-Net allows the model to comprehend the global structure of pedestrian movement and long-term spatio-temporal relationships. Therefore, inspired by this, U-Net architecture combinations are designed as the core framework of M²Tames.

2.3. Transformer

The transformer [23] relies entirely on a self-attention mechanism to capture global dependencies in serialized inputs. It has been applied in multiple domains, including natural language processing [24], visual tasks [7,25,26], and speech recognition [27]. Particularly in visual tasks, such as Vision Transformers (ViT) [28], images are serialized into a series of tokens, and the global dependencies of the image are modeled using a transformer encoder. Recently, some studies have applied transformers to pedestrian trajectory prediction. STAR [7] utilizes temporal and spatial transformers to extract temporal dependencies and spatial interactions, while AgentFormer [5] employs an agent-aware transformer to learn representations from both temporal and spatial dimensions. Differing from previous research, inspired by AgentFormer [5], this paper designs a multi-head mask temporal attention mechanism M²Tea to deeply characterize temporal and interactive features. This mechanism is tailored to extract the dynamic features of pedestrians, thereby enhancing the predictive performance. This approach focuses on learning temporal and interactive features in pedestrian trajectories.

3. Proposed Approach

Figure 2 presents the architecture of the model proposed in this paper, dedicated to predicting pedestrian trajectories in various social environments. The structure of the model is divided into three main parts: the U-Net encoder, the innovative interaction module, and the U-Net decoder. Initially, pedestrian trajectories are processed through the U-Net encoder, which extracts features. This process transforms the trajectory data into high-dimensional feature representations. These features are then fed into the interaction module, specifically designed to capture the interactions between pedestrians and the impact of environmental factors. After the interaction analysis, the U-Net decoder processes the outputs of the interaction module to reconstruct the predicted trajectories. This design enables the model not only to consider individual historical movement information but also to dynamically understand and adapt to the behaviors of the surrounding pedestrians and environmental changes, thereby enhancing the accuracy and applicability of trajectory prediction.

In this section, M²Tames is introduced, a pedestrian trajectory prediction model that fully considers the motion characteristics of the agent itself, the interactions between agents, and the interactions with the environment. As shown in Figure 3, the core of the model architecture lies in modules M²Tea and I², which collaboratively learn the interaction features of agents.

The problem of multimodal trajectory prediction can be formalized as follows. Given an RGB scene $I_{rgb}$ and some historical locations sampled at the same frame rate for past timesteps $t_p=1,2,\dots ,T_{obs}$, M²Tames aims to predict the pedestrian trajectories $Y^{\prime }$ for future timesteps $t_f=T_{obs}+1,\dots ,T_{pred}$, where $T_{pred}$ is the predicted maximum future timestep. The past trajectory of pedestrian i($1\le i\le N$) in scene $I_{rgb}$ is defined as $X_i$, where N is the total number of pedestrians:

$$X_i=\left\{(x_i^{t_p},y_i^{t_p})\right|t_p=1,2,3,\dots ,T_{obs}\}$$

(1)

The future predicted and real trajectories of pedestrian i for timesteps $t_f=T_{obs}+1,\dots ,T_{pred}$ are represented as $Y_i^{\prime }$ and $Y_i$, respectively:

$$Y_i^{\prime }=\left\{(x_i^{t_f},y_i^{t_f})\right|t_f=T_{obs}+1,\dots ,T_{pred}\}$$

(2)

$$Y_i=\left\{(x_i^{t_f},y_i^{t_f})\right|t_f=T_{obs}+1,\dots ,T_{pred}\}$$

(3)

3.1. Feature Initialization

The objective of the feature initialization layer l is to obtain initial features, denoted as $M_i^l\left(t\right)\in {\mathbb{R}}^D$, where the feature of the i-th agent is composed of D feature coordinates, and there are N agents in total. Given the past trajectory of the i-th agent as $X_i\in {\mathbb{R}}^{2\times T_{obs}}$, the initial feature of the i-th agent at the layer $l+1$ is defined as follows:

$$\begin{array}{c}{M}_i^{l+1}\left(t\right)={\Phi }^l\left([Dis{t}_i\left(t\right);V_i\left(t\right);Ac{c}_i\left(t\right);{\theta }_i\left(t\right)]\right)\end{array}$$

(4)

$$\begin{array}{c}{Dist}_i\left(t\right)=X_i\left(t\right)-\overline{X}\end{array}$$

(5)

$$\begin{array}{c}{V}_i\left(t\right)={\parallel (X_i\left(t\right)-X_i(t-\Delta t))/\Delta t\parallel }_2\end{array}$$

(6)

$$\begin{array}{c}{Acc}_i\left(t\right)={\Vert (V_i\left(t\right)-V_i(t-\Delta t))/\Delta t\Vert }_2\end{array}$$

(7)

$$\begin{array}{c}{\theta }_i\left(t\right)=atan2(V_i^x\left(t\right),V_i^y\left(t\right))\end{array}$$

(8)

where ${\Phi }^l$ is the non-linear transformation function, $[·;·;·;·]$ is a concatenation operation, and $\overline{X}$ shows the average position of all agents across all past timesteps. $Dis{t}_i\left(t\right)$, $V_i\left(t\right)$, $V_i^x\left(i\right)$, $V_i^y\left(i\right)$, and $Ac{c}_i\left(t\right)$ represent the relative position, velocity, velocity for x coordinate, velocity for y coordinate, and acceleration of agent i at time instant t, respectively. $\Delta t$ is the time interval, $\left|\right|·{\left|\right|}_2$ represents the Euclidean norm (2-norm) of a vector, $\theta$ represents the current direction of an agent’s motion, and $atan2(·,·)$ is the arctangent function to return the angle for the given coordinates.

3.2. M²Tea: Multi-Head Mask and Temporal Attention

M²Tea is introduced, incorporating time stamping to integrate temporal information and employing a masking mechanism to differentiate between an agent and its surrounding agents. This approach allows the attention mechanism to distinguish agent identities during feature learning and to simultaneously focus on different parts of the input data, thereby capturing the diversity within the data. In pedestrian trajectory prediction, the proposed multi-head mask temporal attention mechanism comprehensively considers various factors affecting pedestrian movement, enhancing the accuracy of future trajectory prediction. The innovative aspects of the module are detailed below.

3.2.1. Time Encoder

In time series problems such as trajectory prediction, traditional attention mechanisms with positional information are insufficient to reflect temporal relationships between elements. Hence, positional encoding with timestamps [29] is introduced. Time encoder involves embedding time-related codes into each input of M²Tea, enabling M²Tea to differentiate between past and future time points. This capability allows us to consider the time serial when processing temporal data. The time encoder is defined as follows:

$$pt(t,2n)=sin({\displaystyle \frac{t}{{10000}^{\frac{2n}{d_{model}}}}})$$

(9)

$$pt(t,2n+1)=cos({\displaystyle \frac{t}{{10000}^{\frac{2n+1}{d_{model}}}}})$$

(10)

where $pt(t,2n)$ and $pt(t,2n+1)$ represent the even and odd dimensional encodings of the position at time t, respectively, the dimension of $pt$ matches the input dimension of the node embedding vector defined by $d_{model}$, and n is the position of the embedding vector.

3.2.2. Mutil-Head Mask Attention

Previous models [5,26] on pedestrian trajectory prediction using transformer models often overlooked the issue of losing the social identity information of agents and their surroundings. To address this problem, M²Tea is employed, the principle of which is explained in Figure 4, where the mask is assigned 0 or 1. If the i-th query $q_i$ and the j-th key $k_j$ belong to the same type of agent, the mask equals 1; otherwise, it is 0. This differentiation allows M²Tea to focus on an individual agent’s motion characteristics while analyzing interactions with other agents or the environment. The mask mechanism ensures that M²Tea can accurately capture and learn the agent-owned behavioral patterns and trajectory characteristics while understanding how agents interact within a group. Such an approach enhances the comprehensiveness and accuracy of trajectory prediction.

The multi-head mechanism constructs a model with multiple attention heads, and each attention head can learn different spatio-temporal relationships and feature representations. The weighted features calculated by each attention head are fused to form a richer and more diverse overall feature representation. To address complex interactions between multiple agents, given a simplified scenario with agents A and B, their interaction can be influenced by various factors, such as their current positions, relative velocity vectors, and goals.

However, traditional single attention mechanisms can not effectively capture these diverse interaction modes. Multiple heads are created in the multi-head attention mechanism, each focusing on different aspects of interaction information. One head can concentrate on capturing agent A’s potential destination and movement direction, while another could focus on agent B’s velocity and position. This multi-head structure allows M²Tames to simultaneously learn features from these different aspects; thereby, M²Tames can understand the complex interactions between A and B more comprehensively.

Introducing timestamps and a masking mechanism enable M²Tames to clearly distinguish between the identity and temporal information of the multiple agents during the learning process. The advantage of the multi-head mechanism lies in its ability to simultaneously process the information of different scales [23], making M²Tames more flexibly adaptable to the diverse spatio-temporal scales in pedestrian trajectory prediction. This design enhances the adaptability of M²Tames, allowing it to capture the complete dynamics of agent movement more effectively. The specific computational formulas are defined as follows:

$$A_{ij}^h\left(t\right)=so ftmax((Q_i^h\left(t\right)·{(K_j^h\left(t\right))}^T)/\sqrt{d_k})$$

(11)

$${\Lambda }_{ij}^h\left(t\right)=A_{ij}^h\left(t\right)·Mas{k}_{ij}\left(t\right)$$

(12)

$$O_{M^{2}Tea,i}\left(t\right)=\sum _{h=1}^{\mathcal{H}}\sum _{j=1}^N{\Lambda }_{ij}^h\left(t\right)·V_j^h\left(t\right)$$

(13)

where $Q_i^h\left(t\right)$ and $K_j^h\left(t\right)$ represent the projections of the i-th query and j-th key in the h-th attention head, respectively. The $d_k$ is the scaling factor to ensure the stability of the attention mechanism. ${\Lambda }_{ij}^h\left(t\right)$ is the attention weight considering the temporal mask. $\mathcal{H}$ is the number of attention heads, and $V_j^h\left(t\right)$ is the projection of the j-th value in the h-th head.

3.3. I²: Interactive Inference

In complex pedestrian trajectory prediction scenarios, the interaction relationships between agents are typically implicit and challenging to grasp directly. To effectively address this issue, an innovative Interaction Inference Module is proposed. The core objective of I² is to infer the potential interaction categories between agents, and the various categories of social behavior patterns include avoidance, leading, following, grouping, fishing, schooling, etc.

The I² is designed to infer the interaction relationships among agents i and j ($i,j\in [1,N]\forall i\ne j$), generating interaction weights $C_{ij}\in {[0,1]}^K$, where K is the number of interaction categories. I² calculates and takes the relative position, velocity, acceleration, distance, and direction of agents from past trajectories as the input of ${\Phi }_c$, a multilayer perception with $so f tmax$ activation function. The specific formulas are defined as follows:

$$\begin{array}{c}{m}_{ij}^l={\Phi }_m^l(M_i^l,M_j^l)\end{array}$$

(14)

$$\begin{array}{c}{P}_i^l={\displaystyle \sum _{j\in \mathcal{N}\left(i\right)}}{m}_{ij}^l\end{array}$$

(15)

$$\begin{array}{c}{M_i^l}^{\prime }={\Phi }_M^l(P_i^l,M_i^l)\end{array}$$

(16)

$$\begin{array}{c}{C}_{ij}^l=so f tmax({\Phi }_c^l({M_i^l}^{\prime },{M_j^l}^{\prime })/\tau )\end{array}$$

(17)

where $m_{ij}^l$ represents the influence of each agent i on its neighbor j. ${\Phi }_m^l$ is a neural network that processes the features of two agents, including a nonlinear activation function to capture complex interaction patterns between agents. $P_i^l$ represents the aggregation of information from neighboring agents. $\mathcal{N}\left(i\right)$ is the i-th agent’s neighboring set. ${\Phi }_M^l$ transforms $P_i^l$ and the initial features of agents $M_i^l$ into ${M_i^l}^{\prime }$, where ${\Phi }_M^l$ is a function implemented by a neural network. ${M_i^l}^{\prime }$ calculates the aggregation weights, and another MLP transforms features into interaction weights $C_{ij}$. The softmax function converts ${\Phi }_c^l({M_i^l}^{\prime },{M_j^l}^{\prime })$ into a probability distribution, ensuring that the output interaction weights are normalized. And $\tau$ is the temperature parameter to control the smoothness of the softmax distribution.

3.4. Aggregation

Aggregation operations can learn aggregation weights from the interaction categories obtained through I². These weights, used to aggregate the interaction features of neighboring agents, enable M²Tames to generate more accurate interaction features. This information is crucial for predicting the future movement of agents. The ${\Phi }_{e,k}^l$ is used to model the mode of interaction. The specific formulas are defined as follows:

$$e_{ij}^l=\sum _{k=1}^K{C}_{ij,k}{\Phi }_{e,k}^l(M_i^l,M_j^l)$$

(18)

$$M_i^l\left(t\right)=M_i^l\left(t\right)+\sum _{j\in \mathcal{N}\left(i\right)}{e}_{ij}^l·(M_j^l\left(t\right)-M_i^l\left(t\right))$$

(19)

where $e_{ij}^l\in {\mathbb{R}}^{\mathrm{D}}$ represents the aggregation weight between agents i and j. ${\Phi }_{e,k}^l$ is an MLP designated for the k-th interaction category, tasked with learning the complexity of the interaction between agents i and j. These weights, used to aggregate the motion features of neighboring agents, effectively capture the dynamics of their interactions.

Through this approach, the interaction weights and categories are integrated to update motion features. This aggregation process not only considers the individual characteristics of agents but also encompasses their relative relationships and the varied impacts of different types of interactions on the update of agent features.

3.5. Goal and Trajectory Prediction Module

The feature map HI from the last layer of the U-Net encoder is concatenated along the channel dimension with the feature distribution map $O_{Agg}$ that depicts past motion history. After concatenation, M²Tames obtains a fused tensor HS of dimensions $W\times H\times (C+O_{Agg})$. This tensor is then fed into a U-Net decoder architecture composed of L blocks, where each of the L decoding blocks processes the fused features from H_l and its up-sampling. These decoding blocks employ bilinear interpolation for up-sampling the feature maps and enhance the resolution of feature maps through two consecutive convolution layers followed by ReLU activation functions. The decoder utilizes skip-connection techniques to merge the feature map H_l outputted by the encoder, enriching the detail in the features. Finally, the decoder outputs a Spatial Probability Distribution Map (SPDM) through an output convolution layer followed by a pixel-level sigmoid activation function. The SPDM represents the probability of the observed agents at their final positions $T_{pred}$, given the history information from $t=0$ to $T_{obs}$.

3.6. Loss Function

This study designs a loss function inspired by Y-Net [30]. Specifically, M²Tames is designed as an end-to-end joint training method, which comprehensively considers two core tasks: predicting the distributions of agents’ trajectories and their goals. By introducing weight factors, a weighted combination strategy of binary cross-entropy loss can balance the importance of different tasks. The design of this loss function enables M²Tames to effectively balance the contributions of multiple tasks during the learning process and to more comprehensively understand the complexities of pedestrian trajectory prediction. By integrating considerations for both the predictive goal and trajectory distribution, M²Tames can flexibly adapt to various task requirements, providing robust support for enhancing the accuracy of future pedestrian trajectory predictions. The end-to-end joint training and multi-task learning strategy offer significant advantages for M²Tames’s performance in long-term pedestrian trajectory prediction tasks.

The goal loss, denoted as ${\mathcal{L}}_{goal}$, is measured using Binary Cross-Entropy ($BCE$) to quantify the difference between the predicted goal point distribution $\widehat{\mathcal{P}}(Y_i^{T_{{pred}^{\prime }}})$ and the actual target point distribution $\mathcal{P}(Y_i^{T_{pred}})$. The trajectory loss ${\mathcal{L}}_{traj}$ calculates the binary cross-entropy loss between the predicted distribution and the actual distribution for all points from $t=T_{obs}+1$ to $t=T_{pred}$. The total loss $\mathcal{L}$ is the weighted sum of these two losses. The importance of trajectory loss within the total loss is adjusted by the weight parameter $\lambda$. This paper utilizes the Adam optimization method to dynamically optimize the value of $\lambda$, achieving a more precise control of loss weights. The specific formulas are defined as follows:

$${\mathcal{L}}_{goal}={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}BCE(\mathcal{P}(Y_i^{T_{pred}}),\widehat{\mathcal{P}}(Y_i^{T_{{pred}^{\prime }}}))$$

(20)

$${\mathcal{L}}_{traj}={\displaystyle \frac{{\sum }_{i=1}^N{\sum }_{t=T_{obs}+1}^{T_{pred}}BCE(\mathcal{P}(Y_i^t),\widehat{\mathcal{P}}(Y_i^{t^{\prime }}))}{N\times (T_{pred}-T_{obs})}}$$

(21)

$$\mathcal{L}={\mathcal{L}}_{goal}+\lambda {\mathcal{L}}_{traj}$$

(22)

where N represents the number of agents. $\mathcal{P}$ represents the true distribution. $\widehat{\mathcal{P}}$ represents the distribution predicted by the model. $Y_i^{T_{pred}}$ represents the true target points. $Y_i^{T_{{pred}^{\prime }}}$ represents the predicted target points.

4. Results

4.1. Dataset and Experiments

4.1.1. Dataset

The Stanford Drone Dataset (SDD) [31] is widely utilized in computer vision and machine learning, focusing on pedestrian trajectory prediction in urban environments. The SDD has become an essential tool for pedestrian trajectory prediction research, especially in high-density and dynamically changing urban scenarios. Captured by drones from Stanford University at various campus locations, the dataset spans a variety of urban settings, from academic areas to residential dining zones and major roadways.

The SDD dataset includes 1136 trajectories of students in campus scenes, sampled at 30 FPS. For short-term predictions, following [32], the data are subsampled to achieve the recommended frame rate of 2.5 FPS. The model observes for 3.2 s and predicts the subsequent 4.8 s. The midpoint of the original bounding boxes is used to achieve a coordinate representation that is consistent with the preprocessed short-term settings.

ETH/UCY Dataset is widely utilized in computer vision and machine learning, which includes pedestrian trajectory data collected from ETH Zurich and the University of Cyprus (UCY). The dataset encompasses five different public scenarios: a busy city center (ETH scene), a spacious hotel environment (HOTEL scene), a university campus (UNIV scene), and two shopping areas (ZARA1 and ZARA2 scenes). Each scenario provides multiple trajectories of pedestrians moving under natural conditions, manually annotated through video analysis. The diversity of these scenarios and the complexity of pedestrian interactions make the ETH/UCY dataset an ideal choice for evaluating and testing pedestrian trajectory prediction models, particularly in analyzing the impact of social interactions and environmental factors on pedestrian behavior.

4.1.2. Metrics

Two error metrics are employed to evaluate the effectiveness of M²Tames: Average Displacement Error (ADE) and Final Displacement Error (FDE). ADE measures the mean Euclidean distance between all corresponding positions in the predicted trajectory and the ground truth trajectory, reflecting the overall alignment of the predicted trajectory with the actual trajectory. The specific formula for ADE is defined as follows:

$$ADE={\displaystyle \frac{{\sum }_{i=1}^N{\sum }_{t=T_{obs}+1}^{T_{pred}}\left|\right|(x_t^i,y_t^i)-({\tilde{x}}_t^i,{\tilde{y}}_t^i){\left|\right|}_2}{N\times (T_{pred}-T_{obs})}}$$

(23)

Final Displacement Error (FDE) considers the Euclidean distance between the endpoint of the predicted and the actual trajectory, reflecting the accuracy of the endpoint position prediction. The specific formula is as follows:

$$FDE={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left|\right|(x_{T_{pred}}^i,y_{T_{pred}}^i)-({\tilde{x}}_{T_{pred}}^i,{\tilde{y}}_{T_{pred}}^i){\left|\right|}_2$$

(24)

Given that experiments take place in a stochastic environment, ADE and FDE are reported based on the minimum error values among 20 prediction samples. This approach generates 20 prediction samples for each input trajectory and selects the sample with the least error. This method provides a more comprehensive assessment of the model’s predictive accuracy under varied conditions.

4.1.3. Data Preprocessing

To adapt to model requirements, to enhance algorithm efficiency, and to address computational resource limitations when processing large datasets, preprocessing is conducted on RGB scene images $I_{rgb}$ and trajectory heatmaps $H_{map}$. For the datasets, a quarter of the samples are randomly selected from the original data for both $I_{rgb}$ and $H_{map}$. All images and trajectories are standardized using a division factor and resized with a scale factor of 0.2, along with perspective transformation processing.

Various data augmentation techniques are employed to increase the diversity of training data. First, all scene images and trajectories undergo horizontal and vertical flips at ${90}^{°}$, ${180}^{°}$, and ${270}^{°}$. Second, a Gaussian convolution kernel, with a kernel size of 31 and a standard deviation of 4, is used for image smoothing, adjusting contrast levels between pixels, and changing the color distribution and overall brightness levels of the images.

Through these methods, the quantity of training data becomes eightfold to provide large training samples to enhance M²Tames’s performance and generalization capability. Such data augmentation strategies not only alleviate overfitting issues but also enable M²Tames to better adapt to various scenes and trajectory conditions.

4.1.4. Experimental Parameters

In all experiments, during the attention calculation phase, the dropout rate is set to 0.1, and the number of heads for multi-head agent-aware attention is 4. This setup allows mapping trajectory to coordinate into different feature spaces to study the mutual influences between trajectories. In the interaction and aggregation learning phases, the numbers of feature learning layers are all set to four. The input for each layer is the output from the previous layer, iterating four times.

In the training stage, M²Tames uses an Adam optimizer as the optimization algorithm for the loss function’s gradient descent, with a weight decay coefficient set at $1\times {10}^{-3}$. The training consists of 300 iterations with a batch size of 32. All Multi-Layer Perceptrons (MLPs) in M²Tames have two layers, each equipped with the ReLU activation function. M²Tames, built using Python 3.8 on Pytorch 1.7.1, is trained on a single NVIDIA RTX-4090 GPU.

4.2. Quantitative Results

Table 1. Short-term pedestrian trajectory prediction on the SDD dataset.

Method	Year	ADE	FDE
Social-LSTM [1]	2016	57.00	31.20
Social-GAN [2]	2018	27.23	41.44
CF-VAE [33]	2019	12.60	22.30
Goal-GAN [34]	2020	12.20	22.10
P2TIRL [35]	2020	12.58	22.07
PECNET [11]	2020	9.96	15.88
SimAug [36]	2020	10.27	19.71
MG-GAN [37]	2021	13.60	25.80
LB-EBM [38]	2021	8.87	15.61
CAGN [38]	2022	9.42	15.93
Multiclass-SGCN [39]	2022	14.36	25.99
D-STGCN [40]	2023	15.18	25.50
MSRL [6]	2023	8.22	13.39
M2Tames	2024	8.02	12.31

presents the results for the Stanford Drone Dataset (SDD) in short-term settings, specifically with $t_p=3.2$ s and $t_f=4.8$ s. According to the established protocol delineated in Trajnet [32], leave-one-out cross-validation was employed. With 20 prediction samples, M²Tames achieved an Average Displacement Error (ADE) of 8.02 and a Final Displacement Error (FDE) of 12.31, surpassing all competing models. Notably, it outperformed the latest MSRL [6] by margins of 2.49% for ADE and 8.77% for FDE. The model produces a separate prediction of $K_a$ for each given goal. In short time horizons, because the total path is short, the paths to a given goal are similar to each other. Thus, the model is set at $K_a=1$.

In short-term pedestrian trajectory prediction, M²Tames demonstrates significant advantages through its innovative combination of M²Tea, I², and the U-Net architectures. This integration enables the model to effectively capture and leverage pedestrians’ historical trajectory, particularly in applications involving complex interactions and dynamic environments. Both the ADE and FDE metrics demonstrate that the model achieves lower errors in predicting short-term pedestrian trajectories. This underscores its advanced capability in understanding pedestrian behavior dynamics, as well as their interactions with the environment and other pedestrians. Consequently, M²Tames exhibits strong practicality and broad application potential in handling short-term prediction tasks.

Table 2. Short-term pedestrian trajectory prediction on the ETH/UCY dataset.

Method	ETH	HOTEL	UNIV	ZARA1	ZARA2	AVG
Social-LSTM [1]	1.09/2.35	0.79/1.76	0.67/1.40	0.47/1.00	0.56/1.17	0.72/1.54
Social-GAN [2]	0.81/1.52	0.72/1.61	0.60/1.26	0.34/0.69	0.42/0.84	0.58/1.18
ST-GAT [41]	0.65/1.12	0.35/0.66	0.52/1.10	0.34/0.69	0.29/0.60	0.43/0.83
Transformer-TF [42]	0.61/1.12	0.18/0.30	0.35/0.65	0.22/0.38	0.17/0.32	0.31/0.55
STAR [40]	0.36/0.65	0.17/0.36	0.31/0.62	0.26/0.55	0.22/0.46	0.26/0.53
Trajectron++ [9]	0.39/0.83	0.12/0.21	0.20/0.44	0.15/0.33	0.11/0.25	0.19/0.41
Goal-GAN [34]	0.59/1.18	0.19/0.35	0.60/1.19	0.43/0.87	0.32/0.65	0.43/0.85
PECNet [11]	0.54/0.87	0.18/0.24	0.35/0.60	0.22/0.39	0.17/0.30	0.29/0.48
$V^2$-Net [42]	0.23/0.37	0.11/0.16	0.21/0.35	0.19/0.30	0.14/0.24	0.18/0.28
MG-GAN [37]	0.47/0.91	0.14/0.24	0.54/1.07	0.36/0.73	0.29/0.60	0.36/0.71
D-STGCN [40]	0.63/1.03	0.37/0.58	0.46/0.78	0.35/0.56	0.29/0.48	0.42/0.68
MSRL [6]	0.28/0.47	0.14/0.22	0.24/0.43	0.17/0.30	0.14/0.23	0.19/0.33
M2Tames	0.27/0.39	0.11/0.16	0.21/0.43	0.17/0.26	0.15/0.20	0.18/0.28

presents the results for the short-term setup on the ETH/UCY datasets. Employing the standard protocol from Trajnet [32], which includes leave-one-out cross-validation and uses 20 prediction samples, M²Tames achieves a mean displacement error (ADE) of 0.18 and a final displacement error (FDE) of 0.28. These metrics surpass those of all competing models, including the recent MSRL.

An analysis of experimental results indicates that the model exhibits exceptional performance on the Hotel, Zara1, and Zara2 datasets. An in-depth examination of these datasets reveals that the model performs particularly well in scenarios involving dense crowds. This superior performance is attributed to the innovative modules introduced, which enable a more precise depiction of interactions among agents, thereby enhancing the model’s predictive accuracy.

M²Tames not only achieves technical innovation but also demonstrates significant effectiveness in practical applications. By integrating advanced attention mechanisms, the model not only excels on a single dataset but also maintains a high level of predictive accuracy across various environments and dynamic conditions. Moreover, the model’s consistent performance across different scenarios provides strong evidence of its scalability and adaptability for real-world applications. These results offer a new technical pathway for pedestrian trajectory prediction and hold important implications and application value for research in intelligent transportation systems, urban planning, and security monitoring.

Table 3. Long-term pedestrian trajectory prediction.

Method	Ka	ADE	FDE
Social-GAN [2]	1	155.32	307.88
PECNET [11]	1	72.22	118.13
Y-net [30]	1	47.94	66.71
M2Tames	1	44.85	65.97
M2Tames	2	44.91	62.65
M2Tames	5	39.23	64.33

presents the results for the Stanford Drone Dataset (SDD) in long-term settings, predicting future trajectories for $t_f=30$ s, based on a past motion history of $t_p=5$ s. All reported errors are in pixels, with lower values indicating better performance. M²Tames achieved an Average Displacement Error (ADE) of 44.85 and a Final Displacement Error (FDE) of 65.97. The proposed model, M²Tames, surpasses all the others, outperforming the best-performing Y-Net [30] by 6.89% (ADE) and 1.12% (FDE). Y-Net introduced a long-term trajectory prediction task with a prediction range of up to 30 s. For benchmarking purposes, the results of Y-Net [30] are assessed using the published source code, while the results for other models come from [30].

For long-term pedestrian trajectory prediction, M²Tames demonstrates its exceptional performance in extended sequence forecasting by adjusting the $K_a$ parameter. As the $K_a$ value increases, the model not only maintains high accuracy but also significantly reduces ADE and FDE metrics, even in the most challenging long-term prediction scenarios. This reflects the model’s effective handling of long-term dependencies. Such performance advantages are particularly suited for applications requiring highly accurate long-term predictions, such as intelligent transportation systems and urban monitoring. This underscores the practical value of M²Tames in urban planning and security monitoring systems.

M²Tames, through its innovative multi-head mask temporal attention mechanism and U-Net architecture, demonstrates superior performance in both short-term and long-term pedestrian trajectory prediction tasks. These performance metrics validate the model’s effectiveness in understanding pedestrian dynamics and interactions in complex scenarios. Whether in short-term or long-term prediction tasks, M²Tames provides robust data support and decision-making foundations for applications.

Further analysis is conducted on the impact of different spatial layers L within the multi-layer spatial module on the model’s performance. As shown in Figure 5, there is a noticeable downward trend in both Average Displacement Error (ADE) and Final Displacement Error (FDE) with the increase in the number of spatial layers. However, as L increases, a significant amount of time and space is required for training and testing. Observing the trend, it is evident that the rate of error reduction diminishes. The benefits gained from adding more layers do not justify the additional time and space required. Therefore, this study limits the number of layers to four.

The generalizability of the model discussed in this paper was also tested based on the number of samples $K_a$ used during testing. As depicted in Figure 6, both Average Displacement Error (ADE) and Final Displacement Error (FDE) decrease as $K_a$ increases. Moreover, under the same number of samples, the model presented in this paper outperforms the MSRL model, indicating that our model achieves the same level of error with fewer samples. These results demonstrate that the M²Tames introduced captures the potential interactions among agents as well as between agents and the environment more accurately, reducing the variance in predicted trajectory distributions. This leads to an enhanced accuracy and a better generalization capability of the predictions.

4.3. Ablation Study

Table 4. Ablation study.

Components				Performance
Variant	${\mathit{M}}^{\mathbf{2}}$	${\mathit{I}}^{\mathbf{2}}$	U	ADE	FDE
(1)	×	×	√	10.85	21.85
(2)	√	×	√	8.75	13.83
our	√	√	√	8.02	12.31

illustrates the results of M²Tames structure’s ablation study. The components, $M^2$ for the multi-head mask temporal attention module, $I^2$ for the Interaction Inference Module, and U for the U-Net architecture, are evaluated to demonstrate their individual and combined impacts on the Average Displacement Error (ADE) and Final Displacement Error (FDE). The outcomes indicate that integrating the agent’s motion characteristics, interaction among agents, and interactions with the environment produces the best results, and removing any component from M²Tames leads to a decrease in performance. These results validate the contribution of each module to M²Tames. The effectiveness of the proposed model M²Tames is thus demonstrated.

From the sustained performance improvements introduced by the M²Tea and I² components, it is evident that the model focuses on deepening the understanding of interactions among agents. The model has targeted and innovative enhancements in this area. Furthermore, the model integrates semantic information from the environmental context, a distinctive combination that enables a more comprehensive analysis and prediction of agents’ future trajectories. This approach, which holistically considers agent interactions and environmental factors, not only enhances the accuracy of trajectory prediction but also boosts the model’s applicability and robustness in complex scenarios.

4.4. Qualitative Results

The predicted trajectories are visualized in several scenarios to illustrate the validity of our M²Tames compared with MSRL, S-GAN, and STGCNN. Figure 7 shows that M²Tames successfully considers the motion characteristics of the agent itself, along with interactions among agents and between agents and their environment, and that it improves the accuracy and adaptability of trajectory prediction. Figure 8 shows the predicted trajectories in both scenarios, indicating that M²Tames makes more realistic trajectory predictions. Figure 8a shows the DeathCircle image: in a complex circular scene that contains many agents, the prediction results of M²Tames show less bias and better convergence to the ground truth. Figure 8b shows the Hyang image: both M²Tames and others are close to the ground truth, but M²Tames presents more stable trajectories with lower amplitude oscillations.

5. Conclusions

In the field of pedestrian trajectory prediction, existing research has made significant progress, particularly in modeling social interactions among agents and the influence of the environment. However, as urban environments become increasingly complex, pedestrian trajectory prediction models need to more comprehensively capture and understand the motion characteristics of agents, the intricate interactions among agents, and agents’ interactions with the environment. The improved M²Tea and I² proposed in this study, combined with the interactive inference module and U-Net architecture, further enhance the characterization of these complex relationships. By accurately capturing pedestrians’ historical trajectory features and the potential interactions among agents, the proposed approach not only excels in prediction accuracy but also demonstrates high adaptability, effectively operating in dynamically changing environments. Compared to other models, the main contribution of this model lies in its comprehensive consideration of various factors that could influence an agent’s future trajectory. It not only analyzes the interactions among agents but also enhances the model’s ability to learn from environmental contextual information to better understand how these factors affect the agents’ future paths. Furthermore, by inferring the potential movement patterns among agents, the model significantly increases the precision and reliability of predictions.

Rigorous testing on the ETH/UCY and SDD datasets, along with comprehensive ablation studies, provides strong empirical support for the effectiveness of the model. This research not only validates theoretical advancements but also demonstrates practical improvements in real-world settings. Long-term and short-term trajectory prediction studies are both conducted. Ablation studies validate the effectiveness of the proposed model. Visualization results demonstrate its capability to predict multimodal trajectories, and the model outperforms recent state-of-the-art models on the ETH/UCY and SDD datasets.

The outcomes of this research are crucial for exploring the application of advanced machine learning techniques in autonomous driving systems, such as path planning for autonomous vehicles and robots. By predicting and interpreting complex pedestrian movements, these technologies can significantly improve the safety and efficiency of related systems.