Human Trajectory Imputation Model: A Hybrid Deep Learning Approach for Pedestrian Trajectory Imputation

Abstract

Pedestrian trajectories are crucial for self-driving cars to plan their paths effectively. The sensors implanted in these self-driving vehicles, despite being state-of-the-art ones, often face inaccuracies in the perception of surrounding environments due to technical challenges in adverse weather conditions, interference from other vehicles’ sensors and electronic devices, and signal reception failure, leading to incompleteness in the trajectory data. But for real-time decision making for autonomous driving, trajectory imputation is no less crucial. Previous attempts to address this issue, such as statistical inference and machine learning approaches, have shown promise. Yet, the landscape of deep learning is rapidly evolving, with new and more robust models emerging. In this research, we have proposed an encoder–decoder architecture, the Human Trajectory Imputation Model, coined HTIM, to tackle these challenges. This architecture aims to fill in the missing parts of pedestrian trajectories. The model is evaluated using the Intersection drone the inD dataset, containing trajectory data at suitable altitudes, preserving naturalistic pedestrian behavior with varied dataset sizes. To assess the effectiveness of our model, we utilize L1, MSE, and quantile and ADE loss. Our experiments demonstrate that HTIM outperforms the majority of the state-of-the-art methods in this field, thus indicating its superior performance in imputing pedestrian trajectories.

1. Introduction

Pedestrian trajectory unpredictability has presented a formidable challenge for autonomous vehicles, necessitating sophisticated trajectory imputation techniques. The erratic and dynamic nature of pedestrians’ movement makes it challenging to accurately predict their future paths. Factors such as sudden changes in direction, varying speeds, and unexpected interactions with the environment contribute to this unpredictability. Figure 1 Bearing this unpredictability in mind, autonomous vehicles have transformed transportation by utilizing state-of-the-art technologies like artificial intelligence, sensor fusion, and machine learning to navigate safely and independently. Central to their operation is the ability to efficiently plan and follow trajectories while adapting to changing environments. In this context, a trajectory refers to the path an entity takes over time, including its spatial coordinates, velocity, and orientation. Mathematically, a trajectory $X\left(t\right)=\left(x\right(t),y(t),v(t),\theta (t\left)\right)$ describes the vehicle’s state in a specific coordinate system, where $\left(x\right(t),y(t\left)\right)$ denotes its position, $v\left(t\right)$ represents its velocity, and $\theta \left(t\right)$ indicates its heading at time t. Trajectory imputation is a critical aspect of motion prediction, focusing on filling in missing points within incomplete trajectories to create a comprehensive representation of an object’s movement. Essentially, trajectory imputation involves using existing trajectory data to interpolate or extrapolate missing segments, compensating for factors like sensor errors, occlusions, or data loss. By integrating predictive models with historical trajectory information, trajectory imputation facilitates the reconstruction of a coherent and uninterrupted trajectory, vital for applications such as autonomous navigation and activity recognition. The primary objective of trajectory imputation is to estimate the missing points inside the incomplete trajectory to yield a complete trajectory represented as follows:

$${\mathit{T}}_{\mathit{complete}}=\{({\mathit{x}}_1,{\mathit{y}}_1,{\mathit{t}}_1),({\mathit{x}}_2,{\mathit{y}}_2,{\mathit{t}}_2)\dots ({\mathit{x}}_n,{\mathit{y}}_n,{\mathit{t}}_n)\}$$

and

$$\begin{array}{cc}\hfill {\mathit{T}}_{\mathit{incomplete}}=& \{({\mathit{x}}_1,{\mathit{y}}_1,{\mathit{t}}_1),({\mathit{x}}_2,{\mathit{y}}_2,{\mathit{t}}_2)\dots \hfill \\ & ({\mathit{x}}_{n-m-1},{\mathit{y}}_{n-m-1},{\mathit{t}}_{n-m-1}),\hfill \\ & ({\mathit{x}}_{n-m},{\mathit{y}}_{n-m},{\mathit{t}}_{n-m}),\hfill \\ & ({\mathit{x}}_{n-m+p},{\mathit{y}}_{n-m+p},{\mathit{t}}_{n-m+p})\dots \hfill \\ & ({\mathit{x}}_n,{\mathit{y}}_n,{\mathit{t}}_n)\}\hfill \end{array}$$

where $p -$ 1 represents the number of missing points to be imputed, n denotes the total points of a complete trajectory, and m is a variable providing the continuity of sequential points.

Since the early days of trajectory imputation, several techniques have been employed, each with its advantages and limitations. These techniques include Kalman filtering [1], which is widely used for state estimation and trajectory prediction in autonomous vehicles. It provides a recursive solution for predicting the phase of a dynamic system dependent on noisy sensor measurements. Kalman filters are particularly effective in scenarios with linear dynamics and Gaussian noise distributions. Particle filters, also known as Monte Carlo localization [2], are probabilistic algorithms used for non-linear and non-Gaussian state estimation. They represent the posterior distribution of the vehicle’s state using a set of weighted particles. These particles are sampled and propagated based on motion models and sensor measurements. Bayesian inference provides a principled framework for updating beliefs about the vehicle’s state based on observed data and prior knowledge. It allows for the incorporation of uncertainty into trajectory prediction and facilitates decision making under uncertainty.

In recent times, machine learning methodologies, like support vector machines [3], neural networks, and Gaussian processes, have been increasingly employed for trajectory imputation in autonomous vehicles. These approaches learn complex relationships from data and can capture intricate patterns in vehicle motion and environmental interactions. The advancement of deep learning [4] fields has significantly influenced trajectory imputation in autonomous vehicle systems. Deep learning techniques have demonstrated remarkable capabilities in capturing intricate patterns and dependencies within trajectory data, leading to enhanced prediction accuracy and robustness. Several key techniques have emerged that harness the strength of deep learning models. Recurrent neural networks (RNNs), particularly Gated Recurrent Units (GRUs) and long–short-term memory (LSTM), and several of LSTM’s variants, like bidirectional LSTM (Bi-LSTM) [5], attention-based LSTM (AttLSTM) [6], stacked LSTM [7], convolutional LSTM (ConvLSTM) [8], LSTM with temporal convolutional networks (TCNs) [9], and peephole LSTM [10], have gained prominence in trajectory imputation tasks.

As we stated earlier, despite advancements in trajectory prediction, accurately imputing pedestrian trajectories remains a significant challenge due to the inherent unpredictability of human movement. Traditional techniques like Kalman filters and particle filters have limitations in handling non-linear and non-Gaussian dynamics, while existing machine learning models often struggle with complex, crowded environments and missing data. There is a critical need for a robust trajectory imputation method that can effectively handle the erratic nature of pedestrian movement and provide accurate predictions in real time.

Our research has made the following noteworthy contributions to this field of research:

• We have explored the challenges and requirements specific to trajectory interpolation for autonomous vehicles, considering the real-time nature and safety criticality of autonomous driving systems.
• We have utilized a masking process for converting complete trajectories into incomplete trajectories, which randomly eliminates a certain portion of points. This introduces more uncertainty and randomness, which ultimately enriches the training process.
• We have examined the use of LSTM and GRU in the travel path restoration of pedestrians, taking into consideration their propensity for managing temporal dependencies and contextual interactions.
• We have designed an encoder–decoder imputation model comprising LSTM blocks for the encoder and GRU blocks for the decoder.
• We have evaluated the effectiveness of our model on a curated version of inD dataset, marking one of the earliest instances of research on this dataset for the trajectory imputation task.

Section 2 provides an overview of the existing related literature. Section 3 identifies gaps in previous research. The dataset description and preprocessing methods are outlined in Section 4. Section 5 elaborates on the proposed methodology. The experimental results and a discussion are presented in Section 6. Finally, Section 7 presents the conclusions for the study and proposes avenues for future research. Moreover,

Table 1. Glossary of symbols and their definitions.

Symbol	Meaning
$\mathit{X}\left(t\right)$	Trajectory of a entity at time t, consisting of spatial coordinates, velocity, and orientation
$\left(\mathit{x}\right(t),\mathit{y}(t\left)\right)$	Position of the entity at time t
$\mathit{v}\left(t\right)$	Velocity of the entity at time t
$\theta \left(t\right)$	Heading of the entity at time t
${\mathit{T}}_{\mathit{complete}}$	Complete trajectory comprising all data points
$\mathit{p}-1$	Number of missing points in the trajectory to be imputed
${\mathit{T}}_{\mathit{incomplete}}$	Incomplete trajectory with missing points
$\alpha$	Masking ratio for simulating missing data in trajectory sequences
$X_i$	Trajectory of the i-th example
$T_i$	Length of the trajectory $X_i$
$T_{\max }$	Maximum length across different trajectories
$M_i$	Binary mask indicating observed and missing points in a trajectory
$H^{\mathrm{enc}}$	Hidden state sequence of the LSTM encoder
$H^{\mathrm{dec}}$	Hidden state sequence of the GRU decoder
Y	Input sequence for the GRU decoder
$Y^{\prime }$	Output sequence of the GRU decoder
$r_t$	Reset gate at time step t
$z_t$	Update gate at time step t
${\tilde{h}}_t$	Candidate hidden state at time step t

presents the symbols used in this paper.

2. Related Work

Missing value handling is one of the most difficult problems in the field of time series data. A wide variety of approaches, from complex convolutional recurrent encoder–decoder architectures to recurrent dynamical systems, have been thoroughly explored by researchers. By studying the literature, we have examined different approaches used to fill the gaps in work related to time series datasets, exposing the advantages and disadvantages of each.

Using a bidirectional recurrent dynamical system, Cao et al. [11] described a technique for inferring time series data that can directly estimate the missing value. There was no significant presumption about the underlying data generation method. The majority of the other current systems, according to the paper, made significant assumptions about the data generation process. According to Kreindler et al. [12], there was no abrupt spike in the data and the data were smoothable while generating the missing data. Therefore, smoothing the surrounding data could produce missing values. Some current methods have used interpolation to infer missing values; however, local interpolation has generated missing values without taking into account the relationship between the variables over time. The model BRITS, devised in the literature, has outperformed all baseline models in terms of mean absolute error and mean relative error.

Nawaz et al. [13] introduced a deep learning-based convolutional recurrent encoder–decoder architecture to address the GPS trajectory imputation problems. There were temporal and spatial components to GPS trajectory data. While recurrent neural networks such as LSTM were effective in predicting consecutive time series, convolutional neural networks excelled at extracting the spatial aspect from data. They were used by the suggested ConvLSTM to achieve better results in trajectory imputation tasks. The authors used GPS trajectories with transportation mode labels from the Microsoft Geolife trajectory dataset [14,15,16] and employed the Average Displacement Error (ADE) as an evaluation metric.

Ma et al. [17] proposed a novel deep neural network approach to address the human trajectory imputation problem, which is a crucial step in the COVID-19 contact tracing process. The representation process using graph embedding was the first component of the solution, and the transformer model was the second. The formation of the human trajectory sequence occurred in a high-dimensional environment, making it difficult to extract significant information. The nodes were mapped from high-dimensional to low-dimensional vector space using the graph embedding model to remove the challenges. This aided in capturing the sequential aspect of human trajectory as well as the geo-spatial aspects. A deep CNN decoder, a softmax classifier, and a transformer encoder-based feature extractor made up the deep learning model. Because it can understand linkages and dependencies, the transformer’s main feature, Multi-head Self-Attention, confirmed that the sequence-to-sequence model exhibited good performance. The CRAWDAD unm/blebeacon [18] dataset, an open-source human mobility dataset, was used for the study.

The problem of trajectory prediction from partial observations owing to the misdetection of dynamic agents (cars, pedestrians, etc.), which can be brought on by poor image quality or occlusion by other dynamic agents, was studied by Fujii et al. [19]. The standard method for handling the incomplete trajectory problem has been to consider misdetection cases to be anomalies and remove them from the dataset. Because such a strategy eliminates the possibility of coordination with the agents that are excluded, there is a higher chance of major mishaps. Traditional methods for trajectory prediction have included those based on Bayesian filters. Their fundamental structure has made them unreliable for long-term prediction. RNNs, most notably, LSTM, are now widely used in modeling to anticipate human motions. The study by Fujii et al. proposed a two-block RNN for trajectory prediction from a partial trajectory due to misdetection, which learned the inference step of Bayesian filters. Two publicly accessible datasets, ETH [20] and UCY [21], were used in the experiment.

Almolegi et al. [22] introduced a novel model called Move, Attend, and Predict (MAP), which forecasts a person’s next location based on motions previously captured by a mobile device. The model used a deep neural network to identify the important components of mobility history for each location. The program also used an approach called embedding representation learning to extract relevant properties from the movement data. The model was tested on two big datasets and demonstrated that it outperformed other models in location prediction. The study solely took into account two types of input data: locations and timings. It ignored other elements that may impact people’s movements, such as preferences, social networks, weather, traffic, etc. The model was tested on two datasets: Geolife and Foursquare [23].

Wang et al. [24] introduced a novel deep learning approach to address the prevalent problem of missing trajectory data in GPS applications. Their proposed method leveraged an RNN with an encoder–decoder architecture and an attention mechanism [25]. To enhance data reconstruction quality, they introduced a new loss function based on the R2 coefficient and implemented a smoothing technique using a moving average and a Savitzky–Golay filter [26]. Evaluating their model on a real-world GPS dataset from the Geolife project, comprising 17,621 trajectories collected over five years from 182 users, they demonstrated its superiority over several baseline methods, including linear interpolation, cubic spline interpolation, a Kalman filter, LSTM, and Bi-LSTM in terms of recovery accuracy and result stability across various missing rates and patterns. Additionally, through ablation tests and parameter sensitivity analysis, they validated the effectiveness of their model components and design choices, underscoring the potential of their deep learning approach for robust trajectory data reconstruction.

3. Gap Analysis

Table 2. Gap analysis of the reviewed literature.

Authors	Method	Dataset	Evaluation Metric	Limitation	Our Situation
Nawaz et al. [13]	Encoder collected input data to create a context vector. Decoder used it for trajectory prediction, aided by Beam Search for sequence optimization.	Geolife trajectory dataset	Average Displacement Error (ADE)	This method can identify patterns in common scenarios but may fail in complex, crowded trajectory movements.	The inD dataset we have used contains complex, crowded trajectory movements, which our model has successfully addressed.
Cao et al. [11]	BRITS-I Algorithm enhanced predictive accuracy with bidirectional recurrent dynamics, enabling error backpropagation from past and future observations in time series analysis.	PhysioNet, Beijing PM2.5, Electricity	Mean absolute error	Treating missing values as variables adds uncertainty and noise, potentially diminishing model performance.	Our model applied data imputation techniques to estimate the missing values based on the available data, thus decreasing the effects of noise and uncertainty.
Ma et al. [17]	A transformer-based deep neural network was used. The Multi-head Self-attention mechanism enabled the transformer to achieve good performance.	BLE data packet generated by 46 participants.	Model training loss, prediction accuracy metrics	The proposed approach may perform well on BLE data, but its effectiveness on other types of trajectory data has not been explored.	This problem also persists in our research, because apart from using the drone dataset, our model did not explore GPS or other kinds of datasets.
Almolegi et al. [22]	MAP consists of three core components: RNNs, an Attention Model, and a softmax classifier. RNNs capture spatial sequences, generating a high-level representation of historical movement summaries.	Geolife	Recall, F1, and Precision	One notable constraint is the model’s incapacity to handle new or previously unseen locations.	This limitation also lies in our research. We did not check our model’s capacity or incapacity to face unseen locations.
Fujii et al. [19]	The authors introduced a two-block recurrent neural network (RNN) architecture.	ETH and UCY	ADE and FDE	The introduction of a Bayesian filter framework provided additional complexity, which may have impacted scalability.	Our model, due to its complex architecture, can also suffer from limitation in scalability.
Wang et al. [24]	The authors leveraged an RNN with an encoder–decoder architecture with an attention mechanism.	Geolife	Accuracy	Other than accuracy metrics with varying p and lambda, no other evaluation metric was used.	We used L1 loss, MSE loss, ADE loss, and quantile training loss to test our model’s correctness from different aspects.

provides a comprehensive analysis of the limitations found in previous research. The gap analysis of the reviewed literature underscores several notable limitations encountered in prior research. Notably, Nawaz et al. demonstrated effectiveness in identifying patterns in common scenarios, but their method may struggle with complex, crowded trajectory movements.

Additionally, Cao et al.’s [11] approach, despite enhancing predictive accuracy, introduces uncertainty and noise by treating missing values as variables. The transformer-based model by Ma et al. [17] exhibits promise with Bluetooth low-energy data but lacks exploration on other trajectory data types. Almolegi et al.’s [22] MAP model faces constraints in handling new or unseen locations, while Fujii et al.’s [19] Bayesian filter framework adds complexity, potentially impacting scalability. Moreover, Wang et al.’s [24] model, leveraging an RNN with an encoder–decoder architecture, may overlook comprehensive performance assessment due to limited evaluation metrics. These findings underscore the necessity for further research to address these limitations comprehensively.

4. Dataset Observation and Prepossessing

In this section, we explore the dataset collection, observation, and prepossessing carried out as part of this research, providing a detailed overview of the steps taken to collect and prepare these data.

4.1. Dataset Observation

Various trajectory datasets exist for road users, but very few have met the needs of autonomous driving research. Examples include BIWI Walking Pedestrians [20], Stanford Drone [27], CITR and DUT [28], highD [29], Interaction [30], and Ko-PER [31]. Most lack sufficient data or public road scenarios. The inD dataset [32] is a new and unique dataset of naturalistic vehicle trajectories recorded at German intersections using a drone. The dataset includes more than 11,500 road users, including vehicles, bicyclists, and pedestrians, at four different locations (Braunschweig, Frankfurt, Lindau, Wurzburg) with varying traffic density and complexity.

Table 3. Overview of trajectory data and characteristics.

Title	Location	# Trajectories	# Locations	Road User Types	Data Frequency	Method
BIWI ETH	University building entry	360	1	pedestrian	2.5 Hz	stat. sensor
Interaction	Urban intersection	18,642	4	vehicles	10 Hz	drone/cam.
Ko-PER	Urban intersection	350	1	pedestrian, bicycle, car, truck	25 Hz	stat. sensor
Crowds UCY/ Zara	Campus, urban street	909	3	pedestrian	2.5 Hz	stat. sensor
Stanford Drone	Campus	10,240	8	pedestrian, bicycle, car, skateboard, cart, bus	25 Hz	drone
BIWI	Hotel sidewalk, hotel entry	389	1	pedestrian	2.5 Hz	stat. sensor
VRU Trajectory	Urban intersection	3278	1	pedestrian, bicycle	25 Hz	stat. sensor
DUT	Campus	1862	2	pedestrian, vehicles	23.98 Hz	drone
CITR	Designed experiment	340	1	pedestrian, golf-cart	29.97 Hz	drone
inD	Urban intersection	11,500	4	pedestrian, bicycle, car, truck, bus	25 Hz	drone

# Number of trajectories and Number of locations.

shows a comparison of the inD dataset with other datasets [32].

The inD dataset surpasses other datasets in capturing naturalistic road user behavior due to its strategic design principles:

• Preserving naturalistic behavior: By avoiding visible sensors resembling traffic surveillance cameras, the inD dataset ensures road users are not impacted by the measurement method.
• Having satisfactory size: The inclusion of trajectories from thousands of road users provides the necessary size and variety for robust data-driven algorithms.
• Differentiating recording locations and times: The inD dataset captures measurements from multiple sites and diverse times of day, including public roads, ensuring the coverage of different road layouts, densities, and traffic rules for enhanced significance in automated driving scenarios.
• Detecting and tracking all road user types: Unlike datasets that limit themselves to specific categories, the inD dataset tracks all road users and recognizes the vital importance of capturing interactions across diverse user types.
• Tracking road users with top accuracy: The inD dataset guarantees trajectories with a positioning inaccuracy of less than 0.1 m, regardless of road user type, ensuring precision in capturing the intricacies of user movement.
• Including infrastructure details: With the precise recording of road layouts and local traffic rules, the inD dataset acknowledges the dependency of road user behavior on these factors, providing comprehensive information within the dataset.

4.2. Using Curated Dataset Prepared for Autonomous Vehicles

Muktadir et al. [33,34] dealt with human trajectories while keeping the concerns of autonomous vehicles in mind. Pedestrians walking by the side of each road should not be our concern. Road crossing pedestrians’ trajectories should be our focal point, as only then does the factor of autonomous vehicles come into play. Only by incorporating the trajectories of road-crossing pedestrians could they create a curated dataset, which we have used [33].

4.3. Making Incomplete Trajectories

In this section, we present our approach to creating incomplete trajectories through a masking strategy. We aim to simulate missing data in the trajectory sequences, which will be used during training.

4.3.1. Masking Strategy

To simulate missing data in trajectory sequences, we employ a comprehensive masking strategy that involves randomly selecting and discarding segments of the trajectory. This method ensures that the simulated missing data reflect realistic scenarios of incomplete data in trajectory analysis. Let $\mathit{X}$ represent a complete trajectory with T data points, where $x_t$ denotes the data point at time step t. We define the masking strategy as follows:

• Masking Ratio ($\alpha$): We introduce a hyperparameter $\alpha$ which represents the percentage of data points we want to mask in each trajectory. The value of $\alpha$ is determined during training and can be fine-tuned for optimal performance.
• Random Selection: To simulate missing data, we randomly select a starting point within the trajectory and discard a segment with a length proportional to $\alpha ·T$. This means that a contiguous portion of the trajectory is removed, simulating realistic missing data scenarios

4.3.2. Creating Incomplete Trajectories

Using the described masking strategy, we generate incomplete trajectories from the complete ones. Let ${\mathit{X}}_{\mathit{incomplete}}$ represent the resulting incomplete trajectory. We define the process mathematically as follows:

$${\mathit{X}}_{\mathit{incomplete}}=\mathit{Mask}(\mathit{X},\alpha )$$

(1)

where $\mathrm{Mask}(·)$ is a function that applies the masking strategy to the complete trajectory $\mathit{X}$ with a masking ratio of $\alpha$. This function is designed to take a complete trajectory of any length, as trajectories can vary in length, and it will generate an incomplete trajectory by randomly discarding a certain portion of the trajectory. The hyperparameter $\alpha$ is responsible for determining the proportion of the trajectory to be discarded. In  Figure 2, the complete portions of the trajectories are presented in the color blue and the incomplete trajectories are represented by the red color.

4.4. Padding and Binary Masking

The trajectories in the training dataset were padded to the maximum length across different trajectories from all four locations. This was a crucial step for efficient batch processing during training. Let $X_i$ represent the trajectory of the i-th example, and $T_i$ be its original length. The trajectories were padded to a length of $T_{\max }$, where $T_{\max }={max}_i{T}_i$.

Additionally, a binary mask was created for each trajectory to indicate the presence of observed points (1) and missing points (0) in the incomplete trajectories. The binary mask is denoted as $M_i$, where $M_{i,t}=1$ if the point at time step t is observed and $M_{i,t}=0$ if it is missing.

5. Methodology

This section explores the complex structural characteristics of the methodology, providing thorough comprehension of the design elements and organizational components that are the basis of our approach.

5.1. LSTM Encoder–GRU Decoder Architecture with Teacher Forcing

The LSTM-GRU (Gated Recurrent Unit)-based encoder–decoder (Figure 3) is a sequence-to-sequence model commonly used in various natural language processing and sequence generation tasks. As the name suggests, it consists of two main components, an encoder and a decoder. We picked LSTM for the the encoder part and GRU for the decoder after thorough experimentation.

5.1.1. LSTM Encoder

The LSTM encoder, a crucial component of sequence-to-sequence models, analyzes input sequences to capture their temporal dependencies. For an input sequence $X=\{x_1,x_2\dots x_T\}$ of length T, the encoder produces a hidden state sequence $H^{\mathrm{enc}}=\{h_1^{\mathrm{e}},h_2^{\mathrm{e}}\dots h_T^{\mathrm{e}}\}$.

The LSTM encoder updates its hidden states using the following equations:

$$\begin{array}{cc}\hfill i_t& =\sigma ({\mathit{W}}_{\mathit{ii}}{x}_t+{\mathit{b}}_{\mathit{ii}}+{\mathit{W}}_{\mathit{hi}}{h}_{t-1}^{\mathrm{e}}+{\mathit{b}}_{\mathit{hi}})\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill f_t& =\sigma ({\mathit{W}}_{\mathit{if}}{x}_t+{\mathit{b}}_{\mathit{if}}+{\mathit{W}}_{\mathit{hf}}{h}_{t-1}^{\mathrm{e}}+{\mathit{b}}_{\mathit{hf}})\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill g_t& =tanh({\mathit{W}}_{\mathit{ig}}{x}_t+{\mathit{b}}_{\mathit{ig}}+{\mathit{W}}_{\mathit{hg}}{h}_{t-1}^{\mathrm{e}}+{\mathit{b}}_{\mathit{hg}})\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill o_t& =\sigma ({\mathit{W}}_{\mathit{io}}{x}_t+{\mathit{b}}_{\mathit{io}}+{\mathit{W}}_{\mathit{ho}}{h}_{t-1}^{\mathrm{e}}+{\mathit{b}}_{\mathit{ho}})\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill c_t& =f_t\odot c_{t-1}+i_t\odot g_t\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill h_t^{\mathrm{e}}& =o_t\odot tanh\left(c_t\right)\hfill \end{array}$$

(7)

where $\sigma$ denotes the logistic sigmoid function; ⊙ represents element-wise multiplication; $x_t$ stands for the input at time step t; $h_t^{\mathrm{e}}$ represents the hidden state at time step t; $i_t$, $f_t$, $g_t$, and $o_t$ denote the input, forget, cell, and output gates at time step t, respectively; and W and b represent the weight and bias matrices for different gates. $c_t$ represents the cell state at time step t. The LSTM’s unusual architecture, comprising input, forget, cell, and output gates, allow it to successfully capture long-range relationships in sequential data, making it well suited for jobs requiring memory retention over longer periods. The LSTM encoder is renowned for its ability to alleviate the vanishing gradient problem commonly encountered in traditional RNNs. This is achieved through its sophisticated gating mechanisms, which regulate the flow of information across time steps. The input gate $i_t$ controls the influx of fresh information into the cell state $c_t$, while the forget gate $f_t$ regulates the confinement of former cell state information. The output gate $o_t$ governs the flow of information from the cell state to the hidden state, ensuring that relevant information is propagated forward while irrelevant information is suppressed.

5.1.2. GRU Decoder with Teacher Forcing

The GRU decoder with teacher forcing is responsible for generating an output sequence based on the hidden states obtained from the encoder. Given the hidden state sequence $H^{\mathrm{enc}}$ and an input sequence $Y=\{y_1,y_2\dots y_U\}$ of length U, the decoder computes an output sequence $Y^{\prime }=\{y_1^{\prime },y_2^{\prime }\dots y_U^{\prime }\}$.

The GRU decoder updates its hidden states and produces the output using the following equations, incorporating the concept of teacher forcing:

$$\begin{array}{c}\hfill h_t^{\mathrm{d}}=\mathrm{GRU}(y_t,h_{t-1}^{\mathrm{d}})\end{array}$$

(8)

$$\begin{array}{c}\hfill r_t=\sigma (W_{yr}{y}_t+U_{hr}{h}_{t-1}^{\mathrm{d}}+b_r)\end{array}$$

(9)

$$\begin{array}{c}\hfill z_t=\sigma (W_{yz}{y}_t+U_{hz}{h}_{t-1}^{\mathrm{d}}+b_z)\end{array}$$

(10)

$$\begin{array}{c}\hfill {\tilde{h}}_t=tanh(W_{y\tilde{h}}{y}_t+r_t\odot \left(U_{h\tilde{h}}{h}_{t-1}^{\mathrm{d}}\right)+b_{\tilde{h}})\end{array}$$

(11)

$$\begin{array}{c}\hfill h_t^{\mathrm{d}}=(1-z_t)\odot h_{t-1}^{\mathrm{d}}+z_t\odot {\tilde{h}}_t\end{array}$$

(12)

$$\begin{array}{c}\hfill y_t^{\prime }=\mathrm{Regressor}\left(h_t^{\mathrm{d}}\right)\end{array}$$

(13)

where GRU represents the operation of a Gated Recurrent Unit; $\sigma$ denotes the logistic sigmoid function; ⊙ represents element-wise multiplication, $y_t$ is the input at time step t; $h_t^{\mathrm{d}}$ represents the hidden state at time step t; $r_t$ and $z_t$ denote the reset and update gates, respectively; ${\tilde{h}}_t$ is the candidate hidden state; W and U are weight matrices; and b is a bias vector. Regressor is a function that maps the hidden state to the output space.

The GRU decoder with teacher forcing employs the use of teacher forcing, a technique where the decoder receives the true values of the previous time steps as inputs during training instead of its predictions. This technique facilitates learning by providing more accurate guidance to the model during training. The probability of using teacher forcing at each time step can be controlled by a parameter $\alpha \in [0,1]$, where $\alpha =1$ indicates full teacher forcing (always using true values), and $\alpha =0$ indicates no teacher forcing (always using model predictions). The probability of using teacher forcing at time step t can be expressed as follows:

$$P(\mathrm{teacher} \mathrm{forcing} \mathrm{at} t)=\alpha$$

(14)

Thus, the input $y_t$ to the decoder is determined by whether teacher forcing is applied or not:

$$y_t=\left\{\begin{array}{cc}{y}_{t-1}\hfill & \mathrm{with} \mathrm{probability} \alpha \hfill \\ y_{t-1}^{\prime }\hfill & \mathrm{with} \mathrm{probability} 1-\alpha \hfill \end{array}\right.$$

(15)

where $y_{t-1}^{\prime }$ represents the predicted output of the decoder at time step $t-1$.

5.1.3. Justification for Using GRU

• Sequential Modeling: Because each point in a trajectory depends on the ones before it, trajectory data are inherently sequential. GRUs excel at sequential modeling as they maintain a hidden state that evolves, allowing them to capture dependencies in sequential data.
• Memory Cells with Gates: GRUs are equipped with memory cells and gating mechanisms, enabling them to reset and update their internal state dynamically. The reset gate allows the model to choose which historical data to disregard, while the update gate determines which fresh data to incorporate. This flexibility mitigates the vanishing gradient issue and facilitates the capture of long-term interdependencies.
• Efficient Training: GRUs are designed for computational efficiency, making them suitable for handling lengthy sequences often encountered in trajectory data. The efficiency of GRUs may contribute to quicker convergence during training.
• Parameter Efficiency: Compared to LSTM networks, GRUs have fewer parameters, enhancing their parameter efficiency. This characteristic is particularly beneficial when working with trajectory data that may have limited samples. The reduced number of parameters helps prevent overfitting and may lead to better generalization for new trajectories.

5.2. Training

Training the trajectory imputation model involves several key components that contribute to its overall performance. In this subsection, we elaborate on some crucial parts of the training phase, like the optimization strategy, gradient clipping, and learning rate scheduler. But firstly, let us talk about the core training strategy. Incomplete trajectories are termed as features and corresponding missing points are termed as labels. This is how the training was performed, so during the testing phase, given some incomplete trajectories, we obtained the missing points of corresponding trajectories as outputs. The dataset was split into training, validation, and testing sets using a ratio of 70% for training, 15% for validation, and 15% for testing

• Optimization
  To update the model parameters, an Adam optimizer was utilized. The Adam optimization algorithm computed the adaptive learning rates for each parameter based on their first-order moment estimate (mean) and the second-order moment estimate (uncentered variance). The update rule for the parameters $\theta$ is given by

  $${\theta }_{t+1}={\theta }_t-{\displaystyle \frac{\mathit{lr}}{\sqrt{{\widehat{v}}_t}+ϵ}}{\widehat{m}}_t$$

  (16)

  where ${\widehat{m}}_t$ and ${\widehat{v}}_t$ are the biased first- and second-moment estimates, $ϵ$ is a small constant to prevent division by zero, and lr is the learning rate.
• Gradient Clipping
  Gradient clipping was employed to prevent exploding gradients during backpropagation. This technique involves scaling the gradients if their norm exceeds a predefined threshold (max_grad_norm). The scaled gradient $g^{\prime }$ was computed as follows:

  $$g^{\prime }=\left\{\begin{array}{cc}{\displaystyle \frac{\max \_\mathrm{grad}\_\mathrm{norm}}{\parallel g\parallel }}·g\hfill & \mathrm{if} \parallel g\parallel >\max \_\mathrm{grad}\_\mathrm{norm}\hfill \\ g\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  (17)
• Learning Rate Scheduler
  The learning rate was scheduled using the ReduceLROnPlateau (a PyTorch library function which decreases learning rate when a metric has ceased improving) scheduler. This scheduler adjusted the learning rate if the validation losses plateaued, enabling the model to fine-tune its parameters more effectively. The learning rate update rule is given by:

  $${\mathit{lr}}_{\mathit{new}}=\mathit{factor}\times {\mathit{lr}}_{\mathit{old}}$$

  (18)

  where ${\mathit{lr}}_{\mathit{old}}$ is the previous learning rate, and factor is a user-defined factor.

5.3. Hyperparameters

In this subsection, we define and elaborate on the hyperparameters used in the trajectory imputation model. Hyperparameters are essential for influencing how the model behaves and performs during training. The hyperparameters are divided into two categories: model parameters, which define the model’s architecture, and training parameters, which affect how the model learns.

5.3.1. Model Parameters

• Dimensionality of input trajectories: This parameter represents the number of features or dimensions in the input data.
• Number of hidden units: The hidden size determines the ability of the model to grasp and represent temporal dependencies.
• Dimensionality of output trajectories: Similar to input size, this parameter defines the number of features in the output predictions.
• Number of layers: This parameter controls the depth of the recurrent neural network, influencing its ability to model complex patterns.

5.3.2. Training Parameters

• Gradient clipping threshold: For gradients during backpropagation, this value establishes the highest permitted norm. It improves training stability and helps stop gradients from blowing up.
• Maximum gradient norm for scaling gradients: Gradient clipping is applied to limit the norm of gradients. If the computed norm exceeds the limit, the gradients are scaled to meet this threshold.

Together, these hyperparameters form the trajectory imputation model’s design and training behavior. To achieve best performance and robustness during training, these variables must be fine-tuned.

6. Experimental Results and Discussion

The “Results and Discussion” section outlines the model’s performance, highlighting significant metrics such as L1 loss and ADE. Graphical representations, including trend plots and a table of training results, provide a thorough picture.

Table 4. Summary of experimental results (minimum values).

Loss Type	Minimum Value
L1 training loss	0.2264
L1 validation loss	0.2231
Training quantile loss	0.1686
Validation quantile loss	0.1636
MSE training	1.4728
MSE validation	1.5025
ADE test loss	4.14

showcases the minimum values for different metrics. Figure 4 clearly illustrates the model’s ability to predict and finish trajectories.

6.1. L1 Loss

L1 loss, sometimes referred to as the mean absolute error (MAE) or L1 norm loss, is a commonly employed loss function in the fields of machine learning and optimization, specifically in regression problems. This metric measures the absolute disparities between the anticipated values ${\widehat{y}}_i$ and the actual target values $y_i$ for a total of n data points. L1 loss is calculated by taking the absolute difference between the predicted value (${\widehat{y}}_i$) and the actual value ($y_i$) for each data point, and then averaging these differences across the total number of data points (n). The inclusion of the absolute value in this formula ensures that both overestimates and underestimates have an equal impact on the loss. L1 loss exhibits lower sensitivity to outliers in comparison to L2 loss (mean squared error), rendering it appropriate for situations where the influence of outliers on the total loss should be constrained. Figure 5 represents L1 training loss and Figure 6 represents L1 validation loss.

The L1 loss formula is given by

$$\mathit{L}\mathit{1}\_\mathit{Loss}={\displaystyle \frac{1}{n}}\sum _{i=1}^n|{\widehat{y}}_i-y_i|$$

(19)

6.2. MSE Loss

Mean Squared Error (MSE) is a common metric used in regression analysis to measure the average squared difference between predicted values and actual values. MSE loss can be represented as follows:

$$\mathit{MSE}={\displaystyle \frac{1}{n}}\sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2$$

(20)

where $y_i$ represents the actual value, ${\widehat{y}}_i$ represents the predicted value, and n is the number of data points. Figure 7 represents the MSE training loss and Figure 8 represents the MSE validation loss.

6.3. ADE Loss

Average Displacement Error (ADE) loss is a widely utilized statistic in trajectory prediction applications. This metric calculates the average Euclidean distance between anticipated points and their associated ground truth points throughout a series of frames. ADE loss is computed using the following formula:

$$ADE={\displaystyle \frac{1}{N}}\sum _{i=1}^N\parallel {\widehat{y}}_i-y_i\parallel ,$$

(21)

where N is the total number of points in the sequence, ${\widehat{y}}_i$ represents the predicted position, and $y_i$ is the ground truth position. Figure 9 shows the ADE loss over time.

6.4. Quantile Loss

Quantile loss is a loss function used in quantile regression to measure the difference between predicted quantiles and actual quantiles of the target distribution. It is particularly useful when modeling non-normally distributed data or when different quantiles are of interest. Quantile loss is defined as

$$\mathit{Quantile}\mathit{loss}=\sum _{i=1}^n(\tau ·max(y_i-{\widehat{y}}_i,0)+(1-\tau )·max({\widehat{y}}_i-y_i,0))$$

(22)

where $y_i$ represents the actual value, ${\widehat{y}}_i$ represents the predicted value, $\tau$ is the desired quantile level, and n is the number of data points. Figure 10 represents the quantile training loss, and Figure 11 represents the quantile validation loss.

Table 4 summarizes the minimum losses obtained from training, validation, and testing across various metrics. These minimum values provide insight into the performance achieved by the model at different evaluation stages.

6.5. Graphical Representations of Predicted Trajectory and Ground Truth Trajectory

In this subsection, we present a visual contrast between the predicted trajectory and the baseline trajectory. Figure 4 illustrates the trajectories plotted on a graph, with velocity values indicated. The x-axis represents the horizontal position, while the y-axis represents the vertical position. The blue line denotes the ground truth trajectory, while the red line represents the generated trajectory. Additionally, velocity values are annotated along the trajectories, providing insights into the speed variations throughout the movement. Arrows indicating the direction of movement are also included to enhance the visualization. This comparison allows for a qualitative assessment of the model’s performance in predicting trajectories and provides valuable insights into the dynamics of the system under study.

6.6. Comparison with Other Models

In this subsection, we conduct a comprehensive comparison of four models: a two-block RNN, BRITS, the GRU encoder–decoder, and our LSTM-GRU encoder–decoder model. The evaluation was primarily based on their respective L1 loss values, providing a quantitative measure of their performance.

In

Table 5. Models and their corresponding L1 loss.

Model	L1 Loss
Two-block RNN [19]	0.363
BRITS [11]	0.312
GRU based encoder–decoder	0.306
LSTM-GRU encoder–decoder (ours)	0.2264

, the L1 loss values for each model are presented. The two-block RNN model yielded an L1 loss of 0.363, followed by BRITS with a loss of 0.312. The GRU-based encoder–decoder model exhibited an L1 loss of 0.306. Notably, our proposed LSTM-GRU encoder–decoder model outperformed the others with an impressively low L1 loss of 0.2264.

7. Conclusions

In this research, we designed a novel deep learning model for imputing missing trajectory data for autonomous cars. Trajectory data are vital for the planning, navigation, and safety of autonomous cars, but they can be damaged or lost due to many circumstances, such as sensor failures, occlusions, noise, etc. Existing methods for trajectory imputation are either too straightforward, such as linear interpolation, or too complicated, such as generative adversarial networks.

We have proposed a hybrid model that incorporates LSTM and GRU to capture the temporal, spatial, and contextual characteristics of the trajectory data. Our model can handle different sorts of missing patterns and impute trajectories with excellent accuracy and efficiency. Long–short-term memory (LSTM) networks are renowned for their ability to retain long-term dependencies in sequential data. By employing an LSTM encoder, our model can effectively capture the temporal dynamics of trajectories. This means it can discern patterns and trends over extended periods, crucial for understanding how trajectories evolve over time.

On the decoding side, using a Gated Recurrent Unit (GRU) architecture brings its own advantages. GRUs are computationally efficient and can learn to adaptively update information in the decoding process. This helps in generating accurate predictions or reconstructions of trajectories while maintaining a balance between complexity and performance.

By fusing LSTM and GRU components, our model benefits from the strengths of both architectures. LSTM excels at retaining long-range dependencies, which is crucial for understanding complex trajectory patterns. On the other hand, GRU offers computational efficiency and simpler gating mechanisms, making it well suited for decoding tasks. This hybrid approach leverages the best of both worlds to enhance the overall performance and flexibility of the model.

The insights gained from trajectory analysis could be crucial for optimizing drone delivery routes in urban environments, where efficiency and safety are paramount. By leveraging trajectory analysis, logistics companies can minimize delivery times and maximize resource utilization, ultimately improving customer satisfaction and operational efficiency.

In future research endeavors, an intriguing avenue for enhancement involves experimenting with transformer models in the context of trajectory imputation for autonomous cars. Another limitation, which we have already mentioned in our study, is that it solely relies on the drone dataset, which is comparatively superior to other datasets in many arenas. But many studies have been carried out with the GPS and BLE datasets, which are no less important. Our model should be utilized on these datasets as a future path of exploration and experimentation. Also, we can experiment by adding different types of embeddings, like positional embedding or graph embedding with an attention mechanism, to our proposed architecture, and anticipate better outcomes in these cases.