Intra-Frame Graph Structure and Inter-Frame Bipartite Graph Matching with ReID-Based Occlusion Resilience for Point Cloud Multi-Object Tracking

Abstract

Three-dimensional multi-object tracking (MOT) using lidar point cloud data is crucial for applications in autonomous driving, smart cities, and robotic navigation. It involves identifying objects in point cloud sequence data and consistently assigning unique identities to them throughout the sequence. Occlusions can lead to missed detections, resulting in incorrect data associations and ID switches. To address these challenges, we propose a novel point cloud multi-object tracker called GBRTracker. Our method integrates an intra-frame graph structure into the backbone to extract and aggregate spatial neighborhood node features, significantly reducing detection misses. We construct an inter-frame bipartite graph for data association and design a sophisticated cost matrix based on the center, box size, velocity, and heading angle. Using a minimum-cost flow algorithm to achieve globally optimal matching, thereby reducing ID switches. For unmatched detections, we design a motion-based re-identification (ReID) feature embedding module, which uses velocity and the heading angle to calculate similarity and association probability, reconnecting them with their corresponding trajectory IDs or initializing new tracks. Our method maintains high accuracy and reliability, significantly reducing ID switches and trajectory fragmentation, even in challenging scenarios. We validate the effectiveness of GBRTracker through comparative and ablation experiments on the NuScenes and Waymo Open Datasets, demonstrating its superiority over state-of-the-art methods.

1. Introduction

Three-dimensional multiple object tracking (MOT) aims to detect objects of interest in point cloud data and assign each object a unique identity, forming a track of objects throughout the sequence. This technology is essential in many applications, including autonomous driving, smart cities, and robotic navigation and interaction [1,2]. The tracking-by-detection paradigm splits the problem into two main tasks: detecting objects in each frame (localization) and performing data association (linking detections over time to form object trajectories). Object localization is typically achieved using powerful detectors, while data association remains a complex problem. The primary challenge in data association arises from occlusions, which can lead to missed detections and, subsequently, ID switches. Therefore, improving detection accuracy and data association correctness under occlusion is crucial for effective 3D MOT.

As can be seen in Figure 1, the tracking-by-detection paradigm uses a detector, which is a stage named Before Data Association, to extract the object’s spatial and contextual features. Most methods design sparse convolutions for voxel-based [3] approaches and accurate down-sampling algorithms [4] to extract object-oriented features for point-based methods. Point-voxel-based detectors [5] leverage both techniques to improve detection performance. Recently, graph-based detectors [6,7,8] and trackers [9,10], which can be used as plug-and-play modules for these three types of detectors, have emerged. By building a K-Nearest Neighborhood (KNN) graph structure, these detectors can extract neighborhood features to aggregate for the center object, enhancing spatial feature embedding. Our tracker pipeline follows a similar approach, incorporating a graph structure to improve the accuracy of the detector.

Data Association in 3D MOT is crucial for accurately linking detected objects across frames to maintain consistent object identities. A common approach to this problem is Bipartite Graph Matching. In this paradigm, detected objects in the current frame are matched to tracklets from the previous frame, which can be visualized as a bipartite graph, where nodes represent detections and tracklets, and edges represent potential matches with association costs. Traditionally, methods like the Hungarian algorithm [11,12,13,14] and greedy approaches [15,16] have been employed to solve this matching problem. However, these methods often do not account for the weights of edges between nodes across frames, which can lead to suboptimal, local solutions. To address this limitation, we construct a comprehensive cost matrix and formulate the problem as a minimum-cost flow optimization to find a globally optimal solution for data association, thereby improving the accuracy and robustness of 3D MOT under conditions of occlusion.

After Data Association is the stage of track management involving updating and maintaining trajectories. Various methods exist for track updates and trajectory management, such as trajectory similarity [17], cross-frame object intersection-over-union (IoU) association [18], outlier removal [19], and trajectory hypothesis [20]. ReID [21,22] is commonly used in video-based multi-object tracking for pedestrian detection. In this context, the detector extracts features and encodes them using an MLP to obtain ReID features for re-identifying objects in subsequent frames. However, in point-cloud-based multi-object tracking, leveraging image appearance features is not feasible due to the unordered nature of points and the lack of color information. Therefore, we utilize velocity- and angle-based ReID features to re-match unmatched tracks with their corresponding trajectory IDs in the event of missed detections or occlusions.

We propose a point cloud multi-object tracking-by-detection tracker: GBPTracker. For detection, we design an intra-frame graph geometrical structure to extract neighborhood features and aggregate them for each object. Unlike many methods that solely rely on object location or motion information and often view each object as an independent entity, neglecting the geometrical relationships between them, our detector enhances spatial and contextual information, thereby reducing missed detections and indirectly decreasing errors in data association. For data association, we simplify the adjacency matrix formed by tracking and detection associations into a bipartite graph matching problem. We view bipartite graph matching as an optimization problem and define the objective function to find the optimal matching with the minimum global cost. We design a sophisticated cost matrix using object attributes such as the center location, bounding box size, velocity, and heading angle. A minimum-cost flow optimization algorithm is then applied to achieve globally optimal matching at the minimum cost. After data association, we use dynamic motion-based ReID embeddings, incorporating velocity and heading angle, to calculate the similarity between current detections and trajectory detections. This helps reconnect unmatched detections to their proper ID trajectories, reducing fragmented trajectories and correctly initializing new trajectories to obtain accurate tracking results.

Our tracker effectively alleviates occlusion-induced missed detections and improves object association accuracy. Furthermore, unmatched detections can reconnect to the proper trajectory ID, helping to decrease ID switches and avoiding the wrong initialization of new tracks for more effective tracking.

In summary, our work makes the following contributions:

• We propose an intra-frame graph structure that leverages adaptive graph convolution to aggregate edge features into central nodes, thereby enhancing the robust representation of each node.
• We view data association as inter-frame bipartite graph matching and define the objective function to minimize the global optimal matching cost. By designing a sophisticated cost matrix and applying a minimum-cost flow optimization algorithm, we achieve globally optimal matching for accurate data association in complex scenarios, thereby reducing ID switches.
• For unmatched objects, we propose a motion-based ReID layer that uses similarity scores and association probabilities to accurately re-associate objects with their previous fragmented trajectory IDs, thereby reducing ID switches and avoiding the wrong initialization of new tracks.

2. Related Works

2.1. Three-Dimensional Multi-Object Tracking

Model-based tracking methods such as PMRA-PMBM [23] and GNN-PMB [24] utilize probabilistic graphical models, making them suitable for handling high-dimensional data with complex relationships and dependencies. These methods offer flexibility and global optimality but suffer from high computational complexity, making them less suitable for real-time applications.

Filter-based tracking methods such as Simpletrack [25] and AB3DMOT [26] rely on predefined motion models to describe the system’s dynamics and measurement processes. However, these methods struggle when multiple objects exhibit nonlinear and diverse motions simultaneously. Recently, graph-based methods such as PolarMOT [9], LOGR [27], and LEGO [28] have built graph structures with neural message passing for online object detection and tracking. BP-Tracker [29] employs a factor graph and a belief propagation algorithm to compute the marginal association probability, enhancing tracking performance.

Our work also leverages graph-based approaches utilizing neighborhood information aggregation and data association to improve tracking performance.

2.2. Data Association in Multi-Object Tracking

The data association performed by the tracking-by-detection paradigm views each detection as a node in a graph, with edges linking nodes over the temporal domain to form trajectories. For instance, PolarMOT [9] constructs node graphs, inter-frame graphs, and intra-frame graphs directly on targets without using an object detector. TrackMPNN [30] utilizes dynamic undirected graphs to tackle data association challenges across multiple timesteps. AGTSSD [8] uses an adaptive graph transformer within the backbone for graphical object detection. STBGtacker leverages a 4D graph spatio-temporal backbone [31] for single-object tracking, proving that objects are not independent and that their relationships need to be explored. Similarly, LearnableTrack [27] employs a graph-based structure to manage detection and track data association simultaneously using graph neural message passing.

The determination of connected nodes can then be addressed using the Hungarian algorithm [11,13,14] or greedy assignment [16,32]. Compared to minimum-cost flow and maximum-cost flow methods, these approaches are more prone to local optimization, potentially missing some matchings. Occlusions or undetected objects can lead to fragmented trajectories over several frames. Various methods employ trajectory update mechanisms, such as trajectory interpolation [33] or the elimination of outliers in tracklets [19], using proposals to directly generate and associate tracklets [34].

ReID [21,35] is commonly employed in pedestrian tracking within crowded scenes to identify individuals. Typically, detection and ReID tasks are decoupled. Our method uses intra-frame graphs, a sophisticated cost matrix with a minimum-cost flow algorithm for bipartite graph matching for data association, and motion-enhanced ReID to properly reconnect previously unmatched trajectories or initialize new tracks. This approach helps reduce ID switches and improve tracking performance by avoiding the direct assignment of new track IDs.

3. Methods

Within this section, as shown in Figure 2, we propose the 3D point cloud multi-object tracking GBRTracker, which comprises several modules: object detection feature embedding, bipartite graph matching for data association, and trajectory management and track update.

3.1. Intra-Frame Graph Structure

To address the challenge of sparse convolution in voxel representation feature maps, which may lack spatial information in voxel-based detectors, we modified our previous work [8] by designing a voxel graph layer. As shown in Figure 3, this enhancement integrates an intra-frame graph structure within the backbone, effectively improving geometric feature embedding.

3.1.1. Voxel Feature Extraction

The input point cloud $\mathbf{P}$ is first voxelized to create a structured grid of voxels. Each voxel aggregates the points within its spatial boundaries, forming initial voxel features $\mathbf{V}$. These voxel features are then processed through a Sparse Convolutional Network to obtain the final voxel feature representation $\mathbf{F}$:

$$\mathbf{F}=\mathrm{SparseConvNet}\left(\mathbf{V}\right),$$

(1)

where $\mathbf{F}$ represents the feature tensor for each voxel. This approach leverages the efficiency of sparse convolutions to handle the high-dimensional and sparse nature of point cloud data, ensuring that important spatial information is preserved and effectively utilized in the detection process. We define the voxel feature $\mathbf{F}$ as follows:

$$\mathbf{F}\in {\mathbb{R}}^{N\times C\times H\times W}$$

(2)

where N is the batch size, C is the number of feature channels, and H and W are the height and width, respectively.

3.1.2. Graph Construction and Adaptive Graph Convolution

For each node feature, we build a KNN graph for each feature channel of voxels in the 3D Euclidean feature space instead of the position space. K represents the number of nearest neighbors considered for each point, forming the set $N\left(i\right)$. Then, we use adaptive graphconv [8] for each edge between similar features pairwise. Finally, the aggregated feature node updates each center feature node using a max-pooling operation as follows:

$$F_i^{\prime }=\underset{j\in N\left(i\right)}{max}\sigma \left(\mathbf{w}·\left[F_i,F_i-F_j\right]\right)$$

(3)

where $\mathbf{w}$ is a shared learnable parameter, $\sigma$ represents a nonlinear activation function such as ReLU, $N\left(i\right)$ denotes the set of neighbors for point i, and $F_i$ and $F_j$ are the feature embeddings of the center and its neighbors, respectively.

Our adaptive graph voxel feature layer effectively captures and aggregates spatial neighborhood features, enhancing the robustness of the 3D point cloud features extracted by the backbone network. This improvement significantly reduces missed detections and enhances overall tracking accuracy, particularly in challenging environments.

3.2. Bipartite Graph Matching for Data Association

In 3D multi-object tracking (MOT), data association is vital for linking detected objects to their corresponding track IDs. Common approaches in 3D MOT involve optimizing an overall association cost matrix. This optimization often involves calculating the overlap or distance between the predicted and detected 3D bounding boxes using algorithms like the global nearest neighbor or the Hungarian algorithm. To improve the effectiveness of tracking-by-detection data association, we define the comprehensive and robust cost of data association in MOT as shown in Figure 4.

3.2.1. Objective Function

We treat the matching of detected objects in the current frame with tracklets from the previous frame as a bipartite graph matching problem for data association. Each object is defined as a node. Let $\mathcal{O}=\left\{{\mathbf{o}}_i\right\}$ be a set of object detections, where ${\mathbf{o}}_i=({\mathbf{p}}_i,{\mathbf{t}}_i)$, with ${\mathbf{p}}_i=(x,y,z)$ representing the position and features and ${\mathbf{t}}_i$ representing the timestamp. A trajectory is defined as a list of object detections ${\mathcal{T}}_k=\{{\mathbf{o}}_{k_1},{\mathbf{o}}_{k_2},\cdots ,{\mathbf{o}}_{k_N}\}$, where k is the index of the trajectory.

The objective function for bipartite graph matching aims to find the minimum cost, defined as

$$\underset{M_{ij}}{min}\sum _{i=1}^{N_t}\sum _{j=1}^{N_r}(1-C{M}_{i,j})M_{ij}$$

(4)

subject to

$$\begin{array}{c}\hfill \sum _{i=1}^{N_t}{M}_{ij}=1,\end{array}$$

(5)

$$\begin{array}{c}\hfill \sum _{j=1}^{N_r}{M}_{ij}=1,\end{array}$$

(6)

where $M_{ij}=1$ if the i-th detection is assigned to the j-th track in the previous frame; otherwise, $M_{ij}=0$.

Here, $N_t$ is the number of detections in the current frame, $N_r$ is the number of tracks in the previous frame, and $C{M}_{i,j}$ is the total cost matrix representing the cost of matching detection i in the current frame with track j in the previous frame. The values in $C{M}_{i,j}$ are normalized to fall between 0 and 1 to ensure that the objective function ranges between 0 and 1 as well.

3.2.2. Association Cost Matrix

Bounding Box EIoU Cost To address challenges in tracking, especially during occlusions and varying object sizes that result in negligible IoUs, we utilize an enhanced intersection-over-union (EIoU) [36] metric that considers both positional and dimensional discrepancies between detected objects and trajectory predictions. The following describes the difference between IoU and EIoU:

$$Io{U}_{3d}={\displaystyle \frac{B^i\cap B^j}{B^i\cup B^j}},$$

(7)

$$EIo{U}_{3d}=Io{U}_{3d}-{\displaystyle \frac{{\rho }^2(b^i,b^j)}{c^2}}-{\displaystyle \frac{{\rho }^2(l^i,l^j)}{{\left(c_l\right)}^2}}-{\displaystyle \frac{{\rho }^2(w^i,w^j)}{{\left(c_w\right)}^2}}-{\displaystyle \frac{{\rho }^2(h^i,h^j)}{{\left(c_h\right)}^2}},$$

(8)

where $B^i$ and $B^j$ denote the 3D bounding boxes of detection and trajectory prediction, respectively. C is the smallest enclosing box, $\rho (b^i,b^j)$ is the center distance between $B^i$ and $B^j$, $\rho (l^i,l^j)=|l^i-l^j|$, $\rho (w^i,w^j)=|w^i-w^j|$, $\rho (h^i,h^j)=|h^i-h^j|$, and $c,c_l,c_w,c_h$ represent the diagonal length, length, width, and height of the smallest enclosing box, respectively. EIoU computes both positional and geometric correlations, including center distance and dimension differences.

Center Point Distance Cost The distance between the center points of objects in two different frames is computed based on the Euclidean distance. The Euclidean distance $d(i,j)$ is computed as follows:

$$d(i,j)=\sqrt{{(x_i^1-x_j^2)}^2+{(y_i^1-y_j^2)}^2},$$

(9)

where $(x_i^1,y_i^1)$ are the coordinates of the center point of object i in the trajectory frame. $(x_j^2,y_j^2)$ are the coordinates of the center point of object j in the detection frame. $d(i,j)$ is the Euclidean distance between object i and object j.

Motion Cost We design the motion cost calculation, which comprises angle prediction, velocity prediction, and a combined motion cost metric:

• Angle Prediction Angle prediction is derived from the predicted values of $\sin \left(\theta \right)$ and $\cos \left(\theta \right)$. The angle $\theta$ is then determined using the formula

  $$\theta =\mathrm{atan}2\left(\sin \right(\theta ),\cos (\theta \left)\right),$$

  (10)
  The $\mathrm{atan}2$ function computes the angle $\theta$ from sin and cos values, accommodating positive or negative inputs for both functions and ensuring accurate angle determination.
• Velocity Prediction The velocity prediction provides the two-dimensional speed components, $v_x$ and $v_y$, of the detected objects.

  $$\mathbf{V}=(v_x,v_y),$$

  (11)

  where $v_x$ and $v_y$ represent the components of velocity in the eastward and northward directions, respectively. These contribute to the comprehensive motion state of the detected objects, encompassing both linear and angular velocities.
• Velocity and Angle-based Motion Cost The $(i,j)$-th entry in the motion cost matrix $M_{i,j}$ is formulated as follows:

  $$M_{i,j}=w_{\theta }·\left({\displaystyle \frac{1-\cos ({\theta }_i-{\theta }_j)}{2}}\right)+w_v·{\parallel {\mathbf{v}}_i-{\mathbf{v}}_j\parallel }^2,$$

  (12)

  where ${\theta }_i$ and ${\theta }_j$ are the angles of the predicted object state in the current and previous frames, respectively. ${\mathbf{v}}_i$ and ${\mathbf{v}}_j$ represent the velocities of the predicted object state in the current and previous frames, respectively. $w_{\theta }$ denotes the weight for heading angle similarity, and $w_v$ denotes the weight for velocity similarity.

  −

  Angle Similarity The cosine similarity $\cos ({\theta }_i-{\theta }_j)$ is utilized to measure the similarity between two angles. The value ranges from −1 to 1, with 1 indicating complete similarity and −1 indicating complete dissimilarity. To ensure that the similarity calculation results fall within [0, 1], we use $1-\cos ({\theta }_i-{\theta }_j)$. Further, dividing the entire expression by 2 normalizes the cost range to a more intuitive scale of 0 to 1.

  −

  Velocity Difference Calculating the square of the velocity differences simplifies the process and serves to amplify larger discrepancies while being less sensitive to smaller variations.

Total Cost Matrix The graph backbone aims to extract features and obtain predicted bounding box information. We have designed a comprehensive cost matrix based on the EIOU, center-point positional information, heading angle, and velocity-based motion cost. The total cost matrix $C{M}_{i,j}$ is composed of the center-point cost, motion cost, and bounding box EIOU cost:

$$C{M}_{ij}=\alpha ·{\mathrm{EIoU}}_{3d}+\beta ·d(i,j)+\gamma ·M_{ij},$$

(13)

where $\alpha$, $\beta$, and $\gamma$ are the corresponding weights for the EIoU cost, center-point distance cost, and motion cost, respectively.

3.3. Trajectory Management and Track Update

As shown in Figure 5, in frame t, we define $\mathcal{D}=\{D_1,D_2,\dots ,D_{N_D}\}$ as the set of multi-object detections in the current frame and $\mathcal{T}=\{T_1,T_2,\dots ,T_{N_T}\}$ as the set of trajectories obtained from past frames. $N_D$ and $N_T$ denote the number of objects detected in frame t and the number of candidate tracklets. The detection is denoted by $D_i=(C_i,{\mathbf{B}}_i,M_i)$, where $C_i$ is the center coordinates of the detection obtained from the feature extraction network. ${\mathbf{B}}_i=(x_i,y_i,z_i)$ represents the size of the bounding box. $M_i$ includes the rotation around the y-axis and the velocities along the x- and y-coordinates. Each tracklet contains a sequence of detection objects sharing the same tracklet ID. For detections that are successfully associated, they are added to the set of tracks, represented by $T_{id}^{t-1}\cup D_{\left(id\right)}^{t-1}\to T_{id}^t$. Since the detection ID does not always correspond to the associated tracklet ID, we use parentheses (id) to indicate the detection ID that aligns with the $id$-th tracklet. After global short-term bipartite graph matching data association, we address occluded objects resulting in missing matches by reconnecting them to the appropriate trajectories using an MLP-based ReID module that encodes motion information (velocity and heading angle).

3.3.1. Association Probability Calculation

The detection output features $z_i^d$, which include motion information (velocity and heading angle), are passed through an MLP to generate ReID feature vectors $r_i^d$. Similarly, the tracklets’ last-frame feature vector $z^T$ is used to generate the track ReID feature vector $r^D$.

Compute Similarity Scores Compute the cosine similarity between the ReID feature vectors of the tracklet’s last-frame and the current frame’s detections:

$$s(r_i^d,r^D)={\displaystyle \frac{{\left(r_i^d\right)}^{\top }{r}^D}{\parallel r_i^d\parallel \parallel r^D\parallel }}$$

(14)

where $s(r_i^d,r^D)$ is the similarity score based on motion-information-encoded ReID features.

Compute Association Probability We use a softmax function to compute the association probability between tracks and detections based on the combined similarity scores:

$$p\left(d_i\right|T)={\displaystyle \frac{\exp \left(s(r_i^d,r^D)\right)}{{\sum }_j\exp \left(s(r_j^d,r^D)\right)}}$$

(15)

3.3.2. Track Creation and Deletion Strategy

As shown in Algorithm 1, to manage the initialization and termination of tracks, we propose a simplified track management strategy:

• Refinement of Scores: For the matched pairs and new tracks, we refine the confidence scores using the embeddings obtained from the architecture head module. This refinement step, which follows a methodology similar to CenterPoint’s [15] second stage, ensures that the final tracking results are accurate and reliable.
• Handling Unmatched Detections: Unmatched detections are reconnected with the corresponding trajectory IDs using ReID features by Equations (14) and (15). If these unmatched detections do not get matched within the next three frames, they are considered noise and removed.
• New Track Creation: In each frame t, if a detection does not match any existing tracklet or unmatched detection and has a high confidence score, a new track is initialized. Each new track is assigned a unique track ID, and the corresponding detection is added to this new tracklet. This step ensures that newly appearing objects are properly tracked from the moment they are first detected.
• Track Deletion: A tracklet is deleted if it has no matching detection for three consecutive frames. This parameter ensures that tracks are not immediately discarded when an object is missed for a few frames, allowing for temporary occlusions or missed detections without losing the track.

Algorithm 1 Trajectory management and track update

Require: Detection set $\mathcal{D}$, trajectory set $\mathcal{T}$, maximum age $max\text{\_}age$ Ensure: Updated trajectory set $\mathcal{T}$   Input:     Detection set $\mathcal{D}=\{D_1,D_2,\dots ,D_{N_D}\}$, where each detection $D_i=M_i$       $M_i$: Motion information (velocity and angle)     Trajectory set $\mathcal{T}=\{T_1,T_2,\dots ,T_{N_T}\}$, where each tracklet $T_j$ contains a sequence of detections with motion information     Maximum age $max\text{\_}age$   Output:     Updated trajectory set $\mathcal{T}$   Hyperparameters:     Similarity score threshold ${\lambda }_s$     Association probability threshold ${\lambda }_p$   Step 1: Refinement of Scores   for each matched pair of detection $D_i$ and tracklet $T_j$ do    Refine confidence scores using embeddings from the architecture head module.   end for   Step 2: Handle Unmatched Detections   for each unmatched detection $D_u\in \mathcal{D}$ do    Compute similarity score $s(D_u,T_j)$ using Equation (14)    Compute association probability $p\left(d_u\right|T_j)$ using Equation (15)    if $s(D_u,T_j)\ge {\lambda }_s$$p\left(d_u\right|T_j)\ge {\lambda }_p$ then      Reconnect $D_u$ to corresponding trajectory ID      Update state of $T_j$ with $D_u$    else      Store $D_u$ as unmatched for next frame    end if   end for   Step 3: New Track Creation   for each detection $D_u\in \mathcal{D}$ do    if $D_u$ does not match any existing tracklet or unmatched detection and has a high confidence score then      Create new tracklet $T_{new}$      Assign new track ID to $D_u$      Add $D_u$ to $T_{new}$    end if   end for   Step 4: Track Deletion   for each tracklet $T_k\in \mathcal{T}$ do    if tracklet $T_k$ has no matching detection for a dynamic number of frames based on confidence and disappearance time then      Delete tracklet $T_k$    end if end for

4. Experiments

4.1. Datasets and Evaluation Metrics

4.1.1. Datasets

In this study, we chose to evaluate the proposed method on the currently popular large-scale autonomous driving public datasets NuScenes and Waymo. Both datasets provide LiDAR point clouds and 3D bounding box labels. NuScenes [37] is a large dataset that contains 1000 driving sequences, and each sequence spans 20 s. LiDAR data in NuScenes are provided at 20 Hz, but 3D labels are only given at 2 Hz. We used the validation set that includes 150 scenes, 6019 frames, and 140 k instances of object annotations. Our algorithm tracks seven categories specified by NuScenes. The Waymo Open Dataset comprises 1150 sequences, with 798 training, 202 validation, and 150 testing sequences, each containing 20 s of continuous driving data within the range of [−75 m, 75 m]. It provides 3D labels for three classes: vehicle, pedestrian, and cyclist. We used LEVEL 2 as the default performance setting. Following the official evaluation metrics specified in [21], we report Multiple Object Tracking Accuracy (MOTA), false positives (FPs), misses, and mismatches for objects at the L2 difficulty.

4.1.2. Evaluation Metrics

We use the official evaluation metrics: Multiple Object Tracking Accuracy (MOTA), Multiple Object Tracking Precision (MOTP) [38], Higher-Order Tracking Accuracy (HOTA) [39], ID switches (IDSs), fragmentation (Frag), Average Multiple Object Tracking Accuracy (AMOTA), and Average Multiple Object Tracking Precision (AMOTP) given by the multi-object tracking challenge to evaluate this work. Among them, MOTA is defined as follows:

$$MOTA=1-{\displaystyle \frac{FP+FN+IDS}{nu{m}_{gt}}},$$

(16)

where $FP$, $FN$, and $IDS$ are the numbers of false positives, false negatives, and ID switches. The definition of AMOTA is as follows:

$$AMOTA={\displaystyle \frac{1}{L}}\sum _{r\in \left\{{\displaystyle \frac{1}{L}},{\displaystyle \frac{2}{L}},{\displaystyle \frac{3}{L}},...,1\right\}}MOT{A}_r,$$

(17)

where r is recall at specific thresholds, and L is the number of confidence thresholds. $MOT{A}_r$ represents $MOTA$ computed at r. MOTP is defined as

$$MOTP={\displaystyle \frac{{\sum }_{i,t}di{s}_{i,t}}{{\sum }_t{c}_t}},$$

(18)

where $c_t$ represents the number of successful matches between the predicted detection and the corresponding ground truth in the $t\mathrm{th}$ frame, and $di{s}_{i,t}$ is defined. On the NuScenes benchmark, $di{s}_{i,t}$ is the Euclidean distance, which means that the smaller the MOTP, the better.

4.2. Implementation Details

In our experiments, we followed the experimental settings of CenterPoint. The method was run on a computer with an Intel(R) Xeon(R) Platinum 8358P CPU (15 vCPU @ 2.60 GHz) and an RTX 3090 GPU (24 GB). The parameters of all compared methods were set according to their best performances. In addition, the hyperparameter settings presented in this work are shown in

Table 1. All hyperparameters set in this study.

Hyperparameter	Source	Value
K	Section 3.1.2	20
$F_i^{\prime }$	Equation (3)	256
$w_{\theta }$	Equation (12)	0.55
$w_v$	Equation (12)	0.45
$\alpha$	Equation (13)	0.5
$\beta$	Equation (13)	0.3
$\gamma$	Equation (13)	0.2
$s(r_i^d,r^D)$	Equation (14)	0.75
$p\left(d_i\right|T)$	Equation (15)	0.8

. The loss functions for the relevant modules are detailed as follows:

ReID Loss

$$L_{reid}=\sum _{i\in \mathrm{unmatched}}\left(1-{\displaystyle \frac{1}{\left|\mathcal{T}\right|}}\sum _{j\in \mathcal{T}}p\left(d_i\right|T_j)\right)$$

(19)

where $\mathcal{T}$ is the set of all trajectories in the current frame. $p\left(d_i\right|T_j)$ is the association probability between detection $d_i$ and trajectory $T_j$. $\left|\mathcal{T}\right|$ is the number of trajectories in the current frame.

By integrating the comprehensive cost matrix and ReID-based re-association in the data association stage, our method effectively enhances the robustness and accuracy of multi-object tracking in 3D point cloud data.

4.3. Comparison with Different Methods

4.3.1. NuScenes Open Dataset

We evaluated our approach on the NuScenes [37] validation set, and the results are presented in

Table 2. Performance of nuScenes 3D tracking validation set. All methods listed are LIDAR only, without multimodal extension.

Method	AMOTA	AMOTP	MOTA	IDS
AB3DMOT [40]	57.8	80.7	51.4	1275
Probabilistic [16]	56.1	80.0	48.3	679
MPN-Baseline [27]	59.3	83.2	51.4	1079
CenterPoint [15]	66.5	56.7	56.2	562
Ours	67.0	56.6	57.3	543

. Our method achieves an AMOTA score of 67.0, surpassing CenterPoint’s 66.5, indicating superior overall tracking accuracy. This improvement is largely attributed to the sophisticated cost matrix optimization and effective ReID-based matching, which reduce false positives (FPs) and false negatives (FNs). Our AMOTP score of 56.6 demonstrates competitive tracking precision, comparable to CenterPoint’s 56.7. The integration of detailed motion features ensures precise tracking even in challenging scenarios. Although the improvement in MOTA is marginal, with our score at 57.3 compared to CenterPoint’s 56.2, this still reflects consistent tracking accuracy. Our MOTA score benefits from the accurate motion model and the minimum-cost flow optimization, which collectively enhance the overall tracking accuracy by reducing errors in track association. One slight improvement in our method is the reduction in identity switches (IDSs) to 543, compared to CenterPoint’s 562. This reduction is primarily due to the combined effects of our backbone aggregating neighborhood information, the sophisticated cost matrix calculation in bipartite graph matching, and the final dynamic feature-based ReID, which collectively minimize instances of identity switching.

Comparison with Methods in Seven Classes Quantitative results using the NuScenes validation dataset are presented in Figure 6. This bar chart displays the performance of four competing methods in terms of the key evaluation metric, AMOTA, for the NuScenes evaluation sequences. Our proposed method achieves the highest overall AMOTA value, excelling in all classes, especially for several specific classes, including pedestrian, bicycle, and motorcycle. The AMOTA metric reflects the overall tracking performance of an MOT method by considering FPs, FNs, and IDSs across all recall values. Therefore, the superior AMOTA value attained by our method indicates that it surpasses its competitors, delivering an enhanced and comprehensive 3D tracking performance.

4.3.2. Waymo Open Dataset

In

Table 3. Tracking performance on the Waymo Open Dataset LEVEL 2 validation split. Arrows indicate whether a higher (↑) or lower (↓) value is better.

Method	Vehicle				Pedestrian				Cyclist
	MOTA↑	FP%↓	Miss%↓	IDS%↓	MOTA↑	FP%↓	Miss%↓	IDS%↓	MOTA↑	FP%↓	Miss%↓	IDS%↓
AB3DMOT [40]	55.7	-	30.2	0.40	52.2	-	-	2.74	-	-	-	-
CenterPoint [15]	55.1	10.8	33.9	0.26	54.9	10.0	34.0	1.13	57.4	13.7	28.1	0.83
SimpleTrack [42]	56.1	10.4	33.4	0.08	57.8	10.9	30.9	0.42	56.9	11.6	30.9	0.56
ImmortalTrack [43]	56.4	10.2	33.4	0.01	58.2	11.3	30.5	0.26	59.1	11.8	28.9	0.10
SpOT [44]	55.7	11.0	28.4	33.2	0.18	60.5	11.3	27.6	0.56	-	-	-
Ours	56.5	10.1	32.2	0.18	62.0	10.7	27.4	0.52	60.9	11.3	27.1	0.26

, we compare our method with other 3D MOT methods on the validation set of the Waymo [41] Open Dataset, where our method exhibits superior performance compared to others. Specifically, it outperforms the highest reported performance by 0.8%, 3.8%, and 1.8% in terms of the MOTA metric for vehicle, pedestrian, and cyclist, respectively, and exceeds the adopted CenterPoint baseline by 1.4%, 7.1%, and 3.5%. More specifically, our method shows significant improvement in the miss metric compared to the employed baseline, indicating that our approach can successfully recover objects missed by the detector. We attribute this success to our intra-frame graph geometrical structure and bipartite graph data association strategy, which captures object features, especially for small objects, from their neighborhood, allowing our model to better capture the spatial features of nearby objects. Additionally, our bipartite graph data association method can associate objects with more motion information through global optimization. Moreover, for pedestrians, our method achieves lower false positive (FP) values compared to other methods, indicating higher quality in our trajectories. Pedestrian trajectories are more complex and crowded compared to other categories, making it challenging for the network to generate correct associations. Therefore, the lower FP rate for pedestrians indicates that our method can handle associations in complex scenarios. This is largely due to our ReID module, which uses the predicted velocity and heading angle as motion information to compute unmatched detection and trajectories’ similarity and association probabilities, allowing our model to better associate correct track IDs during temporary occlusions. The results demonstrate that our model can achieve even higher performance due to the module for handling short-term occlusions. After bipartite graph matching, the unmatched detections are re-sorted based on their confidence scores and are matched again within the current frame and across the next three frames. This strategy ensures that short-term occlusions are handled effectively, enabling our model to maintain accurate tracking even in challenging scenarios.

As shown in Figure 7, our method’s robustness and accuracy improvements are visually demonstrated.

In Figure 7, (a) depicts the tracking scenario in the Waymo dataset involving vehicles, pedestrians, and bicycles. Due to unmatched detections marked in pink, the issue shown in (b) arises. Occlusion causes an ID switch, as illustrated in (c). Our algorithm successfully re-associates the data and matches the correct IDs. Finally, after a brief period of tracking, our algorithm can re-track the targets even after short-term occlusions, as demonstrated in (e). This process illustrates the capability of our method to maintain high tracking accuracy and robustness, even in complex scenarios involving temporary occlusions.

4.3.3. Comparison with Advanced Tracking Methods

• AB3DMOT [40]: This method employs a 3D Kalman filter for state estimation and the Hungarian algorithm for data association, providing robust tracking performance in 3D space.
• SimpleTrack [42]: This method uses non-maximum suppression for detection preprocessing, a Kalman filter for motion modeling, 3D Generalized IoU for association, and trajectory interpolation to achieve object tracking.
• ImmortalTrack [43]: This method uses a simple Kalman filter for trajectory prediction to maintain tracklets when the target is not visible, effectively preventing premature tracklet termination and reducing ID switches and track fragmentation.
• SpOT [44]: SpOT proposes a multi-frame spatio-temporal tracking method that utilizes 4D refinement for frame-by-frame detection data association, achieving efficient object tracking.
• CenterPoint [15]: This method first detects the centers of objects using a keypoint detector and regresses their 3D size, 3D orientation, and velocity. In the second stage, it refines these estimates using additional point features on the object.

4.3.4. Comparative Analysis

The primary differences between our method and the others lie in both the motion model and data association strategies. The Kalman filter (KF) utilized by AB3DMOT, SimpleTrack, and ImmortalTrack leverages historical information to predict the target’s state, providing smoother results when dealing with low-quality detections. However, the KF requires careful parameter initialization, and its robustness can be limited if the parameters are not appropriately set. Our method, on the other hand, employs a Constant Velocity (CV) model for motion prediction, which uses explicit speed predictions to handle abrupt and unpredictable motions more effectively. The simplicity of the CV model allows for straightforward implementation without the need for parameter tuning.

• Advantages and Disadvantages of the KF The Kalman filter can utilize information from multiple frames, resulting in smoother outcomes in scenarios with low-quality detections. However, it necessitates careful parameter initialization, and improper parameter settings can significantly impact its robustness.
• Advantages and Disadvantages of the CV Model The Constant Velocity model better handles abrupt and unpredictable motions with explicit speed predictions and is simpler to implement without requiring parameter tuning. Nevertheless, its effectiveness in motion smoothing is limited, and it may not perform as well in scenarios with low-quality detections.

Overall, while the KF offers smoother results by leveraging historical information, it requires the careful tuning of parameters, which can limit its robustness. In contrast, our CV-based method simplifies the tracking process, providing better handling of abrupt motions but with some limitations in motion smoothing.

4.3.5. Strengths and Weaknesses of Our Method

Our method, based on the CV model, includes in-frame graph construction, inter-frame bipartite matching, and ReID-based trajectory management. These innovations enhance data association and tracking robustness. As shown in Figure 7, our method can handle short-term occlusions and ID switches by re-matching and tracking the objects effectively. In Figure 10(1), even when detection confidence is occasionally low, our robust data association strategy ensures accurate tracking and matching. Weaknesses are discussed in the failure case analysis section.

4.4. Ablation Studies

4.4.1. Ablation Study for GBRTracker

As shown in

Table 4. Multi-object tracking ablation study on the nuScenes validation set. The rows correspond to different configurations of our network, GBRTracker, based on CenterPoint. The components tested are the graph backbone, bipartite graph matching, and ReID ReTrack module. Arrows indicate whether a higher (↑) value is better.

Method	AMOTA↑
CenterPoint	66.5
CenterPoint + graph backbone	66.6
CenterPoint + ReID ReTrack	66.64
CenterPoint + graph backbone + ReID ReTrack	66.68
CenterPoint + bipartite graph matching	66.76
CenterPoint + graph backbone + bipartite graph matching	66.8
CenterPoint + bipartite graph matching + ReID ReTrack	66.83
CenterPoint + graph backbone + bipartite graph matching + ReID ReTrack (GBRTracker)	67.0

, our ablation study on the NuScenes validation set evaluated the impact of various components of GBRTracker on its overall performance. Integrating the graph backbone alone improved AMOTA by +0.1, indicating enhanced feature extraction through a KNN graph structure. Adding the bipartite graph matching module resulted in a +0.26 AMOTA increase, validating its advantage in data association. Incorporating the ReID ReTrack module led to an AMOTA improvement of +0.14, demonstrating its effectiveness in reconnecting lost objects or initiating new tracks.

Combining the graph backbone with bipartite graph matching increased AMOTA by +0.3, highlighting their complementary effects in feature extraction and data association. The combination of the graph backbone and ReID ReTrack improved AMOTA by +0.18, showing better target re-identification. Combining bipartite graph matching with ReID ReTrack resulted in a +0.33 AMOTA increase, indicating improvements in data association and re-identification.

Finally, integrating all three modules, GBRTracker achieved a +0.5 AMOTA improvement, demonstrating that these components work well together and provide the best overall multi-object tracking performance.

4.4.2. Influence of Object Detection Module

Effectiveness of Backbone on Tracker As can be seen in

Table 5. Results of using different backbones on same tracker on NuScenes validation set.

Method	AMOTA	AMOTP	MOTA
CenterPoint
VoxelNet	0.665	0.567	0.562
Ours	0.667	0.565	0.563
GBRTracker (ours)
VoxelNet	0.669	0.558	0.581
Ours	0.670	0.557	0.583

, for CenterPoint, our designed graph VoxelNet backbone shows slight improvements in AMOTA (from 0.665 to 0.667) and MOTA (from 0.562 to 0.563), indicating a marginal positive impact of EdgeConv integration. For GBRTracker, the improvements are more pronounced. AMOTA increases from 0.669 to 0.670 and MOTA from 0.581 to 0.583, suggesting that our tracking approach benefits more from the enhanced feature representation provided by our designed backbone. Overall, our graph-based VoxelNet provides slight performance gains across both tracking frameworks, with our GBRTracker showing a better utilization of the improved backbone features.

Effectiveness of K Value in KNN Graphs

As shown in Figure 8, the ablation study on KNN k values demonstrates that using k = 20 yields the highest AMOTA scores for both cars and pedestrians, indicating optimal performance. Increasing k beyond 20 results in decreased AMOTA, suggesting that larger neighborhood sizes may introduce noise, thereby reducing tracking accuracy.

4.4.3. Effectiveness of Aggregated Pairwise Cost

As shown in

Table 6. Results of using different affinity models on the Nuscenes validation dataset, where IoU, Cen., and Motion denote EIoU-based BBox distance, center distance, and combined velocity and angle pairwise cost matrix, respectively. The arrows indicate the direction of improvement or degradation: ↑ for increase and ↓ for decrease.

Cost			MOTA ↑	IDS ↓	FRAG ↓
IoU	Cen.	Motion
√	√	-	0.53%	2.14%	0.94%
√	-	√	0.89%	3.03%	2.12%
-	√	√	1.42%	3.91%	3.31%
√	√	√	1.96%	4.27%	4.48%

, the ablation study demonstrates the effectiveness of incorporating different costs into the bipartite graph matching process. Notably, the introduction of motion-based costs (velocity and angle) in combination with IoU and center distance metrics yields the highest improvement in performance. The baseline results for CenterPoint are $MOTA=0.562$, $IDS=562$, and $FRAG=424$. Specifically, our tracker with all costs (IoU, center, and motion) achieves a 1.96% increase in MOTA, a 4.27% reduction in IDSs, and a 4.48% reduction in FRAG compared to the baseline. This indicates that while IoU and center distance are essential, the motion information significantly enhances the model’s ability to maintain accurate and stable object tracks, leading to fewer identity switches and fragmentation events. Thus, the motion cost matrix plays a crucial role in improving tracking robustness and overall performance.

4.4.4. Evaluation of Similarity Score and Association Probability for Track Update

Based on the heatmap in Figure 9, the analysis of the effect of different thresholds ${\lambda }_D$ for similarity scores and ${\lambda }_S$ for association probabilities on AMOTA demonstrates significant impacts on tracking performance. As the similarity score threshold ${\lambda }_D$ increases, AMOTA improves, reaching a peak value of 67.0 at a similarity threshold of 0.75 and an association probability threshold ${\lambda }_S$ of 0.8. This indicates that a moderate level of similarity and association probability optimizes the tracking performance. However, thresholds that are either too low or too high lead to decreased AMOTA values. Specifically, lower similarity thresholds (e.g., 0.6) and higher association probability thresholds (e.g., 0.9) result in lower AMOTA scores, indicating that too lenient or too strict criteria for matching detections can reduce tracking accuracy. It is important to carefully tune the similarity score and association probability thresholds to achieve optimal tracking performance.

4.5. Discussion of Failure Cases and Future Challenges

4.5.1. Object Suddenly Appears with Delayed Tracking

Cause Analysis As shown in Figure 10(1), the object suddenly appeared at a distance in (b) with low confidence due to point cloud detection limitations, resulting in no trajectory initialization (as the confidence was too low to start a new track). However, in (c), even with low confidence, it still performed correct tracking due to our bipartite graph matching for data association. This indicates that our current trajectory initialization method is not well suited for handling low-confidence detections of newly appearing distant objects.

Future Challenges To address this, future work should focus on integrating multimodal data, such as utilizing RGB information from images to enhance detection confidence for distant objects. This can improve the robustness of the model in initializing tracks for newly appearing objects.

4.5.2. Complete Occlusion

Cause Analysis As shown in Figure 10(2), complete occlusion by other objects leads to tracking algorithm failure. In (b), the tracking-by-detection-based detector cannot predict proposals for the occluded object, preventing data association between the current detection and last-frame tracking results, resulting in failure, as seen in (c).

Future Challenges Future work could involve using the extended Kalman filter (EKF) or Bayesian estimation methods to predict the object’s state during complete occlusion. By utilizing historical trajectory and motion models, the algorithm can roughly estimate the target’s position. However, optimizing filter parameters for various datasets remains an intrinsic problem, as the robustness of these methods heavily depends on proper parameter initialization. Additionally, complete occlusion cannot be fully addressed by filtering methods alone, and it remains a challenging problem in the field of multi-object tracking that requires further research.

5. Conclusions

In this work, we proposed GBRTracker, a novel approach to 3D multi-object tracking (MOT) in point cloud data. By integrating an intra-frame graph structure into the backbone, our method effectively reduced detection misses through enhanced spatial feature aggregation. Additionally, we constructed an inter-frame bipartite graph for data association, leveraging a customized cost matrix incorporating center, box size, velocity, and heading angle information. Consequently, our method, coupled with a minimum-cost flow algorithm, achieved globally optimal matching and minimized ID switches. For unmatched detections, we designed a motion-based re-identification (ReID) feature embedding module to reconnect lost objects or initiate new tracks, maintaining high tracking accuracy and reliability. The results demonstrated that GBRTracker outperformed state-of-the-art techniques on the NuScenes and Waymo Open Datasets. However, it faced challenges in re-associating objects lost during several frames of short-term tracking, particularly when objects made sudden turns or stopped abruptly, affecting the effectiveness of the motion cost matrix. This highlights a potential limitation in our approach. In future work, we will focus on multi-frame Kalman filter-based methods to use historical information to improve tracking performance or integrate multimodal information through the incorporation of image data for multi-object tracking.